docs: add startup load mechanics, in-memory structure, and dual-write pattern to resolved-path spec

docs/specs/resolved-path.md

@@ -0,0 +1,272 @@
+# Spec: Server-side hop resolution at ingest — `resolved_path`
+
+**Status:** Final  
+**Issue:** [#555](https://github.com/Kpa-clawbot/CoreScope/issues/555)  
+**Related:** [#482](https://github.com/Kpa-clawbot/CoreScope/issues/482), [#528](https://github.com/Kpa-clawbot/CoreScope/issues/528)
+
+## Problem
+
+Any place where 1, 2, or 3-byte prefixes must be resolved to actual full repeater public keys and friendly names should use affinity data first, geo data as fallback. Across frontend, backend, whatever. Efficiently — no 7-second waits, no recomputation, aggressive caching.
+
+Currently, hop paths are stored as short uppercase hex prefixes in `path_json` (e.g. `["D6", "E3", "59"]`). Resolution to full pubkeys happens **client-side** via `HopResolver` (`public/hop-resolver.js`), which:
+
+- Is slow — each page/component re-resolves independently
+- Is inconsistent — different components may resolve the same prefix differently
+- Cannot leverage the server's neighbor affinity graph, which has far richer context for disambiguation
+- Causes redundant `/api/resolve-hops` calls from every client
+
+## Solution
+
+Resolve hop prefixes to full pubkeys **once at ingest time** on the server, using `resolveWithContext()` with 4-tier priority (affinity → geo → GPS → first match) and a **persisted neighbor graph**. Store the result as a new `resolved_path` column on observations alongside `path_json`.
+
+## Design decisions (locked)
+
+1. **`path_json` stays unchanged** — raw firmware prefixes, uppercase hex. Ground truth.
+2. **`resolved_path` is a column on observations** — full 64-char lowercase hex pubkeys, `null` for unresolved.
+3. **Resolved at ingest** using `resolveWithContext(hop, context, graph)` — 4-tier priority: affinity → geo → GPS → first match.
+4. **`null` = unresolved** — ambiguous prefixes store `null`. Frontend falls back to prefix display.
+5. **Both fields coexist** — not interchangeable. Different consumers use different fields.
+
+## Persisted neighbor graph
+
+### SQLite table: `neighbor_edges`
+
+Thin and normalized. Stores ONLY the relationship. SNR, observer names, GPS, roles — all join from existing tables when needed. No duplication.
+
+```sql
+CREATE TABLE IF NOT EXISTS neighbor_edges (
+    node_a TEXT NOT NULL,
+    node_b TEXT NOT NULL,
+    count INTEGER DEFAULT 1,
+    last_seen TEXT,
+    PRIMARY KEY (node_a, node_b)
+);
+```
+
+### Edge extraction rules (ADVERT vs non-ADVERT)
+
+At ingest, for each packet:
+
+- **ADVERT packets** (payload_type 4): originator pubkey is known from `decoded_json.pubKey`. Extract edge: `originator ↔ path[0]` (the first hop is a direct neighbor of the originator).
+- **ALL packets**: observer pubkey is known. Extract edge: `observer ↔ path[last]` (the last hop is a direct neighbor of the observer).
+- **Non-ADVERT packets**: originator is unknown (encrypted). ONLY extract `observer ↔ path[last]`.
+- Each packet produces **1 or 2 edge upserts** depending on type.
+
+Edge upsert uses canonical ordering (`node_a < node_b` lexicographically) to avoid duplicate edges:
+
+```sql
+INSERT INTO neighbor_edges (node_a, node_b, count, last_seen)
+VALUES (min(?, ?), max(?, ?), 1, ?)
+ON CONFLICT(node_a, node_b) DO UPDATE SET
+  count = count + 1,
+  last_seen = excluded.last_seen;
+```
+
+### In-memory structure
+
+```go
+type NeighborGraph struct {
+    edges map[string][]NeighborEdge  // pubkey → list of neighbor edges
+    mu    sync.RWMutex
+}
+```
+
+### Cold startup and backfill
+
+On startup:
+
+```go
+// 1. Query all edges from SQLite
+rows := db.Query("SELECT node_a, node_b, count, last_seen FROM neighbor_edges")
+
+// 2. Build in-memory graph
+graph := NewNeighborGraph()
+for rows.Next() {
+    var a, b string
+    var count int
+    var lastSeen string
+    rows.Scan(&a, &b, &count, &lastSeen)
+    graph.UpsertEdge(a, b, count, lastSeen)
+}
+
+// 3. Attach to PacketStore
+store.graph = graph
+```
+
+1. **Load `neighbor_edges` from SQLite** → build in-memory graph (code above).
+2. **If table empty (first run):** `BuildFromStore(packets)` — scan all existing packets, extract edges per the rules above, INSERT into `neighbor_edges`.
+3. **Load observations from SQLite.**
+4. **For observations without `resolved_path`:** resolve using the graph, UPDATE `resolved_path` in SQLite.
+5. **Ready to serve.**
+
+On subsequent runs, step 2 is skipped (table already populated). Step 4 only processes observations with NULL `resolved_path` (new or previously unresolved).
+
+### Incremental update at ingest
+
+Every edge upsert writes to **both** the in-memory graph and SQLite — they stay in sync. SQLite is the persistence layer, in-memory is the fast lookup layer.
+
+```go
+// Extract edge from packet
+graph.UpsertEdge(nodeA, nodeB, 1, now)
+
+// Also persist to SQLite
+db.Exec(`INSERT INTO neighbor_edges (node_a, node_b, count, last_seen)
+         VALUES (?, ?, 1, ?)
+         ON CONFLICT(node_a, node_b) DO UPDATE SET
+         count = count + 1, last_seen = ?`, a, b, now, now)
+```
+
+## Data model
+
+### Where does `resolved_path` live?
+
+**On observations**, as a column:
+
+```sql
+ALTER TABLE observations ADD COLUMN resolved_path TEXT;
+```
+
+Rationale: Each observer sees the packet from a different vantage point. The same 2-char prefix may resolve to different full pubkeys depending on which observer's neighborhood is considered. The observer's own pubkey provides critical context for `resolveWithContext` (tier 2: neighbor affinity). Storing on observations preserves this per-observer resolution.
+
+`resolved_path` is written in the same INSERT that creates the observation — one write, no double-write problem.
+
+### Field shape
+
+```
+resolved_path TEXT  -- JSON array: ["aabb...64chars", null, "ccdd...64chars"]
+```
+
+- Same length as the `path_json` array
+- Each element is either a 64-char lowercase hex pubkey string, or `null`
+- Stored as a JSON text column (same approach as `path_json`)
+- Uses `omitempty` — absent from JSON when not set
+
+## Every path resolution uses the graph — no exceptions
+
+All existing `pm.resolve()` call sites MUST be migrated to `resolveWithContext` with the persisted graph. No "we'll get to it later."
+
+### Call sites to migrate (exhaustive)
+
+Found via `grep -n "pm.resolve" cmd/server/store.go`:
+
+| Line | Function | Current | After |
+|------|----------|---------|-------|
+| 1192 | `IngestNewFromDB()` | `pm.resolve(hop)` | `resolveWithContext(hop, ctx, graph)` — resolve at ingest, store as `resolved_path` |
+| 1876 | `buildDistanceIndex()` | `pm.resolve(hop)` | Read `resolved_path` from observation — already resolved at ingest |
+| 3537 | `computeAnalyticsTopology()` | `pm.resolve(hop)` | Read `resolved_path` from observation |
+| 5528 | `computeAnalyticsSubpaths()` | `pm.resolve(hop)` | Read `resolved_path` from observation |
+| 5665 | `GetSubpathDetail()` | `pm.resolve(hop)` | `resolveWithContext(hop, ctx, graph)` — ad-hoc resolution for user-provided hops |
+| 5744 | `GetSubpathDetail()` | `pm.resolve(h)` | `resolveWithContext(h, ctx, graph)` — same function, second usage |
+
+**After migration:** `pm.resolve()` (naive prefix-only lookup) is dead code. Remove it. All resolution goes through `resolveWithContext` which uses the persisted neighbor graph for affinity-based disambiguation.
+
+## Ingest pipeline changes
+
+### Where resolution happens
+
+In `PacketStore.IngestNewFromDB()` in `cmd/server/store.go`. For new observations, resolution happens during the observation INSERT — same write. For backfill (cold startup), it's a separate UPDATE pass.
+
+Note on ordering: edge upserts (step 5) happen **after** resolution (step 3-4). This means the very first packet for a new neighbor pair resolves without that edge in the graph yet. This is acceptable — the affinity tier will miss, but geo/GPS/first-match tiers still work. On the next packet, the edge exists and affinity kicks in.
+
+Resolution flow per observation:
+1. Parse `path_json` into hop prefixes
+2. Build context pubkeys from the observation (observer pubkey, source/dest from decoded packet)
+3. Call `resolveWithContext(hop, contextPubkeys, neighborGraph)` for each hop
+4. Store result as `resolved_path` column on the observation (same INSERT)
+5. Upsert neighbor edges into `neighbor_edges` table (incremental update)
+
+### Performance
+
+`resolveWithContext` does:
+- Prefix map lookup (map access, O(1))
+- Optional neighbor graph check (small map lookups)
+- No DB queries, no network calls
+
+Per-hop cost: ~1–5μs. A typical packet has 0–5 hops. At 100 packets/second ingest rate, this adds <0.5ms total overhead per second. **Negligible.**
+
+## All consumers use `resolved_path`
+
+| Consumer | Before | After |
+|---|---|---|
+| Packets detail path names | Client HopResolver (naive) | Read `resolved_path` |
+| Map Show Route | Client HopResolver (naive) | Read `resolved_path` |
+| Live map animated paths | Client HopResolver (naive) | Read `resolved_path` |
+| Node detail paths | Client HopResolver (naive) | Read `resolved_path` |
+| Analytics topology | Server `pm.resolve()` (naive) | Read `resolved_path` from observations |
+| Analytics subpaths | Server `pm.resolve()` (naive) | Read `resolved_path` from observations |
+| Analytics hop distances | Server `pm.resolve()` (naive) | Read `resolved_path` from observations |
+| Subpath detail | Server `pm.resolve()` (naive) | `resolveWithContext` with graph |
+| Show Neighbors | Server neighbors API | Already correct |
+| `/api/resolve-hops` | Server `resolveWithContext` | Already correct |
+| Hex breakdown display | `path_json` raw | Unchanged — shows raw bytes |
+
+## WebSocket broadcast
+
+Include `resolved_path` in broadcast messages. Resolution happens before broadcast assembly — negligible latency impact. The WS broadcast already includes `path_json`; `resolved_path` is added alongside it.
+
+## API changes
+
+### Endpoints that return `resolved_path`
+
+All endpoints that currently return `path_json` also return `resolved_path`:
+
+- `GET /api/packets` — transmission-level (use best observation's `resolved_path`)
+- `GET /api/packets/:hash` — per-observation detail
+- `GET /api/packets/:hash/observations` — each observation includes its own `resolved_path`
+- WebSocket broadcast messages — per-observation
+
+### `/api/resolve-hops`
+
+**Kept.** Useful for ad-hoc resolution of arbitrary prefixes (debug tools, clients resolving prefixes not associated with a packet). Not deprecated.
+
+## Pubkey case convention
+
+- **DB/API:** lowercase
+- **`path_json` display prefixes:** uppercase (raw firmware)
+- **`resolved_path`:** lowercase full pubkeys
+- **Comparison code:** normalizes to lowercase
+
+## Backward compatibility
+
+- Old observations without `resolved_path`: resolved during cold startup backfill (step 4). If still `null` after backfill, frontend falls back to client-side HopResolver.
+- `resolved_path` field uses `omitempty` — absent from JSON when not set.
+
+### Fallback pattern (frontend)
+
+```javascript
+function getResolvedHops(packet) {
+  if (packet.resolved_path) return packet.resolved_path;
+  // Fall back to client-side resolution for old packets
+  return resolveHopsClientSide(packet.path_json);
+}
+```
+
+## Implementation milestones
+
+### M1: Persist graph to SQLite + load on startup + incremental updates at ingest
+- Create `neighbor_edges` table in SQLite (schema above)
+- On first run: `BuildFromStore(packets)` — scan all packets, extract edges per ADVERT/non-ADVERT rules, INSERT into table
+- On subsequent runs: load from SQLite → build in-memory graph (instant startup)
+- Upsert edges incrementally during packet ingest
+- Graph lives on `PacketStore`, not `Server`
+- Tests: graph persistence, load, incremental update, ADVERT vs non-ADVERT edge extraction
+
+### M2: Add `resolved_path` column to observations + resolve at ingest
+- `ALTER TABLE observations ADD COLUMN resolved_path TEXT`
+- Add `ResolvedPath []*string` to `Observation` struct
+- Resolve during `IngestNewFromDB` — same INSERT, one write
+- Cold startup backfill: resolve observations with NULL `resolved_path`, UPDATE in SQLite
+- Migrate ALL 6 `pm.resolve()` call sites to `resolveWithContext` or read from `resolved_path`
+- Remove dead `pm.resolve()` code
+- Tests: unit test resolution at ingest, verify stored values, verify all call sites use graph
+
+### M3: Update all API responses to include `resolved_path`
+- Include `resolved_path` in all packet/observation API responses
+- Include in WebSocket broadcast messages
+- Tests: verify API response shape, WS broadcast shape
+
+### M4: Update frontend consumers to prefer `resolved_path`
+- Update `packets.js`, `map.js`, `live.js`, `analytics.js`, `nodes.js`
+- Add fallback to `path_json` + `HopResolver` for old packets
+- `hop-resolver.js` becomes fallback only
+- Tests: Playwright tests for path display