From 4f265b312d91d2641c84f4b6bccf82d613250106 Mon Sep 17 00:00:00 2001 From: you Date: Sat, 4 Apr 2026 04:07:21 +0000 Subject: [PATCH] docs: add startup load mechanics, in-memory structure, and dual-write pattern to resolved-path spec --- docs/specs/resolved-path.md | 44 ++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/docs/specs/resolved-path.md b/docs/specs/resolved-path.md index 6dc5c9ba..4dc7366e 100644 --- a/docs/specs/resolved-path.md +++ b/docs/specs/resolved-path.md @@ -62,11 +62,38 @@ ON CONFLICT(node_a, node_b) DO UPDATE SET last_seen = excluded.last_seen; ``` +### In-memory structure + +```go +type NeighborGraph struct { + edges map[string][]NeighborEdge // pubkey → list of neighbor edges + mu sync.RWMutex +} +``` + ### Cold startup and backfill On startup: -1. **Load `neighbor_edges` from SQLite** → build in-memory graph. +```go +// 1. Query all edges from SQLite +rows := db.Query("SELECT node_a, node_b, count, last_seen FROM neighbor_edges") + +// 2. Build in-memory graph +graph := NewNeighborGraph() +for rows.Next() { + var a, b string + var count int + var lastSeen string + rows.Scan(&a, &b, &count, &lastSeen) + graph.UpsertEdge(a, b, count, lastSeen) +} + +// 3. Attach to PacketStore +store.graph = graph +``` + +1. **Load `neighbor_edges` from SQLite** → build in-memory graph (code above). 2. **If table empty (first run):** `BuildFromStore(packets)` — scan all existing packets, extract edges per the rules above, INSERT into `neighbor_edges`. 3. **Load observations from SQLite.** 4. **For observations without `resolved_path`:** resolve using the graph, UPDATE `resolved_path` in SQLite. @@ -74,6 +101,21 @@ On startup: On subsequent runs, step 2 is skipped (table already populated). Step 4 only processes observations with NULL `resolved_path` (new or previously unresolved). +### Incremental update at ingest + +Every edge upsert writes to **both** the in-memory graph and SQLite — they stay in sync. SQLite is the persistence layer, in-memory is the fast lookup layer. + +```go +// Extract edge from packet +graph.UpsertEdge(nodeA, nodeB, 1, now) + +// Also persist to SQLite +db.Exec(`INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) + VALUES (?, ?, 1, ?) + ON CONFLICT(node_a, node_b) DO UPDATE SET + count = count + 1, last_seen = ?`, a, b, now, now) +``` + ## Data model ### Where does `resolved_path` live?