mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-10 20:32:01 +00:00
3feb97f16f
# fix(ingestor): write resolved_path on new observations (full restore — closes #1547 + #1560) Fixes #1547. Closes #1560. ## Root cause PR #1289 (the "ingestor owns the neighbor graph; server is read-only" refactor, ~2026-05-21) moved the neighbor graph + schema writes to the ingestor, and as a side-effect removed the server-side writer that populated `observations.resolved_path` AND the context-aware `pm.resolveWithContext` that disambiguated 1-byte prefix collisions. Result: every observation inserted after the deploy has `resolved_path = NULL` (3.1M/6.3M NULL on staging; 100% NULL on fresh deploys; symptom on Cascadia: hops fail to resolve because the small-mesh client-side fallback breaks on prefix collisions). ## Full restore This PR resolves both single-byte and multi-byte prefix paths. Single-byte disambiguation uses NeighborGraph adjacency and ADVERT `from_pubkey` anchoring, ported from pre-#1289 `pm.resolveWithContext` logic (last good at cmd/server/store.go @ commit 450236d5) and the #1144 / #1352 fixes. New file `cmd/ingestor/path_resolver.go`: - `NeighborGraph` + `neighborGraphHolder` — in-memory adjacency snapshot, atomic-published. - `loadNeighborGraph(db)` — one-shot SELECT from `neighbor_edges`. - `resolveHopWithContext(hop, anchor, graph, idx, exclude) *string` — single-hop, tier-1 disambiguator. - `resolvePathWithContext(hops, fromPubkey, graph, idx) []*string` — walks the path, anchoring hop 0 on `from_pubkey` (ADVERTs) and each subsequent hop on the previous resolved hop, excluding already-resolved pubkeys. - `Store.RefreshNeighborGraph()` — called on warm-up and every 60s tick in the neighbor-edges builder alongside `RefreshPrefixIndex`. Existing file `cmd/ingestor/resolved_path.go` (PR #1547 base) is untouched: `resolvePath` + `marshalResolvedPath` + the all-nil → empty-string clobber-guard contract are preserved verbatim. `cmd/ingestor/db.go` — `InsertTransmission` now calls `resolvePathWithContext` instead of the naive `resolvePath`. ## Algorithm (per hop) 1. Look up candidate pubkeys by prefix-match (existing `prefixIndex`). 2. `len==0 → nil`; `len==1 → that pubkey`. 3. `len>1` → filter by `NeighborGraph` adjacency to the anchor. Anchor is `from_pubkey` for hop 0 on ADVERTs, the previous resolved hop otherwise. Exactly 1 surviving candidate → use it; else nil. 4. Previously resolved hops (and the originator) are excluded from downstream candidate pools — a packet does not revisit a node. Tier-2/3/4 from pre-#1289 (geo proximity, GPS preference, observation-count fallback) are intentionally NOT ported — those were noisy in practice and belong in a separate enhancement, not in this regression restore. ## Out of scope - The ~3.1M existing NULL rows from the regression window. Filed as a follow-up backfill task — too risky to bundle here (touches a 6M-row table). - The dead-flag bug #1546 — separate concern. ## TDD red → green - Red commit `80b0f476` — adds five new context-resolver tests; stub `resolvePathWithContext` falls back to naive `resolvePath`. CI run 26946935615 → **failure** with assertion errors on the three collision tests (`TestResolveHopWithContext_OneByteCollision_AdjacencyResolves`, `TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode`, `TestResolvePathWithContext_AdvertAnchoring`); the two regression tests (multi-byte still works + all-nil contract) stayed green. - Green commit `7b4950ce` — real algorithm + InsertTransmission wiring + RefreshNeighborGraph in the builder tick. All five new tests pass; original four `resolved_path` tests stay green. ## Verification - `go test -race ./cmd/ingestor/...` for the 11 affected tests — pass. - `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — exit 0 (all gates clean). - PII grep on body + diff: clean. Tested with: existing `TestInsertTransmissionWritesResolvedPath` + `TestInsertTransmissionDoesNotClobberResolvedPathOnAllNil` (PR #1547 base) plus the new collision-resolution suite: - `TestResolveHopWithContext_OneByteCollision_AdjacencyResolves` — 3-of-5 nodes share `0x5c`, chain A↔B↔C↔D↔E; anchored on A, hop `5c` → B. - `TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode` — path `[5c, 5c]` from_node A → `[B, C]`. - `TestResolveHopWithContext_NoAdjacencyContext_ReturnsNil` — 3 ambiguous candidates, no anchor / non-adjacent anchor → nil. - `TestResolvePathWithContext_AdvertAnchoring` — ADVERT, `from_pubkey=A`, path `[5c]` → only-adjacent neighbor B. - `TestResolvePathWithContext_RegressionMultiByteStillWorks` — unique-prefix path with no graph context still resolves. - `TestResolvePathWithContext_AllNilContractPreserved` — unresolvable path → `marshalResolvedPath==""` (clobber-guard from PR #1548 untouched). ## Browser-validated N/A — backend-only change. Frontend already handles populated `resolved_path` via `getResolvedPath` in `cmd/server/db.go` and `public/packets.js`. ## Round-1 fixes addressed - **MUST-FIX #1 (data-loss clobber on all-nil resolution):** when every hop fails to resolve, `marshalResolvedPath` returns `""` instead of `"[null,null,...]"`, so `nilIfEmpty` → SQL NULL and the `COALESCE(excluded.resolved_path, resolved_path)` UPSERT preserves any previously stored good value on re-ingest. Regression test asserts: insert a transmission, observe `resolved_path` populated, wipe the prefix index, re-ingest the same packet, assert the existing `resolved_path` is unchanged. --------- Co-authored-by: corescope-bot <bot@corescope> Co-authored-by: openclaw-bot <bot@openclaw> Co-authored-by: openclaw-bot <bot@openclaw.local>
226 lines
6.4 KiB
Go
226 lines
6.4 KiB
Go
package main
|
|
|
|
import (
|
|
"database/sql"
|
|
"strings"
|
|
"sync/atomic"
|
|
)
|
|
|
|
// Context-aware hop resolver — full restore of pre-#1289 hop
|
|
// disambiguation semantics, ported into the ingestor (where the
|
|
// neighbor graph + node directory now live, per #1283).
|
|
//
|
|
// Why this exists (issues #1547 / #1560):
|
|
// The naive `resolvePath` only resolves hops whose prefix is unique
|
|
// in the node table. On a >2K-node mesh the dominant case is 1-byte
|
|
// prefix collisions (multiple candidates per prefix). Without
|
|
// adjacency disambiguation those hops always serialize as `nil`
|
|
// and the resolved_path remains effectively empty for the largest
|
|
// meshes — the very deployments that need it most.
|
|
//
|
|
// Algorithm (ported from cmd/server/store.go @ commit 450236d5
|
|
// `pm.resolveWithContext`, intersected with the disambiguation gating
|
|
// from PR #1144 / #1352):
|
|
//
|
|
// For each hop:
|
|
// 1. Collect candidate pubkeys by prefix-match (existing prefixIndex).
|
|
// 2. len==0 → nil.
|
|
// 3. len==1 → that pubkey.
|
|
// 4. len>1 → filter by NeighborGraph adjacency to the anchor:
|
|
// - hop 0 anchor = fromPubkey (ADVERT originator) if known;
|
|
// - hop i (i>0) anchor = previous resolved hop's pubkey;
|
|
// if the previous hop did not resolve, the chain breaks
|
|
// and subsequent >1-candidate hops fall to nil.
|
|
// Surviving candidates after filter:
|
|
// - exactly 1 → use it
|
|
// - 0 or >1 → nil (cannot disambiguate further)
|
|
//
|
|
// This is the conservative tier-1 variant. Pre-#1289 also carried
|
|
// tier-2 (geo proximity), tier-3 (GPS preference), tier-4 (obs-count
|
|
// fallback) — those were noisy in practice and are intentionally NOT
|
|
// ported here; this PR is a regression restore, not an enhancement.
|
|
|
|
// NeighborGraph is the in-memory adjacency snapshot used by the
|
|
// context-aware resolver. Internally lowercased.
|
|
type NeighborGraph struct {
|
|
adj map[string]map[string]struct{}
|
|
}
|
|
|
|
// NewNeighborGraph returns an empty graph.
|
|
func NewNeighborGraph() *NeighborGraph {
|
|
return &NeighborGraph{adj: make(map[string]map[string]struct{})}
|
|
}
|
|
|
|
// AddEdge adds an undirected adjacency a↔b. Self-loops and empty
|
|
// endpoints are ignored.
|
|
func (g *NeighborGraph) AddEdge(a, b string) {
|
|
a = strings.ToLower(a)
|
|
b = strings.ToLower(b)
|
|
if a == "" || b == "" || a == b {
|
|
return
|
|
}
|
|
if g.adj[a] == nil {
|
|
g.adj[a] = make(map[string]struct{})
|
|
}
|
|
if g.adj[b] == nil {
|
|
g.adj[b] = make(map[string]struct{})
|
|
}
|
|
g.adj[a][b] = struct{}{}
|
|
g.adj[b][a] = struct{}{}
|
|
}
|
|
|
|
// IsAdjacent reports whether a and b appear together in any neighbor edge.
|
|
func (g *NeighborGraph) IsAdjacent(a, b string) bool {
|
|
if g == nil {
|
|
return false
|
|
}
|
|
a = strings.ToLower(a)
|
|
b = strings.ToLower(b)
|
|
if a == "" || b == "" {
|
|
return false
|
|
}
|
|
nbrs, ok := g.adj[a]
|
|
if !ok {
|
|
return false
|
|
}
|
|
_, present := nbrs[b]
|
|
return present
|
|
}
|
|
|
|
// neighborGraphHolder caches the graph for the InsertTransmission hot
|
|
// path. atomic.Value lets the 60s rebuild publish without a read-side
|
|
// lock.
|
|
type neighborGraphHolder struct {
|
|
v atomic.Value // holds *NeighborGraph
|
|
}
|
|
|
|
func (h *neighborGraphHolder) load() *NeighborGraph {
|
|
if v := h.v.Load(); v != nil {
|
|
return v.(*NeighborGraph)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (h *neighborGraphHolder) store(g *NeighborGraph) {
|
|
h.v.Store(g)
|
|
}
|
|
|
|
// loadNeighborGraph reads neighbor_edges and returns an in-memory
|
|
// adjacency snapshot. Safe to call against a fresh DB (returns an
|
|
// empty graph).
|
|
func loadNeighborGraph(db *sql.DB) (*NeighborGraph, error) {
|
|
rows, err := db.Query(`SELECT node_a, node_b FROM neighbor_edges`)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer rows.Close()
|
|
g := NewNeighborGraph()
|
|
for rows.Next() {
|
|
var a, b string
|
|
if err := rows.Scan(&a, &b); err != nil {
|
|
continue
|
|
}
|
|
g.AddEdge(a, b)
|
|
}
|
|
return g, nil
|
|
}
|
|
|
|
// resolveHopWithContext resolves a single hop using NeighborGraph
|
|
// adjacency to the anchor. Returns nil when the hop cannot be
|
|
// disambiguated.
|
|
//
|
|
// exclude is a set of pubkeys to discard from the candidate pool
|
|
// (typically the prior hops already resolved on the path — a packet
|
|
// does not revisit a node).
|
|
//
|
|
// Behavior matrix:
|
|
// len(candidates) | anchor | graph | result
|
|
// 0 | — | — | nil
|
|
// 1 | — | — | candidates[0]
|
|
// >1 | "" or no graph|— | nil
|
|
// >1 | non-empty | set | unique adjacent candidate
|
|
// (or nil if 0 or >1 survive)
|
|
func resolveHopWithContext(hop string, anchor string, graph *NeighborGraph, idx prefixIndex, exclude map[string]struct{}) *string {
|
|
if idx == nil {
|
|
return nil
|
|
}
|
|
h := strings.ToLower(hop)
|
|
candidates := idx[h]
|
|
switch len(candidates) {
|
|
case 0:
|
|
return nil
|
|
case 1:
|
|
pk := candidates[0]
|
|
if _, skip := exclude[pk]; skip {
|
|
return nil
|
|
}
|
|
return &pk
|
|
}
|
|
if graph == nil || anchor == "" {
|
|
return nil
|
|
}
|
|
var match string
|
|
survivors := 0
|
|
for _, cand := range candidates {
|
|
if _, skip := exclude[cand]; skip {
|
|
continue
|
|
}
|
|
if graph.IsAdjacent(anchor, cand) {
|
|
survivors++
|
|
if survivors > 1 {
|
|
return nil
|
|
}
|
|
match = cand
|
|
}
|
|
}
|
|
if survivors == 1 {
|
|
return &match
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// resolvePathWithContext walks the hop list, anchoring hop 0 on
|
|
// fromPubkey (for ADVERTs) and each subsequent hop on the previous
|
|
// resolved hop. Previously-resolved pubkeys (plus the originator) are
|
|
// excluded from later candidate pools so the walk doesn't revisit a
|
|
// node. Returns a `[]*string` shape compatible with
|
|
// marshalResolvedPath (and the all-nil clobber-guard from PR #1548).
|
|
func resolvePathWithContext(hops []string, fromPubkey string, graph *NeighborGraph, idx prefixIndex) []*string {
|
|
if len(hops) == 0 {
|
|
return nil
|
|
}
|
|
out := make([]*string, len(hops))
|
|
if idx == nil {
|
|
return out
|
|
}
|
|
prevAnchor := strings.ToLower(fromPubkey)
|
|
seen := make(map[string]struct{}, len(hops)+1)
|
|
if prevAnchor != "" {
|
|
seen[prevAnchor] = struct{}{}
|
|
}
|
|
for i, hop := range hops {
|
|
r := resolveHopWithContext(hop, prevAnchor, graph, idx, seen)
|
|
out[i] = r
|
|
if r != nil {
|
|
lc := strings.ToLower(*r)
|
|
seen[lc] = struct{}{}
|
|
prevAnchor = lc
|
|
} else {
|
|
prevAnchor = ""
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// RefreshNeighborGraph loads the latest neighbor_edges snapshot and
|
|
// publishes it atomically. Called on startup and once per neighbor-
|
|
// edges builder tick (60s) alongside RefreshPrefixIndex.
|
|
func (s *Store) RefreshNeighborGraph() error {
|
|
g, err := loadNeighborGraph(s.db)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
s.neighborGraph.store(g)
|
|
return nil
|
|
}
|