Files
meshcore-analyzer/cmd/ingestor/resolved_path.go
T
Kpa-clawbot 3feb97f16f fix(ingestor): write resolved_path on new observations (regression from #1289) (#1548)
# fix(ingestor): write resolved_path on new observations (full restore —
closes #1547 + #1560)

Fixes #1547. Closes #1560.

## Root cause
PR #1289 (the "ingestor owns the neighbor graph; server is read-only"
refactor, ~2026-05-21) moved the neighbor graph + schema writes to the
ingestor, and as a side-effect removed the server-side writer that
populated `observations.resolved_path` AND the context-aware
`pm.resolveWithContext` that disambiguated 1-byte prefix collisions.
Result: every observation inserted after the deploy has `resolved_path =
NULL` (3.1M/6.3M NULL on staging; 100% NULL on fresh deploys; symptom on
Cascadia: hops fail to resolve because the small-mesh client-side
fallback breaks on prefix collisions).

## Full restore
This PR resolves both single-byte and multi-byte prefix paths.
Single-byte disambiguation uses NeighborGraph adjacency and ADVERT
`from_pubkey` anchoring, ported from pre-#1289 `pm.resolveWithContext`
logic (last good at cmd/server/store.go @ commit 450236d5) and the #1144
/ #1352 fixes.

New file `cmd/ingestor/path_resolver.go`:
- `NeighborGraph` + `neighborGraphHolder` — in-memory adjacency
snapshot, atomic-published.
- `loadNeighborGraph(db)` — one-shot SELECT from `neighbor_edges`.
- `resolveHopWithContext(hop, anchor, graph, idx, exclude) *string` —
single-hop, tier-1 disambiguator.
- `resolvePathWithContext(hops, fromPubkey, graph, idx) []*string` —
walks the path, anchoring hop 0 on `from_pubkey` (ADVERTs) and each
subsequent hop on the previous resolved hop, excluding already-resolved
pubkeys.
- `Store.RefreshNeighborGraph()` — called on warm-up and every 60s tick
in the neighbor-edges builder alongside `RefreshPrefixIndex`.

Existing file `cmd/ingestor/resolved_path.go` (PR #1547 base) is
untouched: `resolvePath` + `marshalResolvedPath` + the all-nil →
empty-string clobber-guard contract are preserved verbatim.

`cmd/ingestor/db.go` — `InsertTransmission` now calls
`resolvePathWithContext` instead of the naive `resolvePath`.

## Algorithm (per hop)
1. Look up candidate pubkeys by prefix-match (existing `prefixIndex`).
2. `len==0 → nil`; `len==1 → that pubkey`.
3. `len>1` → filter by `NeighborGraph` adjacency to the anchor. Anchor
is `from_pubkey` for hop 0 on ADVERTs, the previous resolved hop
otherwise. Exactly 1 surviving candidate → use it; else nil.
4. Previously resolved hops (and the originator) are excluded from
downstream candidate pools — a packet does not revisit a node.

Tier-2/3/4 from pre-#1289 (geo proximity, GPS preference,
observation-count fallback) are intentionally NOT ported — those were
noisy in practice and belong in a separate enhancement, not in this
regression restore.

## Out of scope
- The ~3.1M existing NULL rows from the regression window. Filed as a
follow-up backfill task — too risky to bundle here (touches a 6M-row
table).
- The dead-flag bug #1546 — separate concern.

## TDD red → green
- Red commit `80b0f476` — adds five new context-resolver tests; stub
`resolvePathWithContext` falls back to naive `resolvePath`. CI run
26946935615 → **failure** with assertion errors on the three collision
tests (`TestResolveHopWithContext_OneByteCollision_AdjacencyResolves`,
`TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode`,
`TestResolvePathWithContext_AdvertAnchoring`); the two regression tests
(multi-byte still works + all-nil contract) stayed green.
- Green commit `7b4950ce` — real algorithm + InsertTransmission wiring +
RefreshNeighborGraph in the builder tick. All five new tests pass;
original four `resolved_path` tests stay green.

## Verification
- `go test -race ./cmd/ingestor/...` for the 11 affected tests — pass.
- `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh
origin/master` — exit 0 (all gates clean).
- PII grep on body + diff: clean.

Tested with: existing `TestInsertTransmissionWritesResolvedPath` +
`TestInsertTransmissionDoesNotClobberResolvedPathOnAllNil` (PR #1547
base) plus the new collision-resolution suite:
- `TestResolveHopWithContext_OneByteCollision_AdjacencyResolves` —
3-of-5 nodes share `0x5c`, chain A↔B↔C↔D↔E; anchored on A, hop `5c` → B.
- `TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode` — path
`[5c, 5c]` from_node A → `[B, C]`.
- `TestResolveHopWithContext_NoAdjacencyContext_ReturnsNil` — 3
ambiguous candidates, no anchor / non-adjacent anchor → nil.
- `TestResolvePathWithContext_AdvertAnchoring` — ADVERT,
`from_pubkey=A`, path `[5c]` → only-adjacent neighbor B.
- `TestResolvePathWithContext_RegressionMultiByteStillWorks` —
unique-prefix path with no graph context still resolves.
- `TestResolvePathWithContext_AllNilContractPreserved` — unresolvable
path → `marshalResolvedPath==""` (clobber-guard from PR #1548
untouched).

## Browser-validated
N/A — backend-only change. Frontend already handles populated
`resolved_path` via `getResolvedPath` in `cmd/server/db.go` and
`public/packets.js`.

## Round-1 fixes addressed
- **MUST-FIX #1 (data-loss clobber on all-nil resolution):** when every
hop fails to resolve, `marshalResolvedPath` returns `""` instead of
`"[null,null,...]"`, so `nilIfEmpty` → SQL NULL and the
`COALESCE(excluded.resolved_path, resolved_path)` UPSERT preserves any
previously stored good value on re-ingest. Regression test asserts:
insert a transmission, observe `resolved_path` populated, wipe the
prefix index, re-ingest the same packet, assert the existing
`resolved_path` is unchanged.

---------

Co-authored-by: corescope-bot <bot@corescope>
Co-authored-by: openclaw-bot <bot@openclaw>
Co-authored-by: openclaw-bot <bot@openclaw.local>
2026-06-04 07:35:13 -07:00

114 lines
3.2 KiB
Go

package main
import (
"encoding/json"
"strings"
"sync/atomic"
)
// Issue #1547 — resolved_path writer (ingestor-owned).
//
// Per the #1283 refactor (server is read-only; ingestor owns the
// neighbor graph + node directory), the writer that populated
// `observations.resolved_path` must live here in the ingestor. PR #1289
// removed the server-side writer without porting it — this restores it.
//
// Approach:
// - `resolvePath` is a pure function: hop prefixes → full pubkeys
// using the in-memory prefix index built from `nodes.public_key`.
// - Unique-prefix hops resolve to the full pubkey; ambiguous or
// unknown hops resolve to `nil`. The output shape is `[]*string`
// (with nulls for unresolved positions) — the JSON serialization
// matches what the server's `unmarshalResolvedPath` /
// frontend `getResolvedPath` already consume.
// - The prefix index is rebuilt on startup and once per neighbor-
// builder tick (60s) so new nodes start resolving within a minute
// without blocking the MQTT ingest path.
// resolvePath maps each hop prefix to a full pubkey when the index
// has exactly one candidate; returns nil at that position otherwise.
// Returns nil for empty/no hops.
func resolvePath(hops []string, idx prefixIndex) []*string {
if len(hops) == 0 {
return nil
}
out := make([]*string, len(hops))
if idx == nil {
return out
}
for i, hop := range hops {
h := strings.ToLower(hop)
candidates := idx[h]
if len(candidates) == 1 {
pk := candidates[0]
out[i] = &pk
}
}
return out
}
// marshalResolvedPath JSON-encodes a resolved path. Returns "" when
// the input is empty OR when every element is nil (writer treats "" as
// SQL NULL).
//
// The all-nil case matters because of the UPSERT in InsertTransmission:
//
// resolved_path = COALESCE(excluded.resolved_path, resolved_path)
//
// If we emitted "[null,null]" here, nilIfEmpty() would let it through
// as a non-NULL string and the COALESCE would OVERWRITE a previously
// stored good resolved_path on re-ingest. Returning "" lets nilIfEmpty
// produce SQL NULL so the COALESCE falls through to the existing value.
// See issue #1547 / PR #1548 reviewer findings.
func marshalResolvedPath(rp []*string) string {
if len(rp) == 0 {
return ""
}
allNil := true
for _, p := range rp {
if p != nil {
allNil = false
break
}
}
if allNil {
return ""
}
b, err := json.Marshal(rp)
if err != nil {
return ""
}
return string(b)
}
// prefixIdxHolder caches the prefix index for the InsertTransmission
// hot path. atomic.Value lets the 60s rebuild happen without a lock on
// the read side.
type prefixIdxHolder struct {
v atomic.Value // holds prefixIndex
}
func (h *prefixIdxHolder) load() prefixIndex {
if v := h.v.Load(); v != nil {
return v.(prefixIndex)
}
return nil
}
func (h *prefixIdxHolder) store(idx prefixIndex) {
h.v.Store(idx)
}
// RefreshPrefixIndex rebuilds the in-memory prefix index from the
// nodes table and publishes it atomically. Called on startup and from
// the neighbor-edges builder tick (60s) so new nodes become resolvable
// without per-insert DB scans.
func (s *Store) RefreshPrefixIndex() error {
idx, err := buildPrefixIndex(s.db)
if err != nil {
return err
}
s.prefixIdx.store(idx)
return nil
}