mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-03 17:51:40 +00:00
938153dd92
## Problem A relay node's **activity timeline** — and its per-node `packetsToday` / observer counts — collapses to *"only the hour the server restarted"* after every restart. Before the restart the timeline shows only the node's own adverts (~1–2/hr); all of its relay activity piles into the single post-restart hour. ## Root cause All DB cold-load paths (`Load`, `loadChunk`, `scanAndMergeChunk`) index relay-hop attribution into `byNode` **only** from `observations.resolved_path`. But since #1287 the ingestor persists relay data as aggregate `neighbor_edges` and **never writes `resolved_path`** — it is `NULL` on every deployment (verified on a live DB: 0 of ~440k rows populated). So relay attribution is never reconstructed on startup; it only re-accumulates from live traffic (`IngestNew*`, which re-resolves from `path_json` + the neighbor graph), piling a relay node's whole history into the post-restart window. ## Fix Server read-side only — **no schema / ingestor / migration change**. When `resolved_path` is empty, re-resolve relay hops from the already-persisted `path_json` using the in-memory prefix map + neighbor graph (the same `resolvePathForObs` compute the live ingest path already runs). `main.go` now loads the persisted neighbor graph *before* the packet load so resolution has the graph available. Two correctness details worth a close look: 1. **Fetch the prefix-map/graph snapshot BEFORE opening each load cursor.** `getCachedNodesAndPM` issues its own DB query; doing so while a load cursor is open deadlocks on a single-connection SQLite pool (the test harness uses one). 2. **Index into `byNode` ONLY** — not the `resolved_path` / path-hop indexes. Those are cross-checked by `handleNodePaths` against the persisted `resolved_path` column (NULL here); populating them from an in-memory re-resolution would make that SQL confirmation fail and wrongly drop the tx from paths-through (#1352). ## Tests New coverage asserts a relay pubkey reachable *only* via `path_json` lands in `byNode` after a restart-style load, for both the hot-window (`LoadChunked`) and background-window (`loadChunk`) paths. Existing #1558 (`resolved_path`) and #1352 (paths-through) tests still pass. Full `cd cmd/server && go test ./...` is green under `-race`. ## Perf The fallback runs `resolvePathForObs` per observation with a non-empty `path_json` during cold load — the same per-packet compute the live ingest path already performs, so no new asymptotic cost. The prefix map + graph are snapshotted **once per load** (not per row); `getCachedNodesAndPM` is 30s-cached. In `loadChunk` the resolution runs in the existing lock-free scan and is accumulated locally, matching that function's "build local, merge under lock" design. ## Note on a pre-existing flaky test `TestDistanceConcurrentRequestsDuringBuildReturn202` is timing-fragile (fails ~1/15 on `master` without this change). It relies on the lazy distance build being slow because it's the first caller of `getCachedNodesAndPM` (cold cache). This PR pre-warms that cache during `Load`, narrowing the build window, so the test fails more often in **non-race** local runs. It passes reliably under `-race` (CI mode), where the build stays slow. Flagging in case you want to harden the test separately. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> Co-authored-by: openclaw-bot <bot@openclaw>
91 lines
3.1 KiB
Go
91 lines
3.1 KiB
Go
package main
|
|
|
|
import (
|
|
"database/sql"
|
|
"path/filepath"
|
|
"testing"
|
|
"time"
|
|
|
|
_ "modernc.org/sqlite"
|
|
)
|
|
|
|
// TestLoad_PanicsWhenGraphNotLoadedAndEdgesExist pins the startup-ordering
|
|
// invariant (munger R1 #2). Graph-load-before-packet-load is the entire
|
|
// premise of PR #1643's fix: without an in-memory neighbor graph, the
|
|
// path_json relay-hop fallback cannot resolve hops, so relay-node analytics
|
|
// history collapses. main.go currently does the right thing — but nothing
|
|
// asserts the ordering, so a future refactor could silently regress.
|
|
//
|
|
// Load() must panic when neighbor_edges has rows but s.graph.Load() returns
|
|
// nil. Fast-fail at startup beats silently-wrong attribution.
|
|
func TestLoad_PanicsWhenGraphNotLoadedAndEdgesExist(t *testing.T) {
|
|
dir := t.TempDir()
|
|
dbPath := filepath.Join(dir, "test.db")
|
|
|
|
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer rw.Close()
|
|
|
|
exec := func(s string, args ...interface{}) {
|
|
if _, err := rw.Exec(s, args...); err != nil {
|
|
t.Fatalf("setup exec failed: %v\nSQL: %s", err, s)
|
|
}
|
|
}
|
|
|
|
// Minimal CoreScope schema. PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE transmissions (
|
|
id INTEGER PRIMARY KEY,
|
|
raw_hex TEXT, hash TEXT, first_seen TEXT,
|
|
route_type INTEGER, payload_type INTEGER, payload_version INTEGER,
|
|
decoded_json TEXT
|
|
)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE observations (
|
|
id INTEGER PRIMARY KEY, transmission_id INTEGER,
|
|
observer_id TEXT, observer_name TEXT,
|
|
direction TEXT, snr REAL, rssi REAL, score INTEGER,
|
|
path_json TEXT, timestamp TEXT, raw_hex TEXT, resolved_path TEXT
|
|
)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE nodes (
|
|
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
|
|
last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0
|
|
)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE schema_version (version INTEGER)`)
|
|
exec(`INSERT INTO schema_version (version) VALUES (1)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE neighbor_edges (
|
|
node_a TEXT NOT NULL,
|
|
node_b TEXT NOT NULL,
|
|
count INTEGER DEFAULT 1,
|
|
last_seen TEXT,
|
|
PRIMARY KEY (node_a, node_b)
|
|
)`)
|
|
now := time.Now().UTC().Format(time.RFC3339)
|
|
exec(`INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, ?, ?)`,
|
|
"aaa", "bbb", 5, now)
|
|
|
|
d, err := OpenDB(dbPath)
|
|
if err != nil {
|
|
t.Fatalf("OpenDB: %v", err)
|
|
}
|
|
defer d.conn.Close()
|
|
|
|
// Deliberately DO NOT call store.graph.Store(...). s.graph.Load() returns
|
|
// nil → the bug condition the invariant guard must catch.
|
|
store := NewPacketStore(d, &PacketStoreConfig{RetentionHours: 72})
|
|
|
|
defer func() {
|
|
r := recover()
|
|
if r == nil {
|
|
t.Fatalf("Load() must panic when neighbor_edges has rows but graph is nil; got no panic")
|
|
}
|
|
}()
|
|
_ = store.Load()
|
|
}
|