mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 22:17:57 +00:00
## Summary `observations.observer_idx` points at the **implicit rowid** of `observers`, whose declared key is `id TEXT`. Implicit rowids are not stable: - a dump-and-reload (`sqlite3 .dump`, `iterdump`) renumbers them, which silently re-links observations to the wrong observer; - PostgreSQL has no rowid at all. This change makes the key explicit. Split out of #2396 at review's request, so SQLite installs get it independently of the Postgres work. ## What changes - **One-time rebuild, in the ingestor.** On first start, `observers` is rebuilt as `idx INTEGER PRIMARY KEY AUTOINCREMENT`, copying every rowid into `idx` in one transaction. It fails closed on unknown columns or NULL ids. - **Rollback-safe.** In SQLite an `INTEGER PRIMARY KEY` *is* the rowid, so `rowid == idx` forever. Older binaries that still join on `observers.rowid` keep working, so both upgrade and rollback are safe. - **No inherited orphans.** `AUTOINCREMENT` is seeded past `MAX(observations.observer_idx)`, so a new observer can never inherit a deleted observer's leftover rows. - Fresh installs create the new shape directly, and `AssertReady` requires `observers.idx` as the primary key (a plain column named `idx` does not pass). - **Observer writes update first.** With `AUTOINCREMENT`, SQLite advances `sqlite_sequence` for every INSERT it attempts, even one `ON CONFLICT` turns into an update. The per-packet observer upsert and the clock-skew upsert now run an `UPDATE` and fall back to the unchanged `INSERT ... ON CONFLICT` only for a new observer. So the sequence moves only when an observer is added, and the next new observer gets `max(idx) + 1`. - `packets_v` now has a single definition (`dbschema.EnsurePacketsView`). - Every server and ingestor query, the fixture capture script and the test schemas use `idx`. Guard tests fail if `observers.rowid` comes back. ## Operator impact Every SQLite install gets this on its next ingestor start, with no config change. The rebuild is one transaction over `observers`, which holds one row per observer (typically hundreds), so it's quick. **When the rebuild refuses.** It refuses rather than lose data in two cases, and then the ingestor stops on every start until the cause is fixed: - `observers` has a column CoreScope doesn't know (one added locally). The rebuild would drop it. Save its data, then drop or rename it: `ALTER TABLE observers DROP COLUMN <name>`. - A row has a NULL `id` (the legacy `TEXT PRIMARY KEY` allows one). Find it with `SELECT rowid, * FROM observers WHERE id IS NULL`, then delete or fix it. The error message names the column or the row count and these steps. Starting the previous version also works: nothing was changed, and `packets_v` stays readable for external SQLite readers in the meantime. **A refusal stops the server too, not only ingest.** The server checks the schema at startup (`AssertReady`) and exits when `observers.idx` is not the primary key, so the whole site is down until the cause above is fixed or the previous version is started again. Check for a locally added column or a NULL `id` before upgrading if you are unsure. **Ingest cost.** Measured with `BenchmarkUpsertObserverAtExisting` (`cmd/ingestor`): 20000 `UpsertObserverAt` calls over 500 existing observers, on a store opened by `OpenStore` with its own pragmas (WAL, `synchronous=FULL`), Ryzen 3700X in Docker. A plain upsert on the new table writes 5.23 WAL frames per call; update-first writes 4.23, one frame less. The pre-PR table was not measured. Time per call is 676 to 726 µs before and 665 to 668 µs after; it is bound by fsync, so the difference is within noise. ## Testing - `internal/dbschema/observers_idx_test.go`: the rebuild keeps every observation linked to the same observer, refuses an unknown column or a NULL id, never reuses an `idx` (including one only orphaned observations still reference), is idempotent, and handles the full legacy table. - `TestObserversIdxPreservesLinksOnE2EFixture`: on a copy of `test-fixtures/e2e-fixture.db`, 499/499 observations keep the same observer. - Refusals and rollback: a refused rebuild leaves the table, its rows, `foreign_keys` and a readable `packets_v` as they were; a failure after the old table is dropped rolls all of it back, leaves no `observers_new`, and the next start completes the rebuild (`TestObserversIdxRollsBackMidRebuild`, `TestPacketsViewFollowsObserversShape`). - Guards: `TestNoObserversRowidInSource` (ingestor) and `TestNoObserversRowidInServerSources` fail if a query joins on `observers.rowid` again. `TestFreshObserversMatchesRebuiltShape` pins that a fresh install and a rebuilt one end up identical. - `TestUpsertObserverAtLeavesSequenceAlone` and `TestRecordNaiveSkewLeavesSequenceAlone`: 1000 calls on an existing observer leave the sequence alone (on the previous head both moved it 1 -> 1001), and a new observer then gets `max(idx) + 1`. `TestUpdateObserverMatchesUpsert` and `TestUpdateNaiveSkewMatchesUpsert` check that the update path writes the same row as the upsert (COALESCE of empty fields, `last_seen` never going back, the `can_relay` tri-state, the 24 h skew counter). - `TestAssertReadyRequiresObserversIdxPrimaryKey`: the rebuilt shape passes; a plain `idx` column or none fails. `TestOpenStoreRefusesUnconvertibleObservers`: through `OpenStore`, an unknown column or a NULL id fails the open and leaves the table, its rows and `packets_v` as they were. - `go vet` and `go test` pass on all modules (`scripts/allmod.sh`), with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Suggested order, none of it urgent: 1. **#2408** — never lose rows during startup / a failed poll. 2. **#2407** — explicit `observers.idx` primary key. 3. **#2409** — content-hash migration moved into the ingestor. 4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not affected (zig already links them with 8 MiB thread stacks); it fixes native Alpine builds. After these land, #2396 (Postgres) will be rebased onto master. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
181 lines
7.5 KiB
Go
181 lines
7.5 KiB
Go
package main
|
|
|
|
import (
|
|
"database/sql"
|
|
"fmt"
|
|
"path/filepath"
|
|
"testing"
|
|
"time"
|
|
|
|
_ "github.com/mattn/go-sqlite3"
|
|
)
|
|
|
|
// createTestDBPathJSONNoResolvedPath builds a fixture that mirrors the LIVE
|
|
// deployment state after #1287: observations carry a path_json hop list but
|
|
// observations.resolved_path is NULL (the ingestor no longer writes it; relay
|
|
// data is persisted as aggregate neighbor_edges instead). A single repeater
|
|
// node whose public_key starts with hopPrefix lets the in-memory prefix map
|
|
// resolve that hop unambiguously to relayPubkey.
|
|
//
|
|
// The transmission's decoded_json is empty ({}), so relayPubkey is NOT an
|
|
// endpoint (pubKey/destPubKey/srcPubKey). The ONLY way it can enter
|
|
// s.byNode is via path_json → resolvePathForObs relay-hop resolution.
|
|
func createTestDBPathJSONNoResolvedPath(t *testing.T, relayPubkey, hopPrefix, firstSeen string) string {
|
|
t.Helper()
|
|
dir := t.TempDir()
|
|
dbPath := filepath.Join(dir, "test.db")
|
|
|
|
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer conn.Close()
|
|
|
|
exec := func(s string, args ...interface{}) {
|
|
if _, err := conn.Exec(s, args...); err != nil {
|
|
t.Fatalf("setup exec failed: %v\nSQL: %s", err, s)
|
|
}
|
|
}
|
|
|
|
// PREFLIGHT: async=true reason="test fixture: in-memory t.TempDir SQLite, never touches a real DB. Tables are CREATE-from-empty in a one-shot OpenDB call, not a schema migration over existing data."
|
|
exec(`CREATE TABLE transmissions (
|
|
id INTEGER PRIMARY KEY,
|
|
raw_hex TEXT, hash TEXT, first_seen TEXT,
|
|
route_type INTEGER, payload_type INTEGER, payload_version INTEGER,
|
|
decoded_json TEXT
|
|
)`)
|
|
// resolved_path column present (matches live schema) but left NULL.
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE observations (
|
|
id INTEGER PRIMARY KEY,
|
|
transmission_id INTEGER,
|
|
observer_id TEXT, observer_name TEXT,
|
|
direction TEXT, snr REAL, rssi REAL, score INTEGER,
|
|
path_json TEXT, timestamp TEXT,
|
|
raw_hex TEXT,
|
|
resolved_path TEXT
|
|
)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE observers (idx INTEGER PRIMARY KEY AUTOINCREMENT, id TEXT NOT NULL UNIQUE, name TEXT, iata TEXT, inactive INTEGER)`)
|
|
// Production nodes schema uses public_key (not pubkey) — getAllNodes /
|
|
// buildPrefixMap reads public_key, role, advert_count, first_seen.
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE nodes (
|
|
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
|
|
last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0
|
|
)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE TABLE schema_version (version INTEGER)`)
|
|
exec(`INSERT INTO schema_version (version) VALUES (1)`)
|
|
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
|
|
exec(`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`)
|
|
|
|
// Repeater node so canAppearInPath() admits it to the prefix map.
|
|
exec(`INSERT INTO nodes (public_key, name, role, advert_count) VALUES (?,?,?,?)`,
|
|
relayPubkey, "Relay One", "repeater", 10)
|
|
|
|
exec("INSERT INTO transmissions VALUES (?,?,?,?,0,4,1,?)",
|
|
1, "aa", "hashpjf_1", firstSeen, `{}`)
|
|
// resolved_path explicitly NULL; path_json carries the relay hop prefix.
|
|
exec("INSERT INTO observations (id, transmission_id, observer_id, observer_name, direction, snr, rssi, score, path_json, timestamp, raw_hex, resolved_path) VALUES (?,?,?,?,?,?,?,?,?,?,?,NULL)",
|
|
1, 1, "obs1", "Obs1", "RX", -10.0, -80.0, 5, fmt.Sprintf(`[%q]`, hopPrefix), firstSeen, "")
|
|
|
|
return dbPath
|
|
}
|
|
|
|
// TestLoadChunked_ResolvesRelayHopsFromPathJSON_WhenResolvedPathEmpty pins the
|
|
// fix for the "relay-node analytics empty after every restart" bug.
|
|
//
|
|
// On live, observations.resolved_path is 100% NULL (since #1287 the ingestor
|
|
// persists relay data as neighbor_edges, not per-observation resolved_path).
|
|
// The cold-load paths (Load / scanAndMergeChunk) indexed relay hops ONLY from
|
|
// resolved_path, so a relay node's path-hop attribution was never rebuilt on
|
|
// startup — it only re-accumulated from live traffic, collapsing the activity
|
|
// timeline to "just the hour the server restarted".
|
|
//
|
|
// The fix: when resolved_path is empty, fall back to resolving the hops from
|
|
// the persisted path_json using the in-memory prefix map + neighbor graph
|
|
// (exactly what the live ingest path already does), then index the relay hops.
|
|
func TestLoadChunked_ResolvesRelayHopsFromPathJSON_WhenResolvedPathEmpty(t *testing.T) {
|
|
relayPK := "aabbccddeeff00112233445566778899aabbccddeeff00112233445566778899"
|
|
hop := "aa" // 2-hex-char path hop; unique 2-char prefix of relayPK
|
|
|
|
ts := time.Now().UTC().Format(time.RFC3339)
|
|
dbPath := createTestDBPathJSONNoResolvedPath(t, relayPK, hop, ts)
|
|
|
|
db, err := OpenDB(dbPath)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer db.conn.Close()
|
|
|
|
if !db.hasResolvedPath {
|
|
t.Fatalf("setup: fixture should expose resolved_path column; hasResolvedPath=false")
|
|
}
|
|
|
|
store := NewPacketStore(db, &PacketStoreConfig{RetentionHours: 72})
|
|
// Empty graph is sufficient: a single prefix candidate resolves without
|
|
// neighbor-affinity disambiguation. Mirrors a freshly restarted server
|
|
// that has loaded its neighbor_edges snapshot before the packet load.
|
|
store.graph.Store(NewNeighborGraph())
|
|
|
|
if err := store.LoadChunked(0); err != nil {
|
|
t.Fatalf("LoadChunked: %v", err)
|
|
}
|
|
|
|
// The relay pubkey only reachable through path_json resolution must be
|
|
// indexed in byNode for the transmission.
|
|
if got := len(store.byNode[relayPK]); got != 1 {
|
|
t.Errorf("byNode[%s]: got %d transmissions, want 1 — cold load did not "+
|
|
"resolve relay hops from path_json when resolved_path was NULL "+
|
|
"(relay history lost on restart)", relayPK, got)
|
|
}
|
|
}
|
|
|
|
// TestLoadChunk_ResolvesRelayHopsFromPathJSON_WhenResolvedPathEmpty covers the
|
|
// background-window loader (loadBackgroundChunks → loadChunk), which on live
|
|
// loads everything older than hotStartupHours (24h) up to retentionHours
|
|
// (168h). Without the path_json fallback here, a relay node's analytics for
|
|
// the older 6 days would still vanish on every restart even with the hot
|
|
// window fixed.
|
|
func TestLoadChunk_ResolvesRelayHopsFromPathJSON_WhenResolvedPathEmpty(t *testing.T) {
|
|
relayPK := "ccddeeff00112233445566778899aabbccddeeff00112233445566778899aabb"
|
|
hop := "cc"
|
|
|
|
// Aged 48h so it falls in the background window, not the hot window.
|
|
aged := time.Now().UTC().Add(-48 * time.Hour).Format(time.RFC3339)
|
|
dbPath := createTestDBPathJSONNoResolvedPath(t, relayPK, hop, aged)
|
|
|
|
db, err := OpenDB(dbPath)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer db.conn.Close()
|
|
|
|
store := NewPacketStore(db, &PacketStoreConfig{
|
|
RetentionHours: 72,
|
|
HotStartupHours: 1, // hot load must NOT pick up the 48h-old row
|
|
})
|
|
store.graph.Store(NewNeighborGraph())
|
|
|
|
if err := store.LoadChunked(0); err != nil {
|
|
t.Fatalf("LoadChunked: %v", err)
|
|
}
|
|
if got := len(store.byNode[relayPK]); got != 0 {
|
|
t.Fatalf("setup: hot load unexpectedly picked up 48h-old row; "+
|
|
"byNode[relayPK]=%d (want 0) — test would not exercise loadChunk", got)
|
|
}
|
|
|
|
chunkStart := time.Now().UTC().Add(-72 * time.Hour)
|
|
chunkEnd := time.Now().UTC().Add(-1 * time.Hour)
|
|
if err := store.loadChunk(chunkStart, chunkEnd); err != nil {
|
|
t.Fatalf("loadChunk: %v", err)
|
|
}
|
|
|
|
if got := len(store.byNode[relayPK]); got != 1 {
|
|
t.Errorf("byNode[%s]: got %d transmissions, want 1 — background loadChunk "+
|
|
"did not resolve relay hops from path_json when resolved_path was NULL "+
|
|
"(relay history lost on restart for the older retention window)", relayPK, got)
|
|
}
|
|
}
|