mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 03:17:04 +00:00
## Summary Two pre-existing ways a running server **permanently** missed data. The database was correct; the server's in-memory store was not, and stayed wrong until the next restart. Found while comparing several server replicas side by side in #2396: two replicas that restarted mid-ingest each lacked one transmission and its 12 observations. Split out of #2396 at review's request, because it affects every SQLite install. ## The bugs 1. **Startup window.** The poller took its starting cursors from a fresh `MAX(id)` *after* the startup load finished. A row committed between the load's last chunk snapshot and that query ended up in neither the store nor any later poll. Its later observations were then dropped as orphans. 2. **Failed poll.** A failed observation read still advanced the cursor to its upper bound, skipping up to 500 observations for good. ## The fix - `NewPacketStore` records the transmission and observation high-water marks **before** the load, as the advert-evidence and rewrite feeds already did. - The loaded max transmission id stays as a floor. - The loaded max observation id no longer is one: a chunk-1 transmission can gain an observation whose id is *below* one that chunk 2 loaded. - Rows read twice are deduplicated and broadcast at most once. - A failed read keeps the cursor and retries on the next tick, logging once per failure streak. - Comments and a test pin why an observation of a transmission whose chunk has not merged yet cannot be skipped: the poller's evidence refresh waits on that chunk's lock. ## Performance - **Startup:** two extra indexed `MAX(id)` queries, which are primary-key lookups. - **Poll loop:** unchanged complexity. The only new per-tick work is a dedupe check on rows read twice, a map lookup by id (O(1)). Re-reads happen only for rows in the startup window, which is bounded by what was committed during the load, and on a retry after a failed read. ## Testing `cmd/server/startup_poll_gap_test.go` seeds the ingestor's own DDL plus `dbschema.Apply`: | Test | What it pins | |------|--------------| | `TestStartupLoadPollerGap_SQLite` | A transmission and observations committed between the load's snapshot and the poller's start reach the store | | `TestStartupLoadPollerGapMultiChunk_SQLite` | Same, across 3 chunks, including an observation id below the max loaded one | | `TestStartupObsDuringChunkMerge_SQLite` | An observation of a transmission whose chunk has not merged yet is not skipped | | `TestPollerRetriesFailedRead_SQLite` | A failed observation read does not advance the cursor | **Red/green:** against master's `store.go`, `websocket.go`, `chunked_load.go` and `db.go`, the first, second and fourth tests fail with the rows missing, e.g.: ``` tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever) obs 3, committed while the observation read failed, never reached the store: the cursor passed it ``` The third depends on test hooks this change adds, so it cannot run against the old code. `go vet` and `go test` pass on all modules, with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Recommended **priority** (not a dependency): 1. **#2410** — musl 8 MiB thread stacks. A public, unauthenticated GET can SIGSEGV the whole server on the shipped image, so merge this first. 2. **#2408** — never lose rows during startup / a failed poll. 3. **#2407** — explicit `observers.idx` primary key. 4. **#2409** — content-hash migration moved into the ingestor. After these land, #2396 (Postgres) will be rebased onto master. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
665 lines
23 KiB
Go
665 lines
23 KiB
Go
package main
|
|
|
|
import (
|
|
"database/sql"
|
|
"encoding/json"
|
|
"fmt"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
|
|
_ "github.com/mattn/go-sqlite3"
|
|
"github.com/meshcore-analyzer/dbschema"
|
|
)
|
|
|
|
// A transmission or observation the ingestor commits after the startup load's
|
|
// last chunk has read its snapshot, but before the poller initialises its
|
|
// cursor, must still reach the in-memory store. The poller seeds its cursor
|
|
// from a fresh MAX(id) taken when it starts (websocket.go Poller.Start), so a
|
|
// row committed in that window is above everything loaded yet not above the
|
|
// cursor: neither the load nor the poll ever reads it.
|
|
//
|
|
// The window is opened deterministically with OnChunkLoaded, which fires
|
|
// after a chunk's rows have been read and closed. With one chunk that is the
|
|
// last snapshot the load takes.
|
|
|
|
func TestStartupLoadPollerGap_SQLite(t *testing.T) {
|
|
db, rw := openGapSQLite(t)
|
|
runStartupLoadPollerGap(t, db, rw)
|
|
}
|
|
|
|
func runStartupLoadPollerGap(t *testing.T, db *DB, rw *sql.DB) {
|
|
t.Helper()
|
|
base := time.Now().UTC().Add(-10 * time.Minute).Truncate(time.Second)
|
|
iso := func(d time.Duration) string { return base.Add(d).Format("2006-01-02T15:04:05.000Z") }
|
|
mustExec := func(q string, args ...any) {
|
|
t.Helper()
|
|
if _, err := rw.Exec(q, args...); err != nil {
|
|
t.Fatalf("seed: %v\n%s", err, q)
|
|
}
|
|
}
|
|
addTx := func(id int) {
|
|
mustExec(`INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json, last_seen)
|
|
VALUES (?, ?, ?, ?, 1, 5, 1, '{"type":"GRP_TXT"}', ?)`,
|
|
id, fmt.Sprintf("11%02dAABBCCDD", id), gapHash(id), iso(time.Duration(id)*time.Second), base.Add(time.Duration(id)*time.Second).Unix())
|
|
}
|
|
addObs := func(id, tx, observer int) {
|
|
mustExec(`INSERT INTO observations (id, transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp)
|
|
VALUES (?, ?, ?, 'rx', 5.5, -90, 1, ?, ?)`,
|
|
id, tx, observer, fmt.Sprintf(`["%02X"]`, id), base.Add(time.Duration(id)*time.Second).Unix())
|
|
}
|
|
for idx := 1; idx <= 3; idx++ {
|
|
mustExec(`INSERT INTO observers (rowid, id, name, iata, last_seen, first_seen) VALUES (?, ?, ?, 'SJC', ?, ?)`,
|
|
idx, fmt.Sprintf("%064X", idx), fmt.Sprintf("Obs-%d", idx), iso(0), iso(-time.Hour))
|
|
}
|
|
// Committed before the load.
|
|
addTx(1)
|
|
addObs(1, 1, 1)
|
|
addTx(2)
|
|
addObs(2, 2, 1)
|
|
|
|
store := NewPacketStore(db, &PacketStoreConfig{})
|
|
fired := false
|
|
store.OnChunkLoaded(func(int, int) {
|
|
if fired {
|
|
return
|
|
}
|
|
fired = true
|
|
// Committed after the (only) chunk read its snapshot, before the
|
|
// poller starts: a new transmission with one observation, and a
|
|
// second observation of an already loaded transmission.
|
|
addTx(3)
|
|
addObs(3, 3, 1)
|
|
addObs(4, 1, 2)
|
|
})
|
|
if err := store.RunStartupLoad(100); err != nil {
|
|
t.Fatalf("RunStartupLoad: %v", err)
|
|
}
|
|
if !fired {
|
|
t.Fatal("OnChunkLoaded never fired")
|
|
}
|
|
|
|
loaded := loadedObsPaths(store)
|
|
pr := startGapPoller(t, db, store)
|
|
|
|
time.Sleep(80 * time.Millisecond)
|
|
// Committed after the poller started: a later observation of tx 3 (the
|
|
// 12 observations of the lost transmission in the field), and a control
|
|
// transmission proving the poller itself works.
|
|
addObs(5, 3, 3)
|
|
addTx(4)
|
|
addObs(6, 4, 1)
|
|
|
|
has := func() (tx3, tx4 bool, obs3, obs4, obs5 bool) {
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
_, tx3 = store.byHash[gapHash(3)]
|
|
_, tx4 = store.byHash[gapHash(4)]
|
|
_, obs3 = store.byObsID[3]
|
|
_, obs4 = store.byObsID[4]
|
|
_, obs5 = store.byObsID[5]
|
|
return
|
|
}
|
|
deadline := time.Now().Add(3 * time.Second)
|
|
for time.Now().Before(deadline) {
|
|
if _, tx4, _, _, _ := has(); tx4 {
|
|
break
|
|
}
|
|
time.Sleep(20 * time.Millisecond)
|
|
}
|
|
time.Sleep(100 * time.Millisecond) // a few more ticks for stragglers
|
|
tx3, tx4, obs3, obs4, obs5 := has()
|
|
if !tx4 {
|
|
t.Fatal("control: tx 4, committed after the poller started, never reached the store; the poller is not running")
|
|
}
|
|
if !tx3 {
|
|
t.Errorf("tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever)")
|
|
}
|
|
if !obs3 || !obs5 {
|
|
t.Errorf("observations of tx 3 missing: obs3=%v obs5=%v", obs3, obs5)
|
|
}
|
|
if !obs4 {
|
|
t.Errorf("obs 4 (new observation of already-loaded tx 1, committed in the same window) is missing from the store")
|
|
}
|
|
pr.stop()
|
|
assertStoreMatchesDB(t, store, rw)
|
|
assertNoDuplicateBroadcasts(t, pr.drain(), loaded)
|
|
assertReplayIdempotent(t, store, rw)
|
|
}
|
|
|
|
func gapHash(id int) string { return fmt.Sprintf("feed%012x", id) }
|
|
|
|
// A chunk-1 transmission can gain an observation after chunk 1 has read its
|
|
// snapshot whose id is BELOW an observation a later chunk loads (that one
|
|
// belongs to a chunk-2 transmission and was committed just after it). Using
|
|
// the store's highest loaded observation id as the poller's floor would then
|
|
// skip it. Three chunks of two: {1,2} {3,4} {5}; the last is short, so its
|
|
// callback runs after the load's final snapshot.
|
|
|
|
func TestStartupLoadPollerGapMultiChunk_SQLite(t *testing.T) {
|
|
db, rw := openGapSQLite(t)
|
|
runStartupLoadPollerGapMultiChunk(t, db, rw)
|
|
}
|
|
|
|
func openGapSQLite(t *testing.T) (*DB, *sql.DB) {
|
|
t.Helper()
|
|
path := filepath.Join(t.TempDir(), "gap.db")
|
|
rw, err := sql.Open("sqlite3", path+"?_journal_mode=WAL&_foreign_keys=on")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
t.Cleanup(func() { rw.Close() })
|
|
rw.SetMaxOpenConns(1)
|
|
// The tables the load and the poller read, as the ingestor creates them
|
|
// (cmd/ingestor/db.go), then the server-side migrations it applies on top.
|
|
if _, err := rw.Exec(gapBaseSchema); err != nil {
|
|
t.Fatalf("create base schema: %v", err)
|
|
}
|
|
if err := dbschema.Apply(rw, nil); err != nil {
|
|
t.Fatalf("dbschema.Apply: %v", err)
|
|
}
|
|
db, err := OpenDB(path)
|
|
if err != nil {
|
|
t.Fatalf("OpenDB: %v", err)
|
|
}
|
|
t.Cleanup(func() { db.conn.Close() })
|
|
return db, rw
|
|
}
|
|
|
|
// gapBaseSchema is the ingestor's DDL for the tables these tests seed. Observers
|
|
// are inserted by rowid: observations.observer_idx refers to it.
|
|
const gapBaseSchema = `
|
|
CREATE TABLE nodes (
|
|
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
|
|
last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0,
|
|
battery_mv INTEGER, temperature_c REAL, foreign_advert INTEGER DEFAULT 0
|
|
);
|
|
CREATE TABLE observers (
|
|
id TEXT PRIMARY KEY, name TEXT, iata TEXT, last_seen TEXT, first_seen TEXT,
|
|
packet_count INTEGER DEFAULT 0, model TEXT, firmware TEXT, client_version TEXT,
|
|
radio TEXT, battery_mv INTEGER, uptime_secs INTEGER, noise_floor REAL,
|
|
inactive INTEGER DEFAULT 0, last_packet_at TEXT DEFAULT NULL,
|
|
clock_skew_seconds INTEGER DEFAULT NULL, clock_skew_count_24h INTEGER DEFAULT 0,
|
|
clock_last_naive_at TEXT DEFAULT NULL, can_relay INTEGER DEFAULT 1,
|
|
can_relay_seen INTEGER DEFAULT 0
|
|
);
|
|
CREATE TABLE transmissions (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT, raw_hex TEXT NOT NULL,
|
|
hash TEXT NOT NULL UNIQUE, first_seen TEXT NOT NULL, route_type INTEGER,
|
|
payload_type INTEGER, payload_version INTEGER, decoded_json TEXT,
|
|
from_pubkey TEXT, last_seen INTEGER NOT NULL DEFAULT 0,
|
|
created_at TEXT DEFAULT (datetime('now')), code1 TEXT, code2 TEXT
|
|
);
|
|
CREATE TABLE observations (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
transmission_id INTEGER NOT NULL REFERENCES transmissions(id),
|
|
observer_idx INTEGER, direction TEXT, snr REAL, rssi REAL, score INTEGER,
|
|
path_json TEXT, timestamp INTEGER NOT NULL
|
|
);
|
|
CREATE INDEX idx_observations_transmission_id ON observations(transmission_id);
|
|
CREATE INDEX idx_observations_observer_idx ON observations(observer_idx);
|
|
CREATE INDEX idx_observations_timestamp ON observations(timestamp);
|
|
CREATE TABLE schema_version (version INTEGER);
|
|
`
|
|
|
|
func runStartupLoadPollerGapMultiChunk(t *testing.T, db *DB, rw *sql.DB) {
|
|
t.Helper()
|
|
base := time.Now().UTC().Add(-10 * time.Minute).Truncate(time.Second)
|
|
iso := func(d time.Duration) string { return base.Add(d).Format("2006-01-02T15:04:05.000Z") }
|
|
mustExec := func(q string, args ...any) {
|
|
t.Helper()
|
|
if _, err := rw.Exec(q, args...); err != nil {
|
|
t.Fatalf("seed: %v\n%s", err, q)
|
|
}
|
|
}
|
|
addTx := func(id int) {
|
|
mustExec(`INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json, last_seen)
|
|
VALUES (?, ?, ?, ?, 1, 5, 1, '{"type":"GRP_TXT"}', ?)`,
|
|
id, fmt.Sprintf("11%02dAABBCCDD", id), gapHash(id), iso(time.Duration(id)*time.Second), base.Add(time.Duration(id)*time.Second).Unix())
|
|
}
|
|
// path_json is unique per observation, so none merges with another
|
|
// (dedup is by observer+path) and DB and store counts must agree.
|
|
addObs := func(id, tx, observer int) {
|
|
mustExec(`INSERT INTO observations (id, transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp)
|
|
VALUES (?, ?, ?, 'rx', 5.5, -90, 1, ?, ?)`,
|
|
id, tx, observer, fmt.Sprintf(`["%02X"]`, id), base.Add(time.Duration(id)*time.Second).Unix())
|
|
}
|
|
for idx := 1; idx <= 3; idx++ {
|
|
mustExec(`INSERT INTO observers (rowid, id, name, iata, last_seen, first_seen) VALUES (?, ?, ?, 'SJC', ?, ?)`,
|
|
idx, fmt.Sprintf("%064X", idx), fmt.Sprintf("Obs-%d", idx), iso(0), iso(-time.Hour))
|
|
}
|
|
for id := 1; id <= 5; id++ {
|
|
addTx(id)
|
|
addObs(id, id, 1)
|
|
}
|
|
|
|
store := NewPacketStore(db, &PacketStoreConfig{})
|
|
chunk := 0
|
|
store.OnChunkLoaded(func(int, int) {
|
|
chunk++
|
|
switch chunk {
|
|
case 1:
|
|
// Between chunks 1 and 2: obs 10 of chunk-1 tx 1, then obs 11
|
|
// of tx 3, which chunk 2 loads. The store's max obs id becomes
|
|
// 11 although 10 was never loaded.
|
|
addObs(10, 1, 2)
|
|
addObs(11, 3, 2)
|
|
case 3:
|
|
// After the final snapshot: a new transmission with one
|
|
// observation, and a new observation of loaded tx 2.
|
|
addTx(6)
|
|
addObs(12, 6, 1)
|
|
addObs(13, 2, 3)
|
|
}
|
|
})
|
|
if err := store.RunStartupLoad(2); err != nil {
|
|
t.Fatalf("RunStartupLoad: %v", err)
|
|
}
|
|
if chunk != 3 {
|
|
t.Fatalf("expected 3 chunks, got %d", chunk)
|
|
}
|
|
loaded := loadedObsPaths(store)
|
|
if _, ok := loaded[11]; !ok {
|
|
t.Fatal("setup: obs 11 should have been loaded by chunk 2")
|
|
}
|
|
if _, ok := loaded[10]; ok {
|
|
t.Fatal("setup: obs 10 should not have been loaded (committed after chunk 1)")
|
|
}
|
|
|
|
pr := startGapPoller(t, db, store)
|
|
time.Sleep(80 * time.Millisecond)
|
|
// After the poller started: a later observation of tx 6 and a control.
|
|
addObs(14, 6, 2)
|
|
addTx(7)
|
|
addObs(15, 7, 1)
|
|
|
|
deadline := time.Now().Add(3 * time.Second)
|
|
for time.Now().Before(deadline) {
|
|
store.mu.RLock()
|
|
_, ok := store.byObsID[15]
|
|
store.mu.RUnlock()
|
|
if ok {
|
|
break
|
|
}
|
|
time.Sleep(20 * time.Millisecond)
|
|
}
|
|
time.Sleep(100 * time.Millisecond) // a few more ticks for stragglers
|
|
pr.stop()
|
|
|
|
store.mu.RLock()
|
|
_, tx6 := store.byHash[gapHash(6)]
|
|
_, tx7 := store.byHash[gapHash(7)]
|
|
var missing []int
|
|
for _, id := range []int{10, 12, 13, 14, 15} {
|
|
if _, ok := store.byObsID[id]; !ok {
|
|
missing = append(missing, id)
|
|
}
|
|
}
|
|
store.mu.RUnlock()
|
|
if !tx7 {
|
|
t.Fatal("control: tx 7, committed after the poller started, never reached the store; the poller is not running")
|
|
}
|
|
if !tx6 {
|
|
t.Errorf("tx 6, committed after the load's final snapshot, is missing from the store")
|
|
}
|
|
if len(missing) > 0 {
|
|
t.Errorf("observations missing from the store: %v (10 is below the max loaded obs id 11)", missing)
|
|
}
|
|
assertStoreMatchesDB(t, store, rw)
|
|
assertNoDuplicateBroadcasts(t, pr.drain(), loaded)
|
|
assertReplayIdempotent(t, store, rw)
|
|
}
|
|
|
|
type gapPoller struct {
|
|
client *Client
|
|
stop func()
|
|
}
|
|
|
|
// startGapPoller runs a poller over store with one capturing hub client.
|
|
// stop waits for Start to return, so no tick races the assertions after it.
|
|
func startGapPoller(t *testing.T, db *DB, store *PacketStore) *gapPoller {
|
|
t.Helper()
|
|
hub := NewHub()
|
|
c := &Client{send: make(chan []byte, 1024), release: func() {}}
|
|
hub.mu.Lock()
|
|
hub.clients[c] = true
|
|
hub.mu.Unlock()
|
|
poller := NewPoller(db, hub, 20*time.Millisecond)
|
|
poller.store = store
|
|
done := make(chan struct{})
|
|
go func() {
|
|
poller.Start()
|
|
close(done)
|
|
}()
|
|
var once sync.Once
|
|
stop := func() {
|
|
once.Do(func() {
|
|
poller.Stop()
|
|
<-done
|
|
})
|
|
}
|
|
t.Cleanup(stop)
|
|
return &gapPoller{client: c, stop: stop}
|
|
}
|
|
|
|
// drain returns every message broadcast so far.
|
|
func (p *gapPoller) drain() [][]byte {
|
|
var out [][]byte
|
|
for {
|
|
select {
|
|
case m := <-p.client.send:
|
|
out = append(out, m)
|
|
default:
|
|
return out
|
|
}
|
|
}
|
|
}
|
|
|
|
// loadedObsPaths maps each observation id the store holds to its path_json.
|
|
func loadedObsPaths(store *PacketStore) map[int]string {
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
out := make(map[int]string, len(store.byObsID))
|
|
for id, o := range store.byObsID {
|
|
out[id] = o.PathJSON
|
|
}
|
|
return out
|
|
}
|
|
|
|
// assertStoreMatchesDB checks the store holds exactly the DB's transmissions
|
|
// and observations: nothing lost, nothing counted twice.
|
|
// notLoaded names transmissions outside the loaded window; they and their
|
|
// observations are left out of the DB side.
|
|
func assertStoreMatchesDB(t *testing.T, store *PacketStore, rw *sql.DB, notLoaded ...int) {
|
|
t.Helper()
|
|
txWhere, obsWhere := "", ""
|
|
if len(notLoaded) > 0 {
|
|
ids := strings.Trim(strings.Join(strings.Fields(fmt.Sprint(notLoaded)), ","), "[]")
|
|
txWhere = " WHERE id NOT IN (" + ids + ")"
|
|
obsWhere = " WHERE transmission_id NOT IN (" + ids + ")"
|
|
}
|
|
var dbTx, dbObs int
|
|
if err := rw.QueryRow(`SELECT COUNT(*) FROM transmissions` + txWhere).Scan(&dbTx); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := rw.QueryRow(`SELECT COUNT(*) FROM observations` + obsWhere).Scan(&dbObs); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
sumObs, sumCount := 0, 0
|
|
for _, tx := range store.packets {
|
|
sumObs += len(tx.Observations)
|
|
sumCount += tx.ObservationCount
|
|
}
|
|
if len(store.packets) != dbTx || len(store.byHash) != dbTx {
|
|
t.Errorf("store transmissions: packets=%d byHash=%d, DB has %d", len(store.packets), len(store.byHash), dbTx)
|
|
}
|
|
if store.totalObs != dbObs || len(store.byObsID) != dbObs || sumObs != dbObs || sumCount != dbObs {
|
|
t.Errorf("store observations: totalObs=%d byObsID=%d attached=%d ObservationCount=%d, DB has %d",
|
|
store.totalObs, len(store.byObsID), sumObs, sumCount, dbObs)
|
|
}
|
|
}
|
|
|
|
// assertNoDuplicateBroadcasts checks the poller's catch-up over rows it
|
|
// re-read did not re-announce one the load already held, nor any row twice.
|
|
func assertNoDuplicateBroadcasts(t *testing.T, msgs [][]byte, loaded map[int]string) {
|
|
t.Helper()
|
|
loadedPaths := make(map[string]int, len(loaded))
|
|
for id, p := range loaded {
|
|
loadedPaths[p] = id
|
|
}
|
|
seen := make(map[string]bool)
|
|
for _, raw := range msgs {
|
|
var m struct {
|
|
Type string `json:"type"`
|
|
Data struct {
|
|
Hash string `json:"hash"`
|
|
PathJSON string `json:"path_json"`
|
|
} `json:"data"`
|
|
}
|
|
if err := json.Unmarshal(raw, &m); err != nil {
|
|
t.Fatalf("broadcast: %v", err)
|
|
}
|
|
key := m.Data.Hash + " " + m.Data.PathJSON
|
|
if seen[key] {
|
|
t.Errorf("broadcast twice: %s", key)
|
|
}
|
|
seen[key] = true
|
|
if id, ok := loadedPaths[m.Data.PathJSON]; ok {
|
|
t.Errorf("broadcast obs %d, which the startup load already held", id)
|
|
}
|
|
}
|
|
if len(msgs) == 0 {
|
|
t.Error("no broadcasts captured; the capturing client is not wired")
|
|
}
|
|
}
|
|
|
|
// assertReplayIdempotent re-reads every row from id 0. A store that already
|
|
// holds them all must ingest nothing and announce nothing. notLoaded is as
|
|
// for assertStoreMatchesDB.
|
|
func assertReplayIdempotent(t *testing.T, store *PacketStore, rw *sql.DB, notLoaded ...int) {
|
|
t.Helper()
|
|
// IngestNewFromDB applies no retention window, so with transmissions
|
|
// outside it a replay would rightly load them; replay observations only.
|
|
if len(notLoaded) == 0 {
|
|
if txs, _ := store.IngestNewFromDB(0, 1000); len(txs) != 0 {
|
|
t.Errorf("replaying transmissions from 0 broadcast %d already-held packets", len(txs))
|
|
}
|
|
}
|
|
if obs := store.IngestNewObservations(0, 1000); len(obs) != 0 {
|
|
t.Errorf("replaying observations from 0 broadcast %d already-held observations", len(obs))
|
|
}
|
|
assertStoreMatchesDB(t, store, rw, notLoaded...)
|
|
}
|
|
|
|
// gapSeeder writes rows the way the ingestor does, with explicit ids (one
|
|
// writer: ids commit in order).
|
|
type gapSeeder struct {
|
|
t *testing.T
|
|
rw *sql.DB
|
|
base time.Time
|
|
}
|
|
|
|
func newGapSeeder(t *testing.T, rw *sql.DB) *gapSeeder {
|
|
g := &gapSeeder{t: t, rw: rw, base: time.Now().UTC().Add(-10 * time.Minute).Truncate(time.Second)}
|
|
for idx := 1; idx <= 3; idx++ {
|
|
g.exec(`INSERT INTO observers (rowid, id, name, iata, last_seen, first_seen) VALUES (?, ?, ?, 'SJC', ?, ?)`,
|
|
idx, fmt.Sprintf("%064X", idx), fmt.Sprintf("Obs-%d", idx), g.iso(g.base), g.iso(g.base.Add(-time.Hour)))
|
|
}
|
|
return g
|
|
}
|
|
|
|
func (g *gapSeeder) iso(ts time.Time) string { return ts.Format("2006-01-02T15:04:05.000Z") }
|
|
|
|
// exec may run on a load or poller goroutine, so it reports with Errorf.
|
|
func (g *gapSeeder) exec(q string, args ...any) {
|
|
if _, err := g.rw.Exec(q, args...); err != nil {
|
|
g.t.Errorf("seed: %v\n%s", err, q)
|
|
}
|
|
}
|
|
|
|
// tx inserts a transmission seen at base+id seconds, or two hours ago when
|
|
// old, which puts it outside a one-hour retention window.
|
|
func (g *gapSeeder) tx(id int, old bool) {
|
|
seen := g.base.Add(time.Duration(id) * time.Second)
|
|
if old {
|
|
seen = g.base.Add(-2 * time.Hour)
|
|
}
|
|
g.exec(`INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json, last_seen)
|
|
VALUES (?, ?, ?, ?, 1, 5, 1, '{"type":"GRP_TXT"}', ?)`,
|
|
id, fmt.Sprintf("11%02dAABBCCDD", id), gapHash(id), g.iso(seen), seen.Unix())
|
|
}
|
|
|
|
// obs inserts an observation with a unique path, so none dedupes.
|
|
func (g *gapSeeder) obs(id, tx, observer int) {
|
|
g.exec(`INSERT INTO observations (id, transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp)
|
|
VALUES (?, ?, ?, 'rx', 5.5, -90, 1, ?, ?)`,
|
|
id, tx, observer, fmt.Sprintf(`["%02X"]`, id), g.base.Add(time.Duration(id)*time.Second).Unix())
|
|
}
|
|
|
|
func gapWait(t *testing.T, ch <-chan struct{}, what string) bool {
|
|
select {
|
|
case <-ch:
|
|
return true
|
|
case <-time.After(5 * time.Second):
|
|
t.Errorf("timed out waiting for %s", what)
|
|
return false
|
|
}
|
|
}
|
|
|
|
func storeHasObs(store *PacketStore, id int) bool {
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
_, ok := store.byObsID[id]
|
|
return ok
|
|
}
|
|
|
|
func waitStoreObs(store *PacketStore, id int, within time.Duration) bool {
|
|
deadline := time.Now().Add(within)
|
|
for time.Now().Before(deadline) {
|
|
if storeHasObs(store, id) {
|
|
return true
|
|
}
|
|
time.Sleep(10 * time.Millisecond)
|
|
}
|
|
return storeHasObs(store, id)
|
|
}
|
|
|
|
// The poller runs from the first chunk on, while later chunks still load.
|
|
// Obs 10 of chunk-2 tx 3 is committed after chunk 2's query and polled before
|
|
// chunk 2 merges, when tx 3 is not in the store yet. It must end up in the
|
|
// store exactly once. This pins an ordering, not a mechanism of its own: the
|
|
// poller reads its observation cursor bound before ingestNewObservations polls
|
|
// the advert feed, which waits on advertEvidenceMu that each chunk holds from
|
|
// its query to its merge. So obs 10 lies above the bound of the poll that
|
|
// skips it, and is read again once tx 3 is merged (or the chunk already saw
|
|
// it). Obs 11 belongs to a transmission outside retention; it is skipped and
|
|
// must not hold back the live observation committed after the load.
|
|
|
|
func TestStartupObsDuringChunkMerge_SQLite(t *testing.T) {
|
|
db, rw := openGapSQLite(t)
|
|
runStartupObsDuringChunkMerge(t, db, rw)
|
|
}
|
|
|
|
func runStartupObsDuringChunkMerge(t *testing.T, db *DB, rw *sql.DB) {
|
|
g := newGapSeeder(t, rw)
|
|
for id := 1; id <= 4; id++ {
|
|
g.tx(id, false)
|
|
g.obs(id, id, 1)
|
|
}
|
|
g.tx(5, true) // outside retention: never loaded
|
|
g.obs(5, 5, 1)
|
|
|
|
store := NewPacketStore(db, &PacketStoreConfig{RetentionHours: 1})
|
|
pollerAtFeeds := make(chan struct{})
|
|
releasePoller := make(chan struct{})
|
|
pollerMerged := make(chan struct{})
|
|
var stage atomic.Int32 // 0 idle, 1 poller paused, 2 merge done
|
|
store.obsPollHook = func(st obsPollStage) {
|
|
switch {
|
|
case st == obsPollAfterFeeds && stage.CompareAndSwap(0, 1):
|
|
close(pollerAtFeeds)
|
|
gapWait(t, releasePoller, "chunk 2's query")
|
|
case st == obsPollDone && stage.CompareAndSwap(1, 2):
|
|
close(pollerMerged)
|
|
}
|
|
}
|
|
store.OnChunkLoaded(func(int, int) {
|
|
if store.loadProgressRows.Load() == 2 { // after chunk 1
|
|
gapWait(t, pollerAtFeeds, "the poller's first tick")
|
|
}
|
|
})
|
|
store.chunkQueriedHook = func(chunk int) {
|
|
if chunk != 2 {
|
|
return
|
|
}
|
|
g.obs(10, 3, 2) // of tx 3, which this chunk is about to merge
|
|
g.obs(11, 5, 2) // orphan: tx 5 is outside retention
|
|
close(releasePoller)
|
|
gapWait(t, pollerMerged, "the poller's merge")
|
|
}
|
|
|
|
loadErr := make(chan error, 1)
|
|
go func() { loadErr <- store.RunStartupLoad(2) }()
|
|
<-store.FirstChunkReady()
|
|
pr := startGapPoller(t, db, store)
|
|
if err := <-loadErr; err != nil {
|
|
t.Fatalf("RunStartupLoad: %v", err)
|
|
}
|
|
if stage.Load() != 2 {
|
|
t.Fatal("setup: the poller never polled between chunk 2's query and its merge")
|
|
}
|
|
|
|
// After the load: a live observation must not wait behind orphan 11.
|
|
g.obs(12, 1, 3)
|
|
if !waitStoreObs(store, 12, 3*time.Second) {
|
|
t.Fatal("live obs 12, committed after the load completed, never reached the store: the feed is blocked")
|
|
}
|
|
pr.stop()
|
|
if !storeHasObs(store, 10) {
|
|
t.Error("obs 10 (chunk-2 tx, committed after chunk 2's query, polled before its merge) is missing from the store")
|
|
}
|
|
if storeHasObs(store, 11) {
|
|
t.Error("obs 11 of a transmission outside retention is in the store")
|
|
}
|
|
assertStoreMatchesDB(t, store, rw, 5)
|
|
assertNoDuplicateBroadcasts(t, pr.drain(), nil)
|
|
assertReplayIdempotent(t, store, rw, 5)
|
|
}
|
|
|
|
// A failed transmission or observation read must leave the poller's cursor
|
|
// where it was, so the same rows are read once the database recovers. The
|
|
// observation cursor's bound comes from its own query, which used to advance
|
|
// the cursor past rows the failed read never saw. Renaming observers fails
|
|
// both ingest reads (they join it) while the bound query still succeeds.
|
|
|
|
func TestPollerRetriesFailedRead_SQLite(t *testing.T) {
|
|
db, rw := openGapSQLite(t)
|
|
runPollerRetriesFailedRead(t, db, rw)
|
|
}
|
|
|
|
func runPollerRetriesFailedRead(t *testing.T, db *DB, rw *sql.DB) {
|
|
g := newGapSeeder(t, rw)
|
|
g.tx(1, false)
|
|
g.obs(1, 1, 1)
|
|
store := NewPacketStore(db, &PacketStoreConfig{})
|
|
if err := store.RunStartupLoad(100); err != nil {
|
|
t.Fatalf("RunStartupLoad: %v", err)
|
|
}
|
|
loaded := loadedObsPaths(store)
|
|
pr := startGapPoller(t, db, store)
|
|
|
|
g.exec(`ALTER TABLE observers RENAME TO observers_away`)
|
|
renamed := true
|
|
defer func() {
|
|
if renamed {
|
|
g.exec(`ALTER TABLE observers_away RENAME TO observers`)
|
|
}
|
|
}()
|
|
g.tx(2, false)
|
|
g.obs(2, 2, 1) // arrives with tx 2
|
|
g.obs(3, 1, 2) // a new observation of loaded tx 1
|
|
|
|
time.Sleep(150 * time.Millisecond) // several failing ticks
|
|
if storeHasObs(store, 2) || storeHasObs(store, 3) {
|
|
t.Fatal("setup: the ingest reads did not fail while observers was renamed")
|
|
}
|
|
g.exec(`ALTER TABLE observers_away RENAME TO observers`)
|
|
renamed = false
|
|
|
|
if !waitStoreObs(store, 3, 3*time.Second) {
|
|
t.Error("obs 3, committed while the observation read failed, never reached the store: the cursor passed it")
|
|
}
|
|
if !waitStoreObs(store, 2, time.Second) {
|
|
t.Error("tx 2 and its obs 2, committed while the transmission read failed, never reached the store")
|
|
}
|
|
pr.stop()
|
|
assertStoreMatchesDB(t, store, rw)
|
|
assertNoDuplicateBroadcasts(t, pr.drain(), loaded)
|
|
assertReplayIdempotent(t, store, rw)
|
|
}
|