dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-10-23 23:32:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992)

Browse Source 
Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite
3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4),
and pays the resulting cross-compilation cost with `zig cc`.

Draft because the riskiest part of this deletes rows — see [Please
review this part first](#please-review-this-part-first) — and because
three things remain unverified at the bottom.

`modernc.org/sqlite` is a transpilation of the C amalgamation. This repo
is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out
neighbour/topology/analytics queries per request, and it pays for that
transpilation on exactly those paths. Head-to-head on the same
120k-transmission / 240k-observation database, running our own hot-path
SQL under both drivers (Apple M4, `-count=5`, medians):

| workload | modernc | mattn | |
|---|---:|---:|---|
| chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms |
**2.3×** |
| aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×**
|
| 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** |

Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the
chunk load.

**Superseded by a production run.** @efiten measured both drivers on a
real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 —
as server-only containers against the same live volume, one at a time,
with round 2 reversing the order so the page cache favours the old
driver:

| | audit 7d | audit 24h | background fill (13 chunks) | start →
/api/health |
|---|---:|---:|---:|---:|
| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |

Warm, the old driver improves to 13.46s on the 7d audit and still loses
by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against
0.037s — nothing.

**So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.**
The shape the harness predicted holds — scans and joins gain, small
lookups do not — which is more reassuring than the magnitude would have
been. Quote these numbers.

**The counterweight**, cold and native on that machine: a build goes
from **52s to 163s**. An instance that builds its own image pays that
per deploy.

## The build is cgo now, and one thing about that is a trap

**`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary
dies on its first query with `go-sqlite3 requires cgo to work. This is a
stub`. A green build is not evidence of anything here, which is why
`AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely
cannot cross-compile any more.

A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc
-target {x86_64,aarch64}-linux-musl` and links static, so each artifact
stays a single self-contained file and the `alpine:3.20` runtime no
longer depends on the base image's libc at all.

`-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl
objects zig links in, and without it the server binary is 19.8MB instead
of 12.1MB.

The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU
for compilation — and gains a checksum-pinned zig plus BuildKit cache
mounts. The mounts are not a nicety: without them an image build
recompiles the amalgamation from cold and takes over half an hour.

## Please review this part first

`internal/dbschema/dedup_index.go` **deletes observation rows**. It is
the one part of this change that can lose data, and it exists because
the migration exposed a real bug rather than causing one.

`stmtInsertObservation` resolves its `ON CONFLICT` against
`idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created
inside the branch that creates the `observations` table for the first
time. Any database whose table predates that branch never got one, so
the UPSERT had no conflict target. modernc failed on the first insert;
mattn fails at `OpenStore`. Same bug, found earlier.

Creating the index unconditionally repairs it — but the index is what
was supposed to prevent duplicates, so a database that never had it can
already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this
repo holds one.** So duplicates are collapsed first. Refusing is not the
safer option: without the index the ingestor cannot prepare its UPSERT,
so it cannot start at all.

Replaying that UPSERT faithfully is subtler than it looks, and a first
cut of this got it wrong twice:

- `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a
group in id order the survivor keeps the **last** non-NULL value. Taking
the first silently discarded newer readings.
- The UPSERT names exactly five columns (`snr`, `rssi`, `score`,
`raw_hex`, `resolved_path`). Every other column must keep the surviving
row's own value; merging those too invents history the ingestor would
never have written.

Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split
apart, a writer inserting a duplicate in the gap fails the index
creation while leaving the deletions committed — rows destroyed and no
index to show for it.

Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**,
holding the write lock throughout, once, at ingestor startup before MQTT
subscribe. Materialising the duplicate-group scan once rather than per
column took that from 9.7s; the pathological case (400k of 600k rows
duplicated) is 5.7s, slightly worse than the 4.2s it was before that
change.

## Four more behavioural differences

Full detail in `docs/sqlite-driver-migration.md`. Briefly:

**Statement preparation is eager.** modernc's `newStmt` stored the SQL
and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`,
so SQL naming a missing table fails at *open*. 59 server tests failed on
this alone, all fixtures with partial schemas. `OpenDB` keeps failing
loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the
fixtures now declare what they are prepared against via
`ensurePreparable`. This also exposed nine `nodes(pubkey …)`
declarations across seven files, where production has only ever had
`public_key` — lazy compilation had hidden the mismatch for as long as
it existed.

**`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to
NORMAL and executes the pragma unconditionally, where SQLite's own
default (what modernc left alone) is FULL. In WAL mode that weakens
durability under power loss. Pinned in `dbschema.WriterDSN`, which both
writers now share — `cmd/migrate` kept a bare path at first and so
quietly wrote at NORMAL, which is what a second copy of a DSN buys you.

**The DSN dialects are mutually invisible.** modernc understood only
`_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither
errors on the other's form — a driver-only rename would have dropped
every pragma in silence. `_journal_mode=WAL` is also gone from the
server's read handle: modernc ignored it, mattn honours it, and setting
`journal_mode` on a read-only connection is a write. Dropping
`_busy_timeout` with it costs nothing, since mattn already defaults to
5000ms — which means the read handle finally *gets* the busy timeout it
had silently lacked.

**`mode=ro` survives for a non-obvious reason.** mattn always passes
`READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what
makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the
#1283/#1289 invariant holds with no build flags — but it depends on the
`file:` prefix. `cmd/decrypt` had been building its DSN without one, so
its `mode=ro` had never applied and a missing path was created
read-write. Fixed in passing; never a migration regression.

## What did not change

No modernc-specific API was in use: no `RegisterFunction`, no
`*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No
`time.Time` is ever bound as a query argument, so driver time handling
is not in play. Both drivers convert declared
`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so
`/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an
earlier draft "fixed" that with a `CAST` and would have been the
regression.

## Tests and CI

New regression tests, each written because something got through without
it:

- `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge
ordering. The original test used complementary NULLs, which passes
whichever direction you pick, which is why the bug survived it.
- `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation
must roll the deletions back.
- `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma,
read back through the store's own connection. A separate `sqlite3`
session or the startup log line would prove nothing.
- `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which
now rests on a detail of the driver's C wrapper.
- `TestEnsurePreparableMatchesPrepareStatements` — fails when a new
prepared statement outgrows the fixture helper.

CI gains test execution for `cmd/migrate` and `internal/dbschema`, which
had none and both open the database. A PR-time two-arch build plus an
arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without
it nothing on a PR would exercise zig, static musl linking or arm64, and
the first signal would arrive on master. `cache-dependency-path` widens
from 2 of the 5 tracked `go.sum` files to all of them.

`make test` passes across all 14 modules, `cmd/server` also under `-race
-count=2` with no failures and no races. `gofmt` and `go vet` clean.
Release-routing and Dockerfile COPY-invariant gates pass.

## Verified by running

- All 8 cross-builds static and correct-architecture; both arches of the
container image built, exported and run under QEMU, serving
`/api/health` and `/api/nodes` against a 2.9M-observation production
snapshot.
- The `migrate` binary repairing that snapshot's duplicate on bare
Alpine.
- `CGO_ENABLED=0` producing a binary that builds and then fails on first
query.

## Not verified

- ~~The 2–2.3× figures come from a standalone harness, not this load
under the old driver.~~ **Closed** by @efiten's production run above,
which also corrected the multiplier.
- SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no
total `ORDER BY`.
- Sustained live ingest through the new writer DSN, and the duplicate
collapse against a database an ingestor is actively writing to. Verified
against a static snapshot only, and the collapse is measured at 4.1s on
2.4M synthetic rows with 5 duplicates — well short of an 11M-row
instance. @efiten has offered a staging instance taking real MQTT
traffic; **this is the item to close before the PR leaves draft.**

An earlier revision of this branch shipped the dedup merge in the wrong
direction with a green test suite, and review then found three more
things in the same file: the repair gated on an error string, a
non-atomic TEMP table drop aimed at the wrong connection, and a deletion
whose only record was a row count. All fixed in ac7e8d38. Passing tests
did not establish safety here, which is why the deletion path wanted a
second pair of eyes rather than a rubber stamp.
This commit is contained in:
Sylvain Rabot authored and GitHub committed 2026-09-16 09:02:13 +02:00
1 parent cb994d9f9a
commit a2ea18f778
69 files changed
+1982 -425
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

cmd/server/bounded_load_test.go
+13 -13
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDB creates a temporary SQLite database with N transmissions (1 obs each).
@@ -149,7 +149,7 @@ func createTestDBWithAgedPackets(t *testing.T, numRecent, numOld int) string {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -162,8 +162,8 @@ func createTestDBWithAgedPackets(t *testing.T, numRecent, numOld int) string {
}
execOrFail(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, raw_hex TEXT, hash TEXT, first_seen TEXT, route_type INTEGER, payload_type INTEGER, payload_version INTEGER, decoded_json TEXT)`)
execOrFail(`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT, direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
execOrFail(`CREATE TABLE nodes (pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
execOrFail(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
execOrFail(`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`)
@@ -171,7 +171,7 @@ func createTestDBWithAgedPackets(t *testing.T, numRecent, numOld int) string {
now := time.Now().UTC()
id := 1
// Single transaction for all inserts — see createTestDBAt for the rationale
// (modernc.org/sqlite auto-commit per Exec fsyncs per row). numOld+numRecent
// (auto-commit per Exec fsyncs per row). numOld+numRecent
// is small here today, but wrapping keeps the fixture robust if callers scale
// it up, and is consistent with the other builders.
if _, err := conn.Exec("BEGIN"); err != nil {
@@ -308,7 +308,7 @@ func BenchmarkLoad_30K_Unlimited(b *testing.B) {
// createTestDBAt is like createTestDB but writes to a specific path.
func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
@@ -331,9 +331,9 @@ func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`)
@@ -352,7 +352,7 @@ func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
defer obsStmt.Close()
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
// Wrap the inserts in a single transaction. Without this, modernc.org/sqlite
// Wrap the inserts in a single transaction. Without this, SQLite
// (pure-Go driver) auto-commits every Exec → one fsync per row → ~2N fsyncs
// for N transmissions (tx + obs). At numTx=5000 that is ~10k fsyncs and the
// fixture blows past the test timeout (the #1741 hang). A single
@@ -379,7 +379,7 @@ func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
// createTestDBWithObs creates a test DB with realistic observation counts (1–5 per tx).
func createTestDBWithObs(tb testing.TB, dbPath string, numTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
@@ -398,9 +398,9 @@ func createTestDBWithObs(tb testing.TB, dbPath string, numTx int) {
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`)
@@ -423,7 +423,7 @@ func createTestDBWithObs(tb testing.TB, dbPath string, numTx int) {
obsID := 1
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
// Single transaction for all inserts — see createTestDBAt for the rationale
// (modernc.org/sqlite auto-commit per Exec would fsync per row; at numTx=30000
// (auto-commit per Exec would fsync per row; at numTx=30000
// the benchmarks would otherwise stall for minutes). One BEGIN/COMMIT.
if _, err := conn.Exec("BEGIN"); err != nil {
tb.Fatalf("test DB BEGIN: %v", err)
cmd/server/chunked_load_id_zero_test.go
+3 -3
View File
@@ -29,7 +29,7 @@ import (
func createTestDBWithIDZero(tb testing.TB, dbPath string, extraTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
@@ -48,9 +48,9 @@ func createTestDBWithIDZero(tb testing.TB, dbPath string, extraTx int) {
direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`,
`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`,
`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`,
`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`,
`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`,
cmd/server/chunked_load_oldest_test.go
+3 -3
View File
@@ -31,7 +31,7 @@ import (
// This mirrors the freshen-fixture-shifted e2e DB exactly.
func createTestDBReverseTime(tb testing.TB, dbPath string, numTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
@@ -50,9 +50,9 @@ func createTestDBReverseTime(tb testing.TB, dbPath string, numTx int) {
direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`,
`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`,
`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`,
`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`,
`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`,
cmd/server/coverage_test.go
+2 -2
View File
@@ -15,14 +15,14 @@ import (
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// --- helpers ---
func setupTestDBv2(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
cmd/server/db.go
+19 -4
View File
@@ -13,9 +13,9 @@ import (
"sync"
"time"
_ "github.com/mattn/go-sqlite3"
"github.com/meshcore-analyzer/dbschema"
"github.com/meshcore-analyzer/geofilter"
_ "modernc.org/sqlite"
)
// routeTypeTransport covers TRANSPORT_FLOOD (0) and TRANSPORT_DIRECT (3) —
@@ -103,10 +103,25 @@ type channelMessagesCacheEntry struct {
exp time.Time
}
// OpenDB opens a read-only SQLite connection with WAL mode.
// OpenDB opens a read-only SQLite connection.
//
// The DSN used to pass _journal_mode=WAL and _busy_timeout=5000, which
// modernc.org/sqlite silently ignored: it understood only the
// _pragma=name(value) form. Under github.com/mattn/go-sqlite3 those parameters
// are honoured, and setting journal_mode on a read-only handle is a write — it
// would succeed only because the database is already WAL. Both are gone:
// mattn's own busy_timeout default is already 5000ms, so the read handle keeps
// the timeout (which it had silently lacked) without the pointless write.
//
// What remains is deliberate. mode=ro is the read-only invariant from
// #1283/#1289 and works because mattn's C wrapper ORs SQLITE_OPEN_URI into the
// open flags — see TestOpenDBRefusesMissingDatabase, which fails if that ever
// stops holding. _cache_size pins SQLite's C-allocated page cache at 2 MiB per
// connection; it sits outside GOMEMLIMIT, so it is bounded here rather than
// left to the driver's default (see memlimit.go).
func OpenDB(path string) (*DB, error) {
dsn := fmt.Sprintf("file:%s?mode=ro&_journal_mode=WAL&_busy_timeout=5000", path)
conn, err := sql.Open("sqlite", dsn)
dsn := fmt.Sprintf("file:%s?mode=ro&_cache_size=-2000", path)
conn, err := sql.Open("sqlite3", dsn)
if err != nil {
return nil, err
}
cmd/server/db_test.go
+20 -14
View File
@@ -11,13 +11,13 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// setupTestDB creates an in-memory SQLite database with the v3 schema.
func setupTestDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
@@ -1198,7 +1198,7 @@ func TestGetNodesFiltering(t *testing.T) {
// where observations use observer_id TEXT instead of observer_idx INTEGER.
func setupTestDBV2(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
@@ -1226,7 +1226,8 @@ func setupTestDBV2(t *testing.T) *DB {
last_seen TEXT,
first_seen TEXT,
packet_count INTEGER DEFAULT 0,
last_packet_at TEXT DEFAULT NULL
last_packet_at TEXT DEFAULT NULL,
inactive INTEGER DEFAULT 0
);
CREATE TABLE transmissions (
@@ -1454,7 +1455,7 @@ func TestOpenDBValid(t *testing.T) {
dbPath := filepath.Join(dir, "test.db")
// Create DB with a table using a writable connection first
conn, err := sql.Open("sqlite", dbPath)
conn, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
@@ -1463,6 +1464,7 @@ func TestOpenDBValid(t *testing.T) {
conn.Close()
t.Fatal(err)
}
ensurePreparable(t, conn)
conn.Close()
// Now test OpenDB (read-only)
@@ -1495,7 +1497,7 @@ func TestDetectSchemaScopeName(t *testing.T) {
dbPath := filepath.Join(dir, "detect.db")
// Create file-based DB with the scope_name and default_scope columns.
conn, err := sql.Open("sqlite", dbPath)
conn, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
@@ -1512,6 +1514,7 @@ func TestDetectSchemaScopeName(t *testing.T) {
conn.Close()
t.Fatalf("create observations: %v", err)
}
ensurePreparable(t, conn)
conn.Close()
db, err := OpenDB(dbPath)
@@ -1529,7 +1532,7 @@ func TestDetectSchemaScopeName(t *testing.T) {
// Verify the flags stay false when the columns are absent.
dbPath2 := filepath.Join(dir, "detect2.db")
conn2, err := sql.Open("sqlite", dbPath2)
conn2, err := sql.Open("sqlite3", dbPath2)
if err != nil {
t.Fatal(err)
}
@@ -1537,6 +1540,7 @@ func TestDetectSchemaScopeName(t *testing.T) {
conn2.Exec(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, hash TEXT)`)
conn2.Exec(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY)`)
conn2.Exec(`CREATE TABLE observations (id INTEGER PRIMARY KEY)`)
ensurePreparable(t, conn2)
conn2.Close()
db2, err := OpenDB(dbPath2)
@@ -2233,7 +2237,7 @@ func TestPerObservationRawHexEnrich(t *testing.T) {
}
func TestGetScopeStats(t *testing.T) {
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("sql.Open: %v", err)
}
@@ -2293,7 +2297,7 @@ func TestGetScopeStats(t *testing.T) {
// scope_name states (NULL is unscoped, an empty string is unknown scope, a
// name is a named scope).
func TestGetScopeStatsAdvertsByRole(t *testing.T) {
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("sql.Open: %v", err)
}
@@ -2377,7 +2381,7 @@ func TestGetScopeStatsAdvertsByRole(t *testing.T) {
}
func TestGetScopeStatsAdvertsByRoleEmpty(t *testing.T) {
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("sql.Open: %v", err)
}
@@ -2463,7 +2467,7 @@ func (f *failingQuerier) QueryContext(ctx context.Context, query string, args ..
// a v3 database for the whole process lifetime.
func TestDetectSchemaFailsLoudOnProbeError(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "v3.db")
conn, err := sql.Open("sqlite", dbPath)
conn, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
@@ -2477,7 +2481,7 @@ func TestDetectSchemaFailsLoudOnProbeError(t *testing.T) {
conn.Exec(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY)`)
conn.Close()
real, err := sql.Open("sqlite", "file:"+dbPath+"?mode=ro")
real, err := sql.Open("sqlite3", "file:"+dbPath+"?mode=ro")
if err != nil {
t.Fatal(err)
}
@@ -2499,7 +2503,7 @@ func TestDetectSchemaV3AndV2(t *testing.T) {
dir := t.TempDir()
v3 := filepath.Join(dir, "v3.db")
c, err := sql.Open("sqlite", v3)
c, err := sql.Open("sqlite3", v3)
if err != nil {
t.Fatal(err)
}
@@ -2507,6 +2511,7 @@ func TestDetectSchemaV3AndV2(t *testing.T) {
c.Exec(`CREATE TABLE observations (id INTEGER PRIMARY KEY, observer_idx INTEGER)`)
c.Exec(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, hash TEXT)`)
c.Exec(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY)`)
ensurePreparable(t, c)
c.Close()
db, err := OpenDB(v3)
if err != nil {
@@ -2518,7 +2523,7 @@ func TestDetectSchemaV3AndV2(t *testing.T) {
db.Close()
v2 := filepath.Join(dir, "v2.db")
c2, err := sql.Open("sqlite", v2)
c2, err := sql.Open("sqlite3", v2)
if err != nil {
t.Fatal(err)
}
@@ -2526,6 +2531,7 @@ func TestDetectSchemaV3AndV2(t *testing.T) {
c2.Exec(`CREATE TABLE observations (id INTEGER PRIMARY KEY, observer_id INTEGER)`)
c2.Exec(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, hash TEXT)`)
c2.Exec(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY)`)
ensurePreparable(t, c2)
c2.Close()
db2, err := OpenDB(v2)
if err != nil {
cmd/server/go.mod
+1 -13
View File
@@ -7,7 +7,6 @@ require (
github.com/gorilla/websocket v1.5.3
github.com/meshcore-analyzer/geofilter v0.0.0
github.com/meshcore-analyzer/sigvalidate v0.0.0
modernc.org/sqlite v1.34.5
)
replace github.com/meshcore-analyzer/geofilter => ../../internal/geofilter
@@ -34,23 +33,12 @@ require github.com/meshcore-analyzer/lora v0.0.0
replace github.com/meshcore-analyzer/lora => ../../internal/lora
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/sys v0.22.0 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
)
require github.com/meshcore-analyzer/prunequeue v0.0.0
replace github.com/meshcore-analyzer/prunequeue => ../../internal/prunequeue
require (
github.com/mattn/go-sqlite3 v1.14.52
github.com/meshcore-analyzer/mbcapqueue v0.0.0
golang.org/x/sync v0.10.0
)
cmd/server/go.sum
+2 -43
View File
@@ -1,49 +1,8 @@
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
github.com/mattn/go-sqlite3 v1.14.52 h1:wVbm2Qnf4OXkqhBTSPuCRZDRnxfbVrrmiCEroVdog8U=
github.com/mattn/go-sqlite3 v1.14.52/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
modernc.org/sqlite v1.34.5 h1:Bb6SR13/fjp15jt70CL4f18JIN7p7dnMExd+UFnF15g=
modernc.org/sqlite v1.34.5/go.mod h1:YLuNmX9NKs8wRNK2ko1LW1NGYcc9FkBO69JOt1AR9JE=
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
cmd/server/hexgrid.go
+3 -1
View File
@@ -8,7 +8,9 @@ import (
)
// Pure-Go hexagonal binning for RX coverage display. We deliberately avoid the
// CGO-based uber/h3-go (this project builds with CGO_ENABLED=0). Points are
// CGO-based uber/h3-go. (That predates the SQLite driver move to cgo; this
// stays pure Go because it needs no C library, not because cgo is unavailable.)
// Points are
// projected to Web Mercator and snapped to a pointy-top hex grid whose size
// depends on the display resolution. Cell ids are "res:q:r" (axial coords).
// At city/region scale this looks like H3/mapme.sh coverage without any deps.
cmd/server/hot_startup_test.go
+4 -4
View File
@@ -13,7 +13,7 @@ import (
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDBMultiDay creates a test DB with packets spread across numDays days.
@@ -24,7 +24,7 @@ func createTestDBMultiDay(t *testing.T, numDays, txPerDay int) string {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -37,8 +37,8 @@ func createTestDBMultiDay(t *testing.T, numDays, txPerDay int) string {
}
execOrFail(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, raw_hex TEXT, hash TEXT, first_seen TEXT, route_type INTEGER, payload_type INTEGER, payload_version INTEGER, decoded_json TEXT)`)
execOrFail(`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT, direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
execOrFail(`CREATE TABLE nodes (pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
execOrFail(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
execOrFail(`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`)
cmd/server/issue1690_cold_load_test.go
+1 -1
View File
@@ -19,7 +19,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDBWithLastSeen seeds a DB with the post-fix schema (last_seen
cmd/server/issue1809_bg_load_race_test.go
+3 -3
View File
@@ -138,7 +138,7 @@ func createTestDBSpreadOverDays(t *testing.T, dbPath string, numTx, spanDays int
// (adv #11: prior code carried a duplicated annotation on every line)."
func seedTestDBRows(t *testing.T, dbPath string, numTx, obsPerTx int, rowTimes func(i int) (string, int64)) {
t.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -164,9 +164,9 @@ func seedTestDBRows(t *testing.T, dbPath string, numTx, obsPerTx int, rowTimes f
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
// PREFLIGHT: async=true reason="unit-test fixture seeder"
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
// PREFLIGHT: async=true reason="unit-test fixture seeder"
execOrFail(`CREATE TABLE nodes (pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
// PREFLIGHT: async=true reason="unit-test fixture seeder"
execOrFail(`CREATE TABLE schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
cmd/server/issue673_test.go
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
const issue673NodePK = "7502f19f44cad6d7b626e1d811c00a914af452636182ccded3fd019803395ec9"
cmd/server/load_graph_invariant_test.go
+3 -3
View File
@@ -6,7 +6,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// TestLoad_PanicsWhenGraphNotLoadedAndEdgesExist pins the startup-ordering
@@ -22,7 +22,7 @@ func TestLoad_PanicsWhenGraphNotLoadedAndEdgesExist(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
rw, err := sql.Open("sqlite3", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -49,7 +49,7 @@ func TestLoad_PanicsWhenGraphNotLoadedAndEdgesExist(t *testing.T) {
path_json TEXT, timestamp TEXT, raw_hex TEXT, resolved_path TEXT
)`)
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
cmd/server/loadchunk_ambiguous_prefix_test.go
+3 -3
View File
@@ -7,7 +7,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDBAmbiguousPrefix builds a fixture where TWO repeaters share the
@@ -24,7 +24,7 @@ func createTestDBAmbiguousPrefix(t *testing.T, relayA, relayB, hop, firstSeen st
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -54,7 +54,7 @@ func createTestDBAmbiguousPrefix(t *testing.T, relayA, relayB, hop, firstSeen st
resolved_path TEXT
)`)
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
cmd/server/loadchunk_pathjson_fallback_test.go
+3 -3
View File
@@ -7,7 +7,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDBPathJSONNoResolvedPath builds a fixture that mirrors the LIVE
@@ -25,7 +25,7 @@ func createTestDBPathJSONNoResolvedPath(t *testing.T, relayPubkey, hopPrefix, fi
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -56,7 +56,7 @@ func createTestDBPathJSONNoResolvedPath(t *testing.T, relayPubkey, hopPrefix, fi
resolved_path TEXT
)`)
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
// Production nodes schema uses public_key (not pubkey) — getAllNodes /
// buildPrefixMap reads public_key, role, advert_count, first_seen.
// PREFLIGHT: async=true reason="test fixture, in-memory tmpdir DB"
cmd/server/loadchunk_resolved_path_1558_test.go
+4 -4
View File
@@ -7,7 +7,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// createTestDBWithResolvedPath creates a fixture DB containing numTx old
@@ -20,7 +20,7 @@ func createTestDBWithResolvedPath(t *testing.T, numTx int, relayPubkeys []string
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -47,8 +47,8 @@ func createTestDBWithResolvedPath(t *testing.T, numTx int, relayPubkeys []string
raw_hex TEXT,
resolved_path TEXT
)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE nodes (pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
exec(`CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
exec(`CREATE TABLE schema_version (version INTEGER)`)
exec(`INSERT INTO schema_version (version) VALUES (1)`)
exec(`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`)
cmd/server/memlimit.go
+15
View File
@@ -36,6 +36,21 @@ func applyMemoryLimit(maxMemoryMB int, envSet bool) (int64, string) {
// 1.5x headroom over the steady-state packet store budget covers
// transient peaks (cold-load row-scan / decode pipeline, Go's NextGC
// trigger at ~2x live heap). See issue #836 heap profile.
//
// GOMEMLIMIT covers memory the Go runtime manages — heap, stacks and
// runtime structures — and nothing else. Since the SQLite driver became
// cgo (github.com/mattn/go-sqlite3), what SQLite allocates in C is
// outside it, so this limit is not an RSS ceiling: actual RSS is this
// plus the native allocations plus whatever the runtime has reserved
// but not returned.
//
// The page cache, at least, is deliberately bounded: both DSNs pin
// _cache_size=-2000, i.e. ~2 MiB per connection, ~8 MiB across
// SetMaxOpenConns(4). That fits inside the headroom above rather than
// eating into the store budget. It does not bound SQLite's other native
// allocations, so if the connection count or _cache_size grows
// materially, or a workload starts holding many prepared statements,
// this derivation needs revisiting against measured RSS.
limit := int64(maxMemoryMB) * 1024 * 1024 * 3 / 2
debug.SetMemoryLimit(limit)
return limit, "derived"
cmd/server/memory.go
+16 -3
View File
@@ -32,9 +32,22 @@ import (
// eviction; both can shrink when packets age out. goHeapInuseMB and goSysMB
// fluctuate with GC.
//
// cgoBytesMB intentionally absent: this build uses the pure-Go
// modernc.org/sqlite driver, so there is no cgo allocator to measure.
// Reintroduce only if we ever switch back to mattn/go-sqlite3.
// There is no cgo-allocation field, but for a different reason than before:
// the driver is github.com/mattn/go-sqlite3 now, so SQLite's page cache IS a C
// allocation — the Go runtime simply exposes no byte counter for it.
//
// processRSSMB does include it, since VmRSS counts every resident page whoever
// allocated it. Resist the temptation to read `processRSSMB - goSysMB` as "the
// C share": goSysMB is address space the runtime has reserved, not what is
// resident, so the subtraction mixes two different things and can even go
// negative. It is a smell test, not a measurement. To size native usage,
// measure RSS against a build with the store disabled, or profile.
//
// What is bounded is SQLite's page cache specifically: both DSNs pin
// _cache_size=-2000, i.e. ~2 MiB per connection, so roughly 8 MiB across
// SetMaxOpenConns(4) here and 2 MiB in the ingestor. That is a cap on the page
// cache, not on everything SQLite allocates — statement and schema memory sit
// outside it. See memlimit.go for why any of it matters to GOMEMLIMIT.
type MemorySnapshot struct {
ProcessRSSMB float64 `json:"processRSSMB"`
GoHeapInuseMB float64 `json:"goHeapInuseMB"`
cmd/server/multibyte_capability_test.go
+3 -3
View File
@@ -8,7 +8,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// recentTS returns a timestamp string N hours ago, ensuring test data
@@ -20,7 +20,7 @@ func recentTS(hoursAgo int) string {
// setupCapabilityTestDB creates a minimal in-memory DB with nodes table.
func setupCapabilityTestDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
@@ -446,7 +446,7 @@ func TestMultiByteCapability_AdopterEvidenceTakesPrecedence(t *testing.T) {
// setupPersistTestDB creates an in-memory DB with multibyte_sup/multibyte_evidence columns.
func setupPersistTestDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
cmd/server/neighbor_api_test.go
+12 -9
View File
@@ -9,7 +9,7 @@ import (
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// ─── Helpers ───────────────────────────────────────────────────────────────────
@@ -467,7 +467,7 @@ func TestBuildNodeInfoMap_ObserverEnrichment(t *testing.T) {
tmpDir := t.TempDir()
dbPath := tmpDir + "/test.db"
conn, err := sql.Open("sqlite", dbPath)
conn, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
@@ -476,15 +476,16 @@ func TestBuildNodeInfoMap_ObserverEnrichment(t *testing.T) {
// Create tables
for _, stmt := range []string{
"CREATE TABLE nodes (public_key TEXT, name TEXT, role TEXT, lat REAL, lon REAL)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT, inactive INTEGER)",
"INSERT INTO nodes VALUES ('AAAA1111', 'Repeater-1', 'repeater', 0, 0)",
"INSERT INTO observers VALUES ('BBBB2222', 'Observer-Alpha', '')",
"INSERT INTO observers VALUES ('AAAA1111', 'Obs-also-repeater', '')",
"INSERT INTO observers (id, name, iata) VALUES ('BBBB2222', 'Observer-Alpha', '')",
"INSERT INTO observers (id, name, iata) VALUES ('AAAA1111', 'Obs-also-repeater', '')",
} {
if _, err := conn.Exec(stmt); err != nil {
t.Fatalf("exec %q: %v", stmt, err)
}
}
ensurePreparable(t, conn)
conn.Close()
// Open via our DB wrapper
@@ -542,20 +543,21 @@ func TestBuildNodeInfoMap_FirstSeenIsCached(t *testing.T) {
dbPath := tmpDir + "/test.db"
// Seed via rw connection.
rw, err := sql.Open("sqlite", dbPath)
rw, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
defer rw.Close()
for _, stmt := range []string{
"CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, advert_count INTEGER)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT, inactive INTEGER)",
"INSERT INTO nodes VALUES ('AAAA1111', 'Repeater-1', 'repeater', 0, 0, '', '2024-01-01T00:00:00Z', 0)",
} {
if _, err := rw.Exec(stmt); err != nil {
t.Fatalf("seed exec %q: %v", stmt, err)
}
}
ensurePreparable(t, rw)
db, err := OpenDB(dbPath)
if err != nil {
@@ -608,20 +610,21 @@ func TestGetAllNodes_FirstSeenSchemaFallback(t *testing.T) {
dbPath := tmpDir + "/test.db"
// Seed a nodes table WITHOUT first_seen (advert_count + last_seen present).
rw, err := sql.Open("sqlite", dbPath)
rw, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
defer rw.Close()
for _, stmt := range []string{
"CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, advert_count INTEGER)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT)",
"CREATE TABLE observers (id TEXT, name TEXT, iata TEXT, inactive INTEGER)",
"INSERT INTO nodes VALUES ('BBBB2222', 'Repeater-2', 'repeater', 0, 0, '2024-02-02T00:00:00Z', 3)",
} {
if _, err := rw.Exec(stmt); err != nil {
t.Fatalf("seed exec %q: %v", stmt, err)
}
}
ensurePreparable(t, rw)
db, err := OpenDB(dbPath)
if err != nil {
cmd/server/neighbor_persist_legacy_test.go
+3 -3
View File
@@ -6,7 +6,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// TestNeighborPersist_LegacyEdgeInvariant (#1638 adv-#1): edges loaded from
@@ -18,7 +18,7 @@ import (
func TestNeighborPersist_LegacyEdgeInvariant(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "neighbor_legacy.db")
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
rw, err := sql.Open("sqlite3", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -79,7 +79,7 @@ func TestNeighborPersist_LegacyEdgeInvariant(t *testing.T) {
func TestNeighborPersist_LegacyEdgeMergeOnReload(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "neighbor_legacy_merge.db")
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
rw, err := sql.Open("sqlite3", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
cmd/server/neighbor_recomputer_test.go
+15 -6
View File
@@ -6,8 +6,8 @@ import (
"testing"
"time"
_ "github.com/mattn/go-sqlite3"
"github.com/meshcore-analyzer/dbschema"
_ "modernc.org/sqlite"
)
// TestNeighborGraphRecomputerLoadsSnapshot enforces #1287 Option 4:
@@ -19,7 +19,7 @@ func TestNeighborGraphRecomputerLoadsSnapshot(t *testing.T) {
dbPath := filepath.Join(dir, "neighbor_recomp.db")
// Bootstrap a WAL DB with the neighbor_edges table.
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
rw, err := sql.Open("sqlite3", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -33,6 +33,7 @@ func TestNeighborGraphRecomputerLoadsSnapshot(t *testing.T) {
)`); err != nil {
t.Fatal(err)
}
ensurePreparable(t, rw)
// Stage one edge.
now := time.Now().UTC().Format(time.RFC3339)
@@ -85,9 +86,15 @@ func TestServerStartupRequiresMigratedSchema(t *testing.T) {
// Bootstrap with ONLY transmissions/observations (the things
// server tries to read) but WITHOUT the columns dbschema asserts
// (resolved_path, inactive, last_packet_at, iata, foreign_advert,
// from_pubkey, neighbor_edges).
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
// (resolved_path, last_packet_at, iata, foreign_advert, from_pubkey,
// neighbor_edges).
//
// observers.inactive used to be in that list. It no longer is: OpenDB
// prepares statements eagerly under github.com/mattn/go-sqlite3 and one of
// them selects on inactive, so a fixture without it cannot reach
// AssertReady at all. The other missing surfaces above still make
// AssertReady fail, which is what this test asserts.
rw, err := sql.Open("sqlite3", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -95,7 +102,7 @@ func TestServerStartupRequiresMigratedSchema(t *testing.T) {
for _, s := range []string{
`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, hash TEXT, payload_type INTEGER)`,
`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER)`,
`CREATE TABLE observers (id TEXT PRIMARY KEY, name TEXT)`,
`CREATE TABLE observers (id TEXT PRIMARY KEY, name TEXT, inactive INTEGER)`,
`CREATE TABLE nodes (public_key TEXT PRIMARY KEY)`,
`CREATE TABLE inactive_nodes (public_key TEXT PRIMARY KEY)`,
} {
@@ -104,6 +111,8 @@ func TestServerStartupRequiresMigratedSchema(t *testing.T) {
}
}
ensurePreparable(t, rw)
// Open the read-only server handle and call AssertReady directly
// (production path: main.go does this before any business logic).
d, err := OpenDB(dbPath)
cmd/server/node_reach_bench_test.go
+5 -5
View File
@@ -6,7 +6,7 @@ import (
"fmt"
"testing"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// benchReachDB builds an in-memory DB with nObs observations. matchEvery
@@ -24,13 +24,13 @@ func benchReachDB(b *testing.B, nObs, matchEvery int, lowerHops bool) *DB {
// path_json is uppercase; this measures the worst case Carmack flagged).
matchPath, fillerPath = `["aa","01fa","bb"]`, `["aa","cc","bb"]`
}
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
b.Fatal(err)
}
schema := []string{
`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, hash TEXT, first_seen TEXT, payload_type INTEGER, from_pubkey TEXT)`,
`CREATE TABLE observers (id TEXT PRIMARY KEY, name TEXT)`,
`CREATE TABLE observers (id TEXT PRIMARY KEY, name TEXT, inactive INTEGER)`,
`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_idx INTEGER, snr REAL, path_json TEXT, timestamp INTEGER)`,
`CREATE INDEX idx_obs_ts ON observations(timestamp)`,
}
@@ -155,12 +155,12 @@ func BenchmarkNodeReachAttribute(b *testing.B) {
// must surface an error, not a swallowed nil. Lives in this file because
// the bench callers in the same file rely on the same signature.
func TestScanReachRows_ErrorReturn(t *testing.T) {
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatalf("open: %v", err)
}
// PREFLIGHT: async=true reason="test-only in-memory scratch schema, immediately closed"
if _, err := conn.Exec(`CREATE TABLE observations (id INTEGER); CREATE TABLE transmissions (id INTEGER); CREATE TABLE observers (rowid INTEGER, id TEXT)`); err != nil {
if _, err := conn.Exec(`CREATE TABLE observations (id INTEGER); CREATE TABLE transmissions (id INTEGER); CREATE TABLE observers (rowid INTEGER, id TEXT, inactive INTEGER)`); err != nil {
t.Fatalf("schema: %v", err)
}
conn.Close() // force QueryContext to fail
cmd/server/node_reach_endpoint_test.go
+3 -3
View File
@@ -11,7 +11,7 @@ import (
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
func serveReach(srv *Server, path string) *httptest.ResponseRecorder {
@@ -57,7 +57,7 @@ func resetReachState(t *testing.T, servers ...*Server) {
// build the zero-reach case (identifiable node, no matching observations).
func newReachIntegrationDB(t *testing.T, obsPath string) (*DB, string) {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
@@ -68,7 +68,7 @@ func newReachIntegrationDB(t *testing.T, obsPath string) (*DB, string) {
stmts := []string{
`CREATE TABLE nodes (public_key TEXT, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, advert_count INTEGER)`,
`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, from_pubkey TEXT, payload_type INTEGER)`,
`CREATE TABLE observers (id TEXT)`,
`CREATE TABLE observers (id TEXT, inactive INTEGER)`,
`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_idx INTEGER, snr REAL, path_json TEXT, timestamp INTEGER)`,
`CREATE TABLE neighbor_edges (node_a TEXT, node_b TEXT, count INTEGER)`,
}
cmd/server/node_reach_test.go
+3 -3
View File
@@ -6,20 +6,20 @@ import (
"strconv"
"testing"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// newReachScanTestDB builds a minimal observer_idx-schema DB with two rows whose
// path contains "01FA" and one that does not, for scanReachRows coverage.
func newReachScanTestDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
stmts := []string{
`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, from_pubkey TEXT, payload_type INTEGER)`,
`CREATE TABLE observers (id TEXT)`,
`CREATE TABLE observers (id TEXT, inactive INTEGER)`,
`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_idx INTEGER, snr REAL, path_json TEXT, timestamp INTEGER)`,
`INSERT INTO observers (id) VALUES ('OBS1')`, // rowid 1
`INSERT INTO transmissions (id, from_pubkey, payload_type) VALUES (1,'FF00',4),(2,'',5),(3,'',5)`,
cmd/server/perf_io.go
+1 -1
View File
@@ -260,7 +260,7 @@ func (s *Server) handlePerfSqlite(w http.ResponseWriter, r *http.Request) {
resp.CacheSize = cacheSize
// Cache hit rate: derived from PacketStore cache (rw_cache). We don't
// have a direct SQLite cache counter via the modernc driver, so we
// have a direct SQLite cache counter through the driver, so we
// surface the closest available proxy — the in-process row cache.
if s.store != nil {
cs := s.store.GetCacheStatsTyped()
cmd/server/preparable_schema_test.go
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"database/sql"
"strings"
"testing"
)
// The statements compiled by prepareStatements reference a set of tables and
// columns. Under modernc.org/sqlite that never mattered for test fixtures:
// newStmt only stored the SQL string and compiled it lazily on first use, so a
// fixture could omit half the schema and still open. github.com/mattn/go-sqlite3
// calls sqlite3_prepare_v2 inside Prepare, so OpenDB now fails on an incomplete
// schema — which is the production behaviour we want (#1901), but it means
// fixtures have to declare what they are prepared against.
//
// ensurePreparable is the seam. Fixtures that deliberately build a partial
// schema (to exercise schema detection, say) call it on the writable connection
// just before OpenDB, and it fills in whatever is missing without disturbing the
// shapes the test actually cares about.
var preparableTables = []string{
`CREATE TABLE IF NOT EXISTS transmissions (id INTEGER PRIMARY KEY, hash TEXT)`,
`CREATE TABLE IF NOT EXISTS observations (id INTEGER PRIMARY KEY, timestamp TEXT)`,
`CREATE TABLE IF NOT EXISTS nodes (public_key TEXT PRIMARY KEY)`,
`CREATE TABLE IF NOT EXISTS observers (id TEXT)`,
}
// preparableColumns are added when absent. A fixture that already declared the
// table keeps its own definition, so only the missing columns get appended.
var preparableColumns = []struct{ table, column, decl string }{
{"transmissions", "hash", "TEXT"},
{"observations", "timestamp", "TEXT"},
{"nodes", "name", "TEXT"},
{"nodes", "role", "TEXT"},
{"nodes", "last_seen", "TEXT"},
{"observers", "inactive", "INTEGER"},
}
// ensurePreparable makes conn's schema sufficient for prepareStatements.
// It is idempotent and never overwrites an existing table or column.
func ensurePreparable(tb testing.TB, conn *sql.DB) {
tb.Helper()
for _, ddl := range preparableTables {
if _, err := conn.Exec(ddl); err != nil {
tb.Fatalf("ensurePreparable: %v\nSQL: %s", err, ddl)
}
}
for _, c := range preparableColumns {
_, err := conn.Exec("ALTER TABLE " + c.table + " ADD COLUMN " + c.column + " " + c.decl)
// "duplicate column name" means the fixture already declared it.
if err != nil && !strings.Contains(err.Error(), "duplicate column name") {
tb.Fatalf("ensurePreparable: add %s.%s: %v", c.table, c.column, err)
}
}
}
// TestEnsurePreparableMatchesPrepareStatements keeps the helper honest. If a new
// prepared statement references a table or column ensurePreparable does not
// create, it fails here rather than as a confusing OpenDB error in whichever
// fixture happens to be thinnest.
func TestEnsurePreparableMatchesPrepareStatements(t *testing.T) {
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// prepareStatements compiles against a pooled connection; pin the pool so
// the in-memory database the DDL landed in is the one it prepares against.
conn.SetMaxOpenConns(1)
ensurePreparable(t, conn)
db := &DB{conn: conn}
if err := db.prepareStatements(); err != nil {
t.Fatalf("prepareStatements against the minimal preparable schema: %v\n"+
"Add the missing table/column to preparableTables or preparableColumns.", err)
}
}
cmd/server/readonly_invariant_test.go
+41 -4
View File
@@ -10,7 +10,7 @@ import (
"strings"
"testing"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// TestServerSourceHasNoCachedRWCalls enforces issue #1287: after the
@@ -129,7 +129,7 @@ func TestServerDBConnIsReadOnly(t *testing.T) {
// Bootstrap a minimal DB with the ingestor-style WAL opener so the
// server can attach in read-only mode.
if err := bootstrapMinimalDB(path); err != nil {
if err := bootstrapMinimalDB(t, path); err != nil {
t.Fatalf("bootstrap: %v", err)
}
@@ -149,9 +149,9 @@ func TestServerDBConnIsReadOnly(t *testing.T) {
// need, opened with WAL so the read-only opener in OpenDB can attach.
// Kept in *_test.go so it does NOT add any write capability to the
// production server binary.
func bootstrapMinimalDB(path string) error {
func bootstrapMinimalDB(tb testing.TB, path string) error {
dsn := fmt.Sprintf("file:%s?_journal_mode=WAL&_busy_timeout=5000", path)
rw, err := sql.Open("sqlite", dsn)
rw, err := sql.Open("sqlite3", dsn)
if err != nil {
return err
}
@@ -159,6 +159,9 @@ func bootstrapMinimalDB(path string) error {
if _, err := rw.Exec(`CREATE TABLE IF NOT EXISTS nodes (public_key TEXT PRIMARY KEY, name TEXT)`); err != nil {
return err
}
// prepareStatements compiles eagerly under mattn; give it the rest of the
// surface it references so OpenDB gets far enough to test read-onlyness.
ensurePreparable(tb, rw)
return nil
}
@@ -211,3 +214,37 @@ func nodeTableWritePattern(verb, trailer string) *regexp.Regexp {
}
return regexp.MustCompile(expr)
}
// TestOpenDBRefusesMissingDatabase pins the behaviour that makes mode=ro
// meaningful: OpenDB must fail on a path that does not exist rather than
// creating an empty database there.
//
// This is worth a test of its own because the guarantee is not obvious from the
// call site. github.com/mattn/go-sqlite3 always passes
// SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE to sqlite3_open_v2; what restricts it
// is the mode=ro parameter in the URI, which only takes effect because mattn's
// C wrapper ORs SQLITE_OPEN_URI into the flags (sqlite3.go _sqlite3_open_v2).
// Lose the file: prefix, or the URI flag, and this silently degrades into a
// read-write open that manufactures a fresh empty database — which the server
// would then happily serve. cmd/decrypt shipped exactly that bug for a while by
// building its DSN without the file: prefix.
func TestOpenDBRefusesMissingDatabase(t *testing.T) {
dir := t.TempDir()
missing := filepath.Join(dir, "absent.db")
db, err := OpenDB(missing)
if err == nil {
db.Close()
t.Fatal("OpenDB succeeded against a nonexistent database; mode=ro is not in effect")
}
// Neither the database itself nor a file literally named after the DSN
// (what happens when URI handling is off) may have been created.
entries, rerr := os.ReadDir(dir)
if rerr != nil {
t.Fatalf("read temp dir: %v", rerr)
}
for _, e := range entries {
t.Errorf("OpenDB created %q; a read-only open must not write anything", e.Name())
}
}
cmd/server/resolved_index_test.go
+2 -2
View File
@@ -10,7 +10,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// --- Unit tests ---
@@ -1096,7 +1096,7 @@ func TestLRU_CapacityAfterBulkDelete(t *testing.T) {
// TestConfirmResolvedPathContains_SpecialChars verifies that pubkeys containing
// SQL LIKE wildcards (%, _) don't cause false positives with the INSTR approach.
func TestConfirmResolvedPathContains_SpecialChars(t *testing.T) {
db, err := sql.Open("sqlite", ":memory:")
db, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
cmd/server/scope_audit_test.go
+1 -1
View File
@@ -31,7 +31,7 @@ var (
// fiction, so the join behaves the way it does against a real database.
func setupScopeConformanceDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
conn, err := sql.Open("sqlite3", ":memory:")
if err != nil {
t.Fatal(err)
}
cmd/server/store_memory_diagnostics_test.go
+3 -3
View File
@@ -95,7 +95,7 @@ func TestObsRawHexNotRetainedOnLoad(t *testing.T) {
const txHex = "deadbeefcafe"
const obsHex = "c0ffee0102" // distinct from txHex: proves we DON'T keep it
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -109,9 +109,9 @@ func TestObsRawHexNotRetainedOnLoad(t *testing.T) {
observer_name TEXT, direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`,
`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`,
`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`,
`CREATE TABLE nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`,
`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`,
cmd/server/topology_dedup_test.go
+7 -7
View File
@@ -7,7 +7,7 @@ import (
"testing"
"time"
_ "modernc.org/sqlite"
_ "github.com/mattn/go-sqlite3"
)
// TestTopologyDedup_RepeatersMergeByPubkey verifies that topRepeaters
@@ -15,7 +15,7 @@ import (
func TestTopologyDedup_RepeatersMergeByPubkey(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -34,7 +34,7 @@ func TestTopologyDedup_RepeatersMergeByPubkey(t *testing.T) {
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, frequency REAL
@@ -139,7 +139,7 @@ func TestTopologyDedup_RepeatersMergeByPubkey(t *testing.T) {
func TestTopologyDedup_AmbiguousPrefixNotMerged(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -158,7 +158,7 @@ func TestTopologyDedup_AmbiguousPrefixNotMerged(t *testing.T) {
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, frequency REAL
@@ -245,7 +245,7 @@ func TestTopologyDedup_AmbiguousPrefixNotMerged(t *testing.T) {
func TestTopologyDedup_PairsMergeByPubkey(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
conn, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
@@ -264,7 +264,7 @@ func TestTopologyDedup_PairsMergeByPubkey(t *testing.T) {
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT)`)
exec(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT, iata TEXT, inactive INTEGER)`)
exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, frequency REAL