# SQLite driver: modernc.org/sqlite → github.com/mattn/go-sqlite3 The database driver changed from `modernc.org/sqlite v1.34.5` (pure Go, SQLite 3.46.0) to `github.com/mattn/go-sqlite3 v1.14.52` (cgo, bundled SQLite 3.53.4). This is the record of why, what it measured, and the five behavioural differences that had to be handled — several of which fail silently if you get them wrong, so read this before touching the DSNs or the build. ## Why corescope is read-heavy: the server chunk-loads a graph at startup and fans out neighbor/topology/analytics queries per request. modernc's pure-Go SQLite is a transpilation of the C amalgamation and pays for it on exactly those paths. ## What it measured Head-to-head on the same 120k-transmission / 240k-observation database, running corescope's own hot-path SQL under both drivers (Apple M4, `-count=5`, medians): | Workload | modernc | mattn | Change | |---|---:|---:|---| | Chunk load (the `cmd/server/chunked_load.go` v3 join, 20k transmissions) | 449 ms | 196 ms | **2.3× faster** | | Aggregate scan (240k-row join + `GROUP BY`, stands in for analytics) | 276 ms | 137 ms | **2.0× faster** | | 1500 prepared-statement lookups (`prepareStatements` round trips) | 512 ms | 403 ms | **1.3× faster** | Allocation counts drop with it: 1.12M vs 1.64M allocs (−32%) and 21 MB vs 30 MB (−30%) on the chunk load, 20.3k vs 27.3k on the lookups. ## Confirmed on production The table above comes from a standalone harness. @efiten then ran both drivers against a real instance — 11,077,038 observations, 9.7 GB database, 4-core arm64 — as server-only containers against the same live volume, one at a time. Round 2 reverses the order so the page-cache advantage goes to the old driver: | | audit 7d | audit 24h | background fill (13 chunks) | start → /api/health | |---|---:|---:|---:|---:| | modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s | | mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s | | mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s | | modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s | Warm, the old driver improves to 13.46s on the 7d audit and still loses by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against 0.037s — i.e. nothing. **Real-world gains are smaller than the harness suggests: ~1.4-1.8× on the paths that matter, not 2-2.3×.** The shape holds — scans and joins gain, small lookups do not — but quote these numbers, not the harness ones. **Build time is the counterweight**, cold and native on that machine: **52s on master, 163s on this branch**. An instance that builds its own image pays that on every deploy. ## The cost: the build is cgo now `CGO_ENABLED=0` still *builds*, which is the trap: mattn links a stub, and the binary dies on its first query with `go-sqlite3 requires cgo to work. This is a stub`. A green build proves nothing here. `GOOS=linux go build` from a Mac, on the other hand, genuinely cannot cross-compile any more — cgo needs a C compiler that can target the other platform. That compiler is [`zig`](https://ziglang.org/download/): ```bash make build # host make crossbuild # static linux/amd64 + linux/arm64 via `zig cc -target …-linux-musl` ``` Targeting musl and linking with `-extldflags "-static -Wl,-s"` keeps the output a single self-contained binary, so the `alpine:3.20` runtime image no longer depends on the base image's libc at all. `-Wl,-s` matters: Go's own `-s -w` does not reach the musl objects zig links in, and without it the server binary is 19.8 MB instead of 12.1 MB. The `Dockerfile` builder stage installs a checksum-pinned zig and does the same thing, still on a single `$BUILDPLATFORM` builder with no QEMU for compilation. Build-cache mounts are not optional there: compiling the SQLite amalgamation twice from cold takes over half an hour. ## The five behavioural differences ### 1. Statement preparation is eager modernc's `newStmt` only stored the SQL and compiled lazily on first use; mattn calls `sqlite3_prepare_v2` inside `Prepare`. SQL referencing a missing table or column now fails at **open** time. This is the migration's largest single effect: 59 server tests failed on it, purely from fixtures with partial schemas. `OpenDB` keeps failing loudly (that is the #1901 behaviour we want in production, and `cmd/server/main.go` also gates on `dbschema.AssertReady`); the fixtures instead declare what they are prepared against, via `ensurePreparable` in `cmd/server/preparable_schema_test.go`. If you add a prepared statement referencing something new, `TestEnsurePreparableMatchesPrepareStatements` tells you to extend that helper. It also surfaced nine `nodes(pubkey …)` declarations across seven files, when production has only ever had `public_key`. Lazy compilation had hidden the mismatch. ### 2. It exposed a real bug in the observation UPSERT `stmtInsertObservation` resolves `ON CONFLICT(transmission_id, observer_idx, COALESCE(path_json, ''))` against the unique expression index `idx_observations_dedup` — which `cmd/ingestor/db.go` only ever created inside the branch that creates the `observations` table for the first time. Databases whose table predates that branch never had one, so the UPSERT had no conflict target. modernc failed on the first insert; mattn fails at `OpenStore`. Same bug, found earlier. `internal/dbschema` now creates it unconditionally. Because the index is what was supposed to prevent duplicates, a database that never had it can already hold rows violating it — the repo's own `test-fixtures/e2e-fixture.db` held one — so duplicates are collapsed first. Refusing would not have been safer: without the index the ingestor cannot prepare its UPSERT, so it cannot start at all. The collapse has to *replay* the UPSERT, and getting that subtly wrong is easy. `DO UPDATE SET snr = COALESCE(excluded.snr, snr)` means the **incoming** value wins when it is non-NULL, so down a group in id order the survivor ends up with the **last** non-NULL value, not the first. It also names exactly five columns — `snr`, `rssi`, `score`, `raw_hex`, `resolved_path` — so every other column keeps the surviving row's own value; merging those too would invent history the ingestor would never have written. An earlier version of this change took the first non-NULL value and merged every column, which silently discarded newer readings. Merge, delete and index creation all share one transaction. Split apart, a writer inserting a duplicate in the gap makes the index creation fail while the deletions stay committed — rows destroyed and no index to show for it. The repair logs what it is about to destroy — group count, rows to remove, and the first 20 group keys with the id it keeps — before deleting anything, because a bare row count is not enough to reconstruct from if the merge direction is ever wrong again. It also says up front that it holds the write lock, since on a large table that pause at ingestor startup is otherwise unexplained. Two hazards worth knowing before an upgrade: - **Reads inside the repair must use the transaction, not the pool.** `cmd/ingestor` runs `SetMaxOpenConns(1)`, so a query issued against the pool while the repair holds its transaction waits for a connection that transaction has checked out, forever. It deadlocked a staging ingestor at boot — logs stop after "Repairing now", the write lock is free, the process is simply blocked. Anything reading in there takes a `Querier` and is passed `tx`. - **NULL is not a duplicate.** `GROUP BY` folds NULLs into one group; a UNIQUE index treats them as distinct, so a row with a NULL in an indexed column can never violate it. Grouping without excluding them does not delete the rows — the `DELETE` joins on `observer_idx = observer_idx`, which NULL never satisfies — it is the *merge* that does the damage, matching nothing and writing NULL over the survivor's real readings. The rows stay put and their measurements disappear. On an 11.2M-row instance 198 of 222 reported groups were `observer_idx IS NULL`. The failing `CREATE UNIQUE INDEX` that triggers all this is itself a stall: on 11.2M rows it held the write lock long enough for a concurrent writer to hit the full 5s `busy_timeout`. So an instance that needs the repair pauses writers for seconds *before* the repair starts. The repair itself does not hold the write lock throughout, tempting as that is to assume. The grouping scan (~18s at that size) reads, and a concurrent writer can proceed during it; the lock is taken when the repair first writes. A writer that gets in between makes the repair fail and roll back, which is the intended outcome — but "holds the write lock until it completes" is the wrong mental model. The collapse itself is measured at 4.1s on 2.4M synthetic rows holding 5 duplicates. That is well short of a real deployment: an 11M-row instance has not been measured to completion, and the collapse has not been run against a database an ingestor is actively writing to. Note `COALESCE(path_json, '')` makes a NULL path and an empty-string path the same key, but `NULL` and `'[]'` different keys. See `internal/dbschema/dedup_index_test.go`, where the last-wins and atomicity properties each have a test. ### 3. `synchronous` silently dropped from FULL to NORMAL mattn defaults `synchronousMode` to `NORMAL` and runs `PRAGMA synchronous = NORMAL` unconditionally, where SQLite's own compile default (what modernc left in place) is FULL. In WAL mode that changes durability under power loss. The writer DSN pins `_synchronous=FULL`, and lives in one place — `dbschema.WriterDSN` — because there are two writers. `cmd/migrate` originally kept a bare path and so silently wrote at NORMAL, which is exactly what a second copy of a DSN buys you. `TestOpenStorePragmas` reads every pragma back **through the store's own connection**, and `TestWriterDSNPragmas` covers the DSN itself; a separate `sqlite3` session or the startup log line would prove nothing. ### 4. The DSN dialect is different, and each driver ignores the other's modernc understood only `_pragma=name(value)`; mattn understands only `_`-prefixed parameters. Neither errors on the other's form, so a driver-only rename would have dropped every pragma on the floor in silence. All five `_pragma=` DSNs were rewritten (one production, four test seeds). Also removed: `_journal_mode=WAL` on the server's read handle. modernc had been ignoring it all along; mattn honours it, and setting `journal_mode` on a read-only connection is a write. Dropping `_busy_timeout` with it costs nothing — mattn's default is already 5000 ms, so the read handle finally *gets* the busy timeout it had silently lacked. ### 5. `mode=ro` still works — but not for the reason you would guess mattn always passes `SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE` to `sqlite3_open_v2`, and the bundled amalgamation has `SQLITE_USE_URI=0`. What makes `file:…?mode=ro` work anyway is that mattn's C wrapper `_sqlite3_open_v2` ORs `SQLITE_OPEN_URI` into the flags itself. So the read-only invariant from #1283/#1289 holds with no build flags — but it depends on the `file:` prefix being present. `TestOpenDBRefusesMissingDatabase` fails if any of that stops holding. `cmd/decrypt` had been building its DSN *without* the `file:` prefix, so both drivers stripped the query string and its `mode=ro` had never applied — a missing path was created read-write. Fixed in passing; it was never a migration regression. ## Memory `runtime/debug.SetMemoryLimit` (GOMEMLIMIT) covers what the Go runtime manages — heap, stacks, runtime structures — and nothing else, so it is not an RSS ceiling now that SQLite allocates in C. What is bounded is the page cache specifically: both DSNs pin `_cache_size=-2000`, i.e. ~2 MiB per connection, so ~8 MiB across the server's `SetMaxOpenConns(4)` and ~2 MiB in the ingestor, comfortably inside the 1.5× headroom `applyMemoryLimit` derives. That caps the page cache, not everything SQLite allocates — statement and schema memory sit outside it — so revisit against measured RSS if the connection count or `_cache_size` grows, or if a workload starts holding many prepared statements. There is no cgo-bytes metric because Go exposes no counter for one. Do not read `processRSSMB - goSysMB` as the C share either: `goSysMB` is reserved address space rather than resident memory, so the subtraction mixes two different quantities. It is a smell test, not a measurement. ## Things that did not change No modernc-specific API was in use — no `RegisterFunction`, no `*sqlite.Conn`, no `modernc.org/sqlite/lib` error constants, no `sql.Register`. No `time.Time` is ever bound as a query argument (every retention cutoff is pre-formatted), so driver time handling is not in play. Both drivers convert declared `DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so `/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 exactly as before.