perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992)

Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite
3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4),
and pays the resulting cross-compilation cost with `zig cc`.

Draft because the riskiest part of this deletes rows — see [Please
review this part first](#please-review-this-part-first) — and because
three things remain unverified at the bottom.

`modernc.org/sqlite` is a transpilation of the C amalgamation. This repo
is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out
neighbour/topology/analytics queries per request, and it pays for that
transpilation on exactly those paths. Head-to-head on the same
120k-transmission / 240k-observation database, running our own hot-path
SQL under both drivers (Apple M4, `-count=5`, medians):

| workload | modernc | mattn | |
|---|---:|---:|---|
| chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms |
**2.3×** |
| aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×**
|
| 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** |

Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the
chunk load.

**Superseded by a production run.** @efiten measured both drivers on a
real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 —
as server-only containers against the same live volume, one at a time,
with round 2 reversing the order so the page cache favours the old
driver:

| | audit 7d | audit 24h | background fill (13 chunks) | start →
/api/health |
|---|---:|---:|---:|---:|
| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |

Warm, the old driver improves to 13.46s on the 7d audit and still loses
by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against
0.037s — nothing.

**So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.**
The shape the harness predicted holds — scans and joins gain, small
lookups do not — which is more reassuring than the magnitude would have
been. Quote these numbers.

**The counterweight**, cold and native on that machine: a build goes
from **52s to 163s**. An instance that builds its own image pays that
per deploy.

## The build is cgo now, and one thing about that is a trap

**`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary
dies on its first query with `go-sqlite3 requires cgo to work. This is a
stub`. A green build is not evidence of anything here, which is why
`AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely
cannot cross-compile any more.

A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc
-target {x86_64,aarch64}-linux-musl` and links static, so each artifact
stays a single self-contained file and the `alpine:3.20` runtime no
longer depends on the base image's libc at all.

`-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl
objects zig links in, and without it the server binary is 19.8MB instead
of 12.1MB.

The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU
for compilation — and gains a checksum-pinned zig plus BuildKit cache
mounts. The mounts are not a nicety: without them an image build
recompiles the amalgamation from cold and takes over half an hour.

## Please review this part first

`internal/dbschema/dedup_index.go` **deletes observation rows**. It is
the one part of this change that can lose data, and it exists because
the migration exposed a real bug rather than causing one.

`stmtInsertObservation` resolves its `ON CONFLICT` against
`idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created
inside the branch that creates the `observations` table for the first
time. Any database whose table predates that branch never got one, so
the UPSERT had no conflict target. modernc failed on the first insert;
mattn fails at `OpenStore`. Same bug, found earlier.

Creating the index unconditionally repairs it — but the index is what
was supposed to prevent duplicates, so a database that never had it can
already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this
repo holds one.** So duplicates are collapsed first. Refusing is not the
safer option: without the index the ingestor cannot prepare its UPSERT,
so it cannot start at all.

Replaying that UPSERT faithfully is subtler than it looks, and a first
cut of this got it wrong twice:

- `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a
group in id order the survivor keeps the **last** non-NULL value. Taking
the first silently discarded newer readings.
- The UPSERT names exactly five columns (`snr`, `rssi`, `score`,
`raw_hex`, `resolved_path`). Every other column must keep the surviving
row's own value; merging those too invents history the ingestor would
never have written.

Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split
apart, a writer inserting a duplicate in the gap fails the index
creation while leaving the deletions committed — rows destroyed and no
index to show for it.

Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**,
holding the write lock throughout, once, at ingestor startup before MQTT
subscribe. Materialising the duplicate-group scan once rather than per
column took that from 9.7s; the pathological case (400k of 600k rows
duplicated) is 5.7s, slightly worse than the 4.2s it was before that
change.

## Four more behavioural differences

Full detail in `docs/sqlite-driver-migration.md`. Briefly:

**Statement preparation is eager.** modernc's `newStmt` stored the SQL
and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`,
so SQL naming a missing table fails at *open*. 59 server tests failed on
this alone, all fixtures with partial schemas. `OpenDB` keeps failing
loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the
fixtures now declare what they are prepared against via
`ensurePreparable`. This also exposed nine `nodes(pubkey …)`
declarations across seven files, where production has only ever had
`public_key` — lazy compilation had hidden the mismatch for as long as
it existed.

**`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to
NORMAL and executes the pragma unconditionally, where SQLite's own
default (what modernc left alone) is FULL. In WAL mode that weakens
durability under power loss. Pinned in `dbschema.WriterDSN`, which both
writers now share — `cmd/migrate` kept a bare path at first and so
quietly wrote at NORMAL, which is what a second copy of a DSN buys you.

**The DSN dialects are mutually invisible.** modernc understood only
`_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither
errors on the other's form — a driver-only rename would have dropped
every pragma in silence. `_journal_mode=WAL` is also gone from the
server's read handle: modernc ignored it, mattn honours it, and setting
`journal_mode` on a read-only connection is a write. Dropping
`_busy_timeout` with it costs nothing, since mattn already defaults to
5000ms — which means the read handle finally *gets* the busy timeout it
had silently lacked.

**`mode=ro` survives for a non-obvious reason.** mattn always passes
`READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what
makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the
#1283/#1289 invariant holds with no build flags — but it depends on the
`file:` prefix. `cmd/decrypt` had been building its DSN without one, so
its `mode=ro` had never applied and a missing path was created
read-write. Fixed in passing; never a migration regression.

## What did not change

No modernc-specific API was in use: no `RegisterFunction`, no
`*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No
`time.Time` is ever bound as a query argument, so driver time handling
is not in play. Both drivers convert declared
`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so
`/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an
earlier draft "fixed" that with a `CAST` and would have been the
regression.

## Tests and CI

New regression tests, each written because something got through without
it:

- `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge
ordering. The original test used complementary NULLs, which passes
whichever direction you pick, which is why the bug survived it.
- `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation
must roll the deletions back.
- `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma,
read back through the store's own connection. A separate `sqlite3`
session or the startup log line would prove nothing.
- `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which
now rests on a detail of the driver's C wrapper.
- `TestEnsurePreparableMatchesPrepareStatements` — fails when a new
prepared statement outgrows the fixture helper.

CI gains test execution for `cmd/migrate` and `internal/dbschema`, which
had none and both open the database. A PR-time two-arch build plus an
arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without
it nothing on a PR would exercise zig, static musl linking or arm64, and
the first signal would arrive on master. `cache-dependency-path` widens
from 2 of the 5 tracked `go.sum` files to all of them.

`make test` passes across all 14 modules, `cmd/server` also under `-race
-count=2` with no failures and no races. `gofmt` and `go vet` clean.
Release-routing and Dockerfile COPY-invariant gates pass.

## Verified by running

- All 8 cross-builds static and correct-architecture; both arches of the
container image built, exported and run under QEMU, serving
`/api/health` and `/api/nodes` against a 2.9M-observation production
snapshot.
- The `migrate` binary repairing that snapshot's duplicate on bare
Alpine.
- `CGO_ENABLED=0` producing a binary that builds and then fails on first
query.

## Not verified

- ~~The 2–2.3× figures come from a standalone harness, not this load
under the old driver.~~ **Closed** by @efiten's production run above,
which also corrected the multiplier.
- SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no
total `ORDER BY`.
- Sustained live ingest through the new writer DSN, and the duplicate
collapse against a database an ingestor is actively writing to. Verified
against a static snapshot only, and the collapse is measured at 4.1s on
2.4M synthetic rows with 5 duplicates — well short of an 11M-row
instance. @efiten has offered a staging instance taking real MQTT
traffic; **this is the item to close before the PR leaves draft.**

An earlier revision of this branch shipped the dedup merge in the wrong
direction with a green test suite, and review then found three more
things in the same file: the repair gated on an error string, a
non-atomic TEMP table drop aimed at the wrong connection, and a deletion
whose only record was a row count. All fixed in ac7e8d38. Passing tests
did not establish safety here, which is why the deletion path wanted a
second pair of eyes rather than a rubber stamp.

docs/sqlite-driver-migration.md

@@ -0,0 +1,245 @@
+# SQLite driver: modernc.org/sqlite → github.com/mattn/go-sqlite3
+
+The database driver changed from `modernc.org/sqlite v1.34.5` (pure Go, SQLite
+3.46.0) to `github.com/mattn/go-sqlite3 v1.14.52` (cgo, bundled SQLite 3.53.4).
+
+This is the record of why, what it measured, and the five behavioural
+differences that had to be handled — several of which fail silently if you get
+them wrong, so read this before touching the DSNs or the build.
+
+## Why
+
+corescope is read-heavy: the server chunk-loads a graph at startup and fans out
+neighbor/topology/analytics queries per request. modernc's pure-Go SQLite is a
+transpilation of the C amalgamation and pays for it on exactly those paths.
+
+## What it measured
+
+Head-to-head on the same 120k-transmission / 240k-observation database, running
+corescope's own hot-path SQL under both drivers (Apple M4, `-count=5`, medians):
+
+| Workload | modernc | mattn | Change |
+|---|---:|---:|---|
+| Chunk load (the `cmd/server/chunked_load.go` v3 join, 20k transmissions) | 449 ms | 196 ms | **2.3× faster** |
+| Aggregate scan (240k-row join + `GROUP BY`, stands in for analytics) | 276 ms | 137 ms | **2.0× faster** |
+| 1500 prepared-statement lookups (`prepareStatements` round trips) | 512 ms | 403 ms | **1.3× faster** |
+
+Allocation counts drop with it: 1.12M vs 1.64M allocs (−32%) and 21 MB vs 30 MB
+(−30%) on the chunk load, 20.3k vs 27.3k on the lookups.
+
+## Confirmed on production
+
+The table above comes from a standalone harness. @efiten then ran both drivers
+against a real instance — 11,077,038 observations, 9.7 GB database, 4-core arm64
+— as server-only containers against the same live volume, one at a time. Round 2
+reverses the order so the page-cache advantage goes to the old driver:
+
+| | audit 7d | audit 24h | background fill (13 chunks) | start → /api/health |
+|---|---:|---:|---:|---:|
+| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
+| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
+| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
+| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |
+
+Warm, the old driver improves to 13.46s on the 7d audit and still loses by
+~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against 0.037s —
+i.e. nothing.
+
+**Real-world gains are smaller than the harness suggests: ~1.4-1.8× on the
+paths that matter, not 2-2.3×.** The shape holds — scans and joins gain, small
+lookups do not — but quote these numbers, not the harness ones.
+
+**Build time is the counterweight**, cold and native on that machine:
+**52s on master, 163s on this branch**. An instance that builds its own image
+pays that on every deploy.
+
+## The cost: the build is cgo now
+
+`CGO_ENABLED=0` still *builds*, which is the trap: mattn links a stub, and the
+binary dies on its first query with `go-sqlite3 requires cgo to work. This is a
+stub`. A green build proves nothing here. `GOOS=linux go build` from a Mac, on
+the other hand, genuinely cannot cross-compile any more — cgo needs a C compiler
+that can target the other platform. That compiler is
+[`zig`](https://ziglang.org/download/):
+
+```bash
+make build        # host
+make crossbuild   # static linux/amd64 + linux/arm64 via `zig cc -target …-linux-musl`
+```
+
+Targeting musl and linking with `-extldflags "-static -Wl,-s"` keeps the output a
+single self-contained binary, so the `alpine:3.20` runtime image no longer
+depends on the base image's libc at all. `-Wl,-s` matters: Go's own `-s -w` does
+not reach the musl objects zig links in, and without it the server binary is
+19.8 MB instead of 12.1 MB.
+
+The `Dockerfile` builder stage installs a checksum-pinned zig and does the same
+thing, still on a single `$BUILDPLATFORM` builder with no QEMU for compilation.
+Build-cache mounts are not optional there: compiling the SQLite amalgamation
+twice from cold takes over half an hour.
+
+## The five behavioural differences
+
+### 1. Statement preparation is eager
+
+modernc's `newStmt` only stored the SQL and compiled lazily on first use; mattn
+calls `sqlite3_prepare_v2` inside `Prepare`. SQL referencing a missing table or
+column now fails at **open** time.
+
+This is the migration's largest single effect: 59 server tests failed on it,
+purely from fixtures with partial schemas. `OpenDB` keeps failing loudly (that is
+the #1901 behaviour we want in production, and `cmd/server/main.go` also gates on
+`dbschema.AssertReady`); the fixtures instead declare what they are prepared
+against, via `ensurePreparable` in `cmd/server/preparable_schema_test.go`. If you
+add a prepared statement referencing something new,
+`TestEnsurePreparableMatchesPrepareStatements` tells you to extend that helper.
+
+It also surfaced nine `nodes(pubkey …)` declarations across seven files, when
+production has only ever had `public_key`. Lazy compilation had hidden the
+mismatch.
+
+### 2. It exposed a real bug in the observation UPSERT
+
+`stmtInsertObservation` resolves `ON CONFLICT(transmission_id, observer_idx,
+COALESCE(path_json, ''))` against the unique expression index
+`idx_observations_dedup` — which `cmd/ingestor/db.go` only ever created inside
+the branch that creates the `observations` table for the first time. Databases
+whose table predates that branch never had one, so the UPSERT had no conflict
+target. modernc failed on the first insert; mattn fails at `OpenStore`. Same bug,
+found earlier.
+
+`internal/dbschema` now creates it unconditionally. Because the index is what was
+supposed to prevent duplicates, a database that never had it can already hold
+rows violating it — the repo's own `test-fixtures/e2e-fixture.db` held one — so
+duplicates are collapsed first. Refusing would not have been safer: without the
+index the ingestor cannot prepare its UPSERT, so it cannot start at all.
+
+The collapse has to *replay* the UPSERT, and getting that subtly wrong is easy.
+`DO UPDATE SET snr = COALESCE(excluded.snr, snr)` means the **incoming** value
+wins when it is non-NULL, so down a group in id order the survivor ends up with
+the **last** non-NULL value, not the first. It also names exactly five columns —
+`snr`, `rssi`, `score`, `raw_hex`, `resolved_path` — so every other column keeps
+the surviving row's own value; merging those too would invent history the
+ingestor would never have written. An earlier version of this change took the
+first non-NULL value and merged every column, which silently discarded newer
+readings.
+
+Merge, delete and index creation all share one transaction. Split apart, a
+writer inserting a duplicate in the gap makes the index creation fail while the
+deletions stay committed — rows destroyed and no index to show for it.
+
+The repair logs what it is about to destroy — group count, rows to remove, and
+the first 20 group keys with the id it keeps — before deleting anything, because
+a bare row count is not enough to reconstruct from if the merge direction is
+ever wrong again. It also says up front that it holds the write lock, since on a
+large table that pause at ingestor startup is otherwise unexplained.
+
+Two hazards worth knowing before an upgrade:
+
+- **Reads inside the repair must use the transaction, not the pool.**
+  `cmd/ingestor` runs `SetMaxOpenConns(1)`, so a query issued against the pool
+  while the repair holds its transaction waits for a connection that
+  transaction has checked out, forever. It deadlocked a staging ingestor at
+  boot — logs stop after "Repairing now", the write lock is free, the process
+  is simply blocked. Anything reading in there takes a `Querier` and is passed
+  `tx`.
+- **NULL is not a duplicate.** `GROUP BY` folds NULLs into one group; a UNIQUE
+  index treats them as distinct, so a row with a NULL in an indexed column can
+  never violate it. Grouping without excluding them does not delete the rows —
+  the `DELETE` joins on `observer_idx = observer_idx`, which NULL never
+  satisfies — it is the *merge* that does the damage, matching nothing and
+  writing NULL over the survivor's real readings. The rows stay put and their
+  measurements disappear. On an 11.2M-row instance 198 of 222 reported groups
+  were `observer_idx IS NULL`.
+
+The failing `CREATE UNIQUE INDEX` that triggers all this is itself a stall: on
+11.2M rows it held the write lock long enough for a concurrent writer to hit the
+full 5s `busy_timeout`. So an instance that needs the repair pauses writers for
+seconds *before* the repair starts.
+
+The repair itself does not hold the write lock throughout, tempting as that is
+to assume. The grouping scan (~18s at that size) reads, and a concurrent writer
+can proceed during it; the lock is taken when the repair first writes. A writer
+that gets in between makes the repair fail and roll back, which is the intended
+outcome — but "holds the write lock until it completes" is the wrong mental
+model.
+
+The collapse itself is measured at 4.1s on 2.4M synthetic rows holding 5
+duplicates. That is well short of a real deployment: an 11M-row instance has not
+been measured to completion, and the collapse has not been run against a
+database an ingestor is actively writing to.
+
+Note `COALESCE(path_json, '')` makes a NULL path and an empty-string path the
+same key, but `NULL` and `'[]'` different keys. See
+`internal/dbschema/dedup_index_test.go`, where the last-wins and atomicity
+properties each have a test.
+
+### 3. `synchronous` silently dropped from FULL to NORMAL
+
+mattn defaults `synchronousMode` to `NORMAL` and runs `PRAGMA synchronous =
+NORMAL` unconditionally, where SQLite's own compile default (what modernc left in
+place) is FULL. In WAL mode that changes durability under power loss.
+
+The writer DSN pins `_synchronous=FULL`, and lives in one place —
+`dbschema.WriterDSN` — because there are two writers. `cmd/migrate` originally
+kept a bare path and so silently wrote at NORMAL, which is exactly what a second
+copy of a DSN buys you. `TestOpenStorePragmas` reads every pragma back
+**through the store's own connection**, and `TestWriterDSNPragmas` covers the DSN
+itself; a separate `sqlite3` session or the startup log line would prove
+nothing.
+
+### 4. The DSN dialect is different, and each driver ignores the other's
+
+modernc understood only `_pragma=name(value)`; mattn understands only
+`_`-prefixed parameters. Neither errors on the other's form, so a driver-only
+rename would have dropped every pragma on the floor in silence. All five
+`_pragma=` DSNs were rewritten (one production, four test seeds).
+
+Also removed: `_journal_mode=WAL` on the server's read handle. modernc had been
+ignoring it all along; mattn honours it, and setting `journal_mode` on a
+read-only connection is a write. Dropping `_busy_timeout` with it costs nothing —
+mattn's default is already 5000 ms, so the read handle finally *gets* the busy
+timeout it had silently lacked.
+
+### 5. `mode=ro` still works — but not for the reason you would guess
+
+mattn always passes `SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE` to
+`sqlite3_open_v2`, and the bundled amalgamation has `SQLITE_USE_URI=0`. What
+makes `file:…?mode=ro` work anyway is that mattn's C wrapper `_sqlite3_open_v2`
+ORs `SQLITE_OPEN_URI` into the flags itself. So the read-only invariant from
+#1283/#1289 holds with no build flags — but it depends on the `file:` prefix
+being present. `TestOpenDBRefusesMissingDatabase` fails if any of that stops
+holding.
+
+`cmd/decrypt` had been building its DSN *without* the `file:` prefix, so both
+drivers stripped the query string and its `mode=ro` had never applied — a missing
+path was created read-write. Fixed in passing; it was never a migration
+regression.
+
+## Memory
+
+`runtime/debug.SetMemoryLimit` (GOMEMLIMIT) covers what the Go runtime manages —
+heap, stacks, runtime structures — and nothing else, so it is not an RSS ceiling
+now that SQLite allocates in C.
+
+What is bounded is the page cache specifically: both DSNs pin
+`_cache_size=-2000`, i.e. ~2 MiB per connection, so ~8 MiB across the server's
+`SetMaxOpenConns(4)` and ~2 MiB in the ingestor, comfortably inside the 1.5×
+headroom `applyMemoryLimit` derives. That caps the page cache, not everything
+SQLite allocates — statement and schema memory sit outside it — so revisit
+against measured RSS if the connection count or `_cache_size` grows, or if a
+workload starts holding many prepared statements.
+
+There is no cgo-bytes metric because Go exposes no counter for one. Do not read
+`processRSSMB - goSysMB` as the C share either: `goSysMB` is reserved address
+space rather than resident memory, so the subtraction mixes two different
+quantities. It is a smell test, not a measurement.
+
+## Things that did not change
+
+No modernc-specific API was in use — no `RegisterFunction`, no `*sqlite.Conn`, no
+`modernc.org/sqlite/lib` error constants, no `sql.Register`. No `time.Time` is
+ever bound as a query argument (every retention cutoff is pre-formatted), so
+driver time handling is not in play. Both drivers convert declared
+`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so `/api/dropped-packets`
+keeps emitting `dropped_at` as RFC3339 exactly as before.