Closes#921
## Summary
Follow-up to #920 (incremental auto-vacuum). Addresses both items from
the adversarial review:
### 1. RW connection caching
Previously, every call to `openRW(dbPath)` opened a new SQLite RW
connection and closed it after use. This happened in:
- `runIncrementalVacuum` (~4x/hour)
- `PruneOldPackets`, `PruneOldMetrics`, `RemoveStaleObservers`
- `buildAndPersistEdges`, `PruneNeighborEdges`
- All neighbor persist operations
Now a single `*sql.DB` handle (with `MaxOpenConns(1)`) is cached
process-wide via `cachedRW(dbPath)`. The underlying connection pool
manages serialization. The original `openRW()` function is retained for
one-shot test usage.
### 2. DBConfig dedup
`DBConfig` was defined identically in both `cmd/server/config.go` and
`cmd/ingestor/config.go`. Extracted to `internal/dbconfig/` as a shared
package; both binaries now use a type alias (`type DBConfig =
dbconfig.DBConfig`).
## Tests added
| Test | File |
|------|------|
| `TestCachedRW_ReturnsSameHandle` | `cmd/server/rw_cache_test.go` |
| `TestCachedRW_100Calls_SingleConnection` |
`cmd/server/rw_cache_test.go` |
| `TestGetIncrementalVacuumPages_Default` |
`internal/dbconfig/dbconfig_test.go` |
| `TestGetIncrementalVacuumPages_Configured` |
`internal/dbconfig/dbconfig_test.go` |
## Verification
```
ok github.com/corescope/server 20.069s
ok github.com/corescope/ingestor 47.117s
ok github.com/meshcore-analyzer/dbconfig 0.003s
```
Both binaries build cleanly. 100 sequential `cachedRW()` calls return
the same handle with exactly 1 entry in the cache map.
---------
Co-authored-by: you <you@example.com>
Closes#919
## Summary
Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.
## Changes
### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this
### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor
### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility
### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM
### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides
## Migration path for existing databases
1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration
## Test results
```
ok github.com/corescope/server 19.448s
ok github.com/corescope/ingestor 30.682s
```
---------
Co-authored-by: you <you@example.com>
Kpa-clawbot