fix: enable SQLite incremental auto-vacuum so DB shrinks after retention (#919) (#920)

Closes #919

## Summary

Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.

## Changes

### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this

### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor

### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility

### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM

### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides

## Migration path for existing databases

1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration

## Test results

```
ok  github.com/corescope/server    19.448s
ok  github.com/corescope/ingestor  30.682s
```

---------

Co-authored-by: you <you@example.com>

docs/user-guide/database.md

@@ -0,0 +1,82 @@
+# Database
+
+CoreScope uses SQLite in WAL (Write-Ahead Log) mode for both the server
+(read-only) and ingestor (read-write).
+
+## WAL mode
+
+WAL mode allows concurrent reads while writes happen. It is set automatically
+at connection time via `PRAGMA journal_mode=WAL`. No operator action needed.
+
+The WAL file (`meshcore.db-wal`) grows during writes and is checkpointed
+(merged back into the main DB) periodically and at clean shutdown.
+
+## Auto-vacuum
+
+By default, SQLite does not shrink the database file after `DELETE` operations.
+Deleted pages are marked free and reused by future writes, but the file size
+on disk stays the same. This is surprising when lowering retention settings.
+
+### New databases
+
+Databases created after this feature was added automatically have
+`PRAGMA auto_vacuum = INCREMENTAL`. After each retention reaper cycle,
+CoreScope runs `PRAGMA incremental_vacuum(N)` to return free pages to the OS.
+
+### Existing databases
+
+The `auto_vacuum` mode is stored in the database header and can only be changed
+by rewriting the entire file with `VACUUM`. CoreScope will **not** do this
+automatically — on large databases (5+ GB seen in the wild) it takes minutes
+and holds an exclusive lock.
+
+**To migrate an existing database:**
+
+1. At startup, CoreScope logs a warning:
+   ```
+   [db] auto_vacuum=NONE — DB needs one-time VACUUM to enable incremental auto-vacuum.
+   ```
+2. **Ensure at least 2× the database file size in free disk space.** Full VACUUM
+   creates a temporary copy of the entire file — on a near-full disk it will fail.
+3. Set `db.vacuumOnStartup: true` in your `config.json`:
+   ```json
+   {
+     "db": {
+       "vacuumOnStartup": true
+     }
+   }
+   ```
+4. Restart CoreScope. The one-time `VACUUM` will run and block startup.
+5. After migration, remove or set `vacuumOnStartup: false` — it's not needed again.
+
+### Configuration
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `db.vacuumOnStartup` | `false` | One-time full VACUUM to enable incremental auto-vacuum |
+| `db.incrementalVacuumPages` | `1024` | Pages returned to OS per reaper cycle |
+
+## Manual VACUUM
+
+You can also run a manual vacuum from the SQLite CLI:
+
+```bash
+sqlite3 data/meshcore.db "PRAGMA auto_vacuum = INCREMENTAL; VACUUM;"
+```
+
+This is equivalent to `vacuumOnStartup: true` but can be done offline.
+
+> ⚠️ Full VACUUM requires **2× the database file size** in free disk space (it
+> creates a temporary copy). Check with `ls -lh data/meshcore.db` before running.
+
+## Checking current mode
+
+```bash
+sqlite3 data/meshcore.db "PRAGMA auto_vacuum;"
+```
+
+- `0` = NONE (default for old databases)
+- `1` = FULL (automatic, but slower writes)
+- `2` = INCREMENTAL (recommended — CoreScope triggers vacuum after deletes)
+
+See [#919](https://github.com/Kpa-clawbot/CoreScope/issues/919) for background on this feature.