## Problem
`Load()` loaded all transmissions from the DB regardless of
`retentionHours`, so `buildSubpathIndex()` processed the full DB history
on every startup. On a DB with ~280K paths this produces ~13.5M subpath
index entries, OOM-killing the process before it ever starts listening —
causing a supervisord crash loop with no useful error message.
## Fix
Apply the same `retentionHours` cutoff to `Load()`'s SQL that
`EvictStale()` already uses at runtime. Both conditions
(`retentionHours` window and `maxPackets` cap) are combined with AND so
neither safety limit is bypassed.
Startup now builds indexes only over the retention window, making
startup time and memory proportional to recent activity rather than
total DB history.
## Docs
- `config.example.json`: adds `retentionHours` to the `packetStore`
block with recommended value `168` (7 days) and a warning about `0` on
large DBs
- `docs/user-guide/configuration.md`: documents the field and adds an
explicit OOM warning
## Test plan
- [x] `cd cmd/server && go test ./... -run TestRetentionLoad` — covers
the retention-filtered load: verifies packets outside the window are
excluded, and that `retentionHours: 0` still loads everything
- [x] Deploy on an instance with a large DB (>100K paths) and
`retentionHours: 168` — server reaches "listening" in seconds instead of
OOM-crashing
- [x] Verify `config.example.json` has `retentionHours: 168` in the
`packetStore` block
- [x] Verify `docs/user-guide/configuration.md` documents the field and
warning
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Closes#919
## Summary
Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.
## Changes
### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this
### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor
### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility
### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM
### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides
## Migration path for existing databases
1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration
## Test results
```
ok github.com/corescope/server 19.448s
ok github.com/corescope/ingestor 30.682s
```
---------
Co-authored-by: you <you@example.com>
## Summary
- Add missing `geo_filter` block to `config.example.json` with polygon
example, `bufferKm`, and inline `_comment`
- Add `docs/user-guide/geofilter.md`: full operator guide covering
config schema, GeoFilter Builder workflow, and prune script as one-time
migration tool
- Add Geographic filtering section to `docs/user-guide/configuration.md`
with link to the full guide
Closes#669 (M1: documentation)
## Test plan
- [x] `config.example.json` parses cleanly (no JSON errors)
- [x] `docs/user-guide/geofilter.md` renders correctly in GitHub preview
- [x] Link from `configuration.md` to `geofilter.md` resolves
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
efiten