mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-05-12 14:24:43 +00:00
Kpa-clawbot
aeae7813bc
Closes #919 ## Summary Enables SQLite incremental auto-vacuum so the database file actually shrinks after retention reaper deletes old data. Previously, `DELETE` operations freed pages internally but never returned disk space to the OS. ## Changes ### 1. Auto-vacuum on new databases - `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before `journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval` - Must be set before any tables are created; DSN ordering ensures this ### 2. Post-reaper incremental vacuum - `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle (packets, metrics, observers, neighbor edges) - N defaults to 1024 pages, configurable via `db.incrementalVacuumPages` - Noop on `auto_vacuum=NONE` databases (safe before migration) - Added to both server and ingestor ### 3. Opt-in full VACUUM for existing databases - Startup check logs a clear warning if `auto_vacuum != INCREMENTAL` - `db.vacuumOnStartup: true` config triggers one-time `PRAGMA auto_vacuum = INCREMENTAL; VACUUM` - Logs start/end time for operator visibility ### 4. Documentation - `docs/user-guide/configuration.md`: retention section notes that lowering retention doesn't immediately shrink the DB - `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum, migration, manual VACUUM ### 5. Tests - `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2` - `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0` - `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2` - `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks freelist - `TestCheckAutoVacuumLogs` — handles both modes without panic - `TestConfigIncrementalVacuumPages` — config defaults and overrides ## Migration path for existing databases 1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs one-time VACUUM...` 2. Set `db.vacuumOnStartup: true` in config.json 3. Restart — VACUUM runs (blocks startup, minutes on large DBs) 4. Remove `vacuumOnStartup` after migration ## Test results ``` ok github.com/corescope/server 19.448s ok github.com/corescope/ingestor 30.682s ``` --------- Co-authored-by: you <you@example.com>
83 lines
2.8 KiB
Markdown
83 lines
2.8 KiB
Markdown
# Database
|
||
|
||
CoreScope uses SQLite in WAL (Write-Ahead Log) mode for both the server
|
||
(read-only) and ingestor (read-write).
|
||
|
||
## WAL mode
|
||
|
||
WAL mode allows concurrent reads while writes happen. It is set automatically
|
||
at connection time via `PRAGMA journal_mode=WAL`. No operator action needed.
|
||
|
||
The WAL file (`meshcore.db-wal`) grows during writes and is checkpointed
|
||
(merged back into the main DB) periodically and at clean shutdown.
|
||
|
||
## Auto-vacuum
|
||
|
||
By default, SQLite does not shrink the database file after `DELETE` operations.
|
||
Deleted pages are marked free and reused by future writes, but the file size
|
||
on disk stays the same. This is surprising when lowering retention settings.
|
||
|
||
### New databases
|
||
|
||
Databases created after this feature was added automatically have
|
||
`PRAGMA auto_vacuum = INCREMENTAL`. After each retention reaper cycle,
|
||
CoreScope runs `PRAGMA incremental_vacuum(N)` to return free pages to the OS.
|
||
|
||
### Existing databases
|
||
|
||
The `auto_vacuum` mode is stored in the database header and can only be changed
|
||
by rewriting the entire file with `VACUUM`. CoreScope will **not** do this
|
||
automatically — on large databases (5+ GB seen in the wild) it takes minutes
|
||
and holds an exclusive lock.
|
||
|
||
**To migrate an existing database:**
|
||
|
||
1. At startup, CoreScope logs a warning:
|
||
```
|
||
[db] auto_vacuum=NONE — DB needs one-time VACUUM to enable incremental auto-vacuum.
|
||
```
|
||
2. **Ensure at least 2× the database file size in free disk space.** Full VACUUM
|
||
creates a temporary copy of the entire file — on a near-full disk it will fail.
|
||
3. Set `db.vacuumOnStartup: true` in your `config.json`:
|
||
```json
|
||
{
|
||
"db": {
|
||
"vacuumOnStartup": true
|
||
}
|
||
}
|
||
```
|
||
4. Restart CoreScope. The one-time `VACUUM` will run and block startup.
|
||
5. After migration, remove or set `vacuumOnStartup: false` — it's not needed again.
|
||
|
||
### Configuration
|
||
|
||
| Field | Default | Description |
|
||
|-------|---------|-------------|
|
||
| `db.vacuumOnStartup` | `false` | One-time full VACUUM to enable incremental auto-vacuum |
|
||
| `db.incrementalVacuumPages` | `1024` | Pages returned to OS per reaper cycle |
|
||
|
||
## Manual VACUUM
|
||
|
||
You can also run a manual vacuum from the SQLite CLI:
|
||
|
||
```bash
|
||
sqlite3 data/meshcore.db "PRAGMA auto_vacuum = INCREMENTAL; VACUUM;"
|
||
```
|
||
|
||
This is equivalent to `vacuumOnStartup: true` but can be done offline.
|
||
|
||
> ⚠️ Full VACUUM requires **2× the database file size** in free disk space (it
|
||
> creates a temporary copy). Check with `ls -lh data/meshcore.db` before running.
|
||
|
||
## Checking current mode
|
||
|
||
```bash
|
||
sqlite3 data/meshcore.db "PRAGMA auto_vacuum;"
|
||
```
|
||
|
||
- `0` = NONE (default for old databases)
|
||
- `1` = FULL (automatic, but slower writes)
|
||
- `2` = INCREMENTAL (recommended — CoreScope triggers vacuum after deletes)
|
||
|
||
See [#919](https://github.com/Kpa-clawbot/CoreScope/issues/919) for background on this feature.
|