mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-05-11 22:54:44 +00:00

Files

T

Kpa-clawbot aeae7813bc fix: enable SQLite incremental auto-vacuum so DB shrinks after retention (#919 ) (#920 )

Closes #919

## Summary

Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.

## Changes

### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this

### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor

### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility

### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM

### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides

## Migration path for existing databases

1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration

## Test results

```
ok  github.com/corescope/server    19.448s
ok  github.com/corescope/ingestor  30.682s
```

---------

Co-authored-by: you <you@example.com>

2026-04-30 23:45:00 -07:00

2.8 KiB

Raw Blame History

Database

CoreScope uses SQLite in WAL (Write-Ahead Log) mode for both the server (read-only) and ingestor (read-write).

WAL mode

WAL mode allows concurrent reads while writes happen. It is set automatically at connection time via PRAGMA journal_mode=WAL. No operator action needed.

The WAL file (meshcore.db-wal) grows during writes and is checkpointed (merged back into the main DB) periodically and at clean shutdown.

Auto-vacuum

By default, SQLite does not shrink the database file after DELETE operations. Deleted pages are marked free and reused by future writes, but the file size on disk stays the same. This is surprising when lowering retention settings.

New databases

Databases created after this feature was added automatically have PRAGMA auto_vacuum = INCREMENTAL. After each retention reaper cycle, CoreScope runs PRAGMA incremental_vacuum(N) to return free pages to the OS.

Existing databases

The auto_vacuum mode is stored in the database header and can only be changed by rewriting the entire file with VACUUM. CoreScope will not do this automatically — on large databases (5+ GB seen in the wild) it takes minutes and holds an exclusive lock.

To migrate an existing database:

At startup, CoreScope logs a warning:

[db] auto_vacuum=NONE — DB needs one-time VACUUM to enable incremental auto-vacuum.

Ensure at least 2× the database file size in free disk space. Full VACUUM creates a temporary copy of the entire file — on a near-full disk it will fail.

Set db.vacuumOnStartup: true in your config.json:

{
  "db": {
    "vacuumOnStartup": true
  }
}

Restart CoreScope. The one-time VACUUM will run and block startup.
After migration, remove or set vacuumOnStartup: false — it's not needed again.

Configuration

Field	Default	Description
`db.vacuumOnStartup`	`false`	One-time full VACUUM to enable incremental auto-vacuum
`db.incrementalVacuumPages`	`1024`	Pages returned to OS per reaper cycle

Manual VACUUM

You can also run a manual vacuum from the SQLite CLI:

sqlite3 data/meshcore.db "PRAGMA auto_vacuum = INCREMENTAL; VACUUM;"

This is equivalent to vacuumOnStartup: true but can be done offline.

⚠️ Full VACUUM requires 2× the database file size in free disk space (it creates a temporary copy). Check with ls -lh data/meshcore.db before running.

Checking current mode

sqlite3 data/meshcore.db "PRAGMA auto_vacuum;"

0 = NONE (default for old databases)
1 = FULL (automatic, but slower writes)
2 = INCREMENTAL (recommended — CoreScope triggers vacuum after deletes)

See #919 for background on this feature.

2.8 KiB Raw Blame History Unescape Escape