Kpa-clawbot
aeae7813bc
Closes #919 ## Summary Enables SQLite incremental auto-vacuum so the database file actually shrinks after retention reaper deletes old data. Previously, `DELETE` operations freed pages internally but never returned disk space to the OS. ## Changes ### 1. Auto-vacuum on new databases - `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before `journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval` - Must be set before any tables are created; DSN ordering ensures this ### 2. Post-reaper incremental vacuum - `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle (packets, metrics, observers, neighbor edges) - N defaults to 1024 pages, configurable via `db.incrementalVacuumPages` - Noop on `auto_vacuum=NONE` databases (safe before migration) - Added to both server and ingestor ### 3. Opt-in full VACUUM for existing databases - Startup check logs a clear warning if `auto_vacuum != INCREMENTAL` - `db.vacuumOnStartup: true` config triggers one-time `PRAGMA auto_vacuum = INCREMENTAL; VACUUM` - Logs start/end time for operator visibility ### 4. Documentation - `docs/user-guide/configuration.md`: retention section notes that lowering retention doesn't immediately shrink the DB - `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum, migration, manual VACUUM ### 5. Tests - `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2` - `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0` - `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2` - `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks freelist - `TestCheckAutoVacuumLogs` — handles both modes without panic - `TestConfigIncrementalVacuumPages` — config defaults and overrides ## Migration path for existing databases 1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs one-time VACUUM...` 2. Set `db.vacuumOnStartup: true` in config.json 3. Restart — VACUUM runs (blocks startup, minutes on large DBs) 4. Remove `vacuumOnStartup` after migration ## Test results ``` ok github.com/corescope/server 19.448s ok github.com/corescope/ingestor 30.682s ``` --------- Co-authored-by: you <you@example.com>
5.9 KiB
Configuration
CoreScope is configured via config.json in the server's working directory. Copy config.example.json to get started.
Core settings
| Field | Default | Description |
|---|---|---|
port |
3000 |
HTTP server port |
apiKey |
— | Secret key for admin API endpoints (POST/PUT routes) |
dbPath |
— | Path to SQLite database file (optional, defaults to meshcore.db) |
MQTT
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
}
The ingestor connects to this MQTT broker and subscribes to the topic pattern.
Multiple MQTT sources
Use mqttSources for multiple brokers:
"mqttSources": [
{
"name": "local",
"broker": "mqtt://localhost:1883",
"topics": ["meshcore/#"]
},
{
"name": "remote",
"broker": "mqtts://mqtt.example.com:8883",
"username": "user",
"password": "pass",
"topics": ["meshcore/SJC/#"]
}
]
Branding
| Field | Description |
|---|---|
branding.siteName |
Site title shown in the nav bar |
branding.tagline |
Subtitle on the home page |
branding.logoUrl |
URL to a custom logo image |
branding.faviconUrl |
URL to a custom favicon |
Theme
Colors used throughout the UI. All values are hex color codes.
| Field | Description |
|---|---|
theme.accent |
Primary accent color (links, buttons) |
theme.navBg |
Navigation bar background |
theme.navBg2 |
Secondary nav background |
theme.statusGreen |
Healthy status color |
theme.statusYellow |
Degraded status color |
theme.statusRed |
Silent/error status color |
See Customization for the full list — the theme customizer exposes every color.
Node colors
Default marker colors by role:
"nodeColors": {
"repeater": "#dc2626",
"companion": "#2563eb",
"room": "#16a34a",
"sensor": "#d97706",
"observer": "#8b5cf6"
}
Health thresholds
How long (in hours) before a node is marked degraded or silent:
| Field | Default | Description |
|---|---|---|
healthThresholds.infraDegradedHours |
24 |
Repeaters/rooms → degraded after this many hours |
healthThresholds.infraSilentHours |
72 |
Repeaters/rooms → silent after this many hours |
healthThresholds.nodeDegradedHours |
1 |
Companions/others → degraded |
healthThresholds.nodeSilentHours |
24 |
Companions/others → silent |
Retention
| Field | Default | Description |
|---|---|---|
retention.nodeDays |
7 |
Nodes not seen in N days move to inactive |
retention.packetDays |
30 |
Packets older than N days are deleted daily |
Note: Lowering retention does not immediately shrink the database file. SQLite marks deleted pages as free but does not return them to the filesystem unless incremental auto-vacuum is enabled. New databases created after v0.x.x have auto-vacuum enabled automatically. Existing databases require a one-time migration — see the Database guide.
Database
| Field | Default | Description |
|---|---|---|
db.vacuumOnStartup |
false |
Run a one-time full VACUUM on startup to enable incremental auto-vacuum (blocks for minutes on large DBs) |
db.incrementalVacuumPages |
1024 |
Free pages returned to the OS after each retention reaper cycle |
See Database for details on SQLite auto-vacuum, WAL, and manual maintenance. See #919 for background.
Channel decryption
| Field | Description |
|---|---|
channelKeys |
Object of "label": "hex-key" pairs for decrypting channel messages |
hashChannels |
Array of channel names (e.g., "#LongFast") to match by hash |
See Channels for details.
Map defaults
"mapDefaults": {
"center": [37.45, -122.0],
"zoom": 9
}
Initial map center and zoom level.
Regions
"regions": {
"SJC": "San Jose, US",
"SFO": "San Francisco, US"
}
Named regions for the region filter dropdown. The defaultRegion field sets which region is selected by default.
Cache TTL
All values in seconds. Controls how long the server caches API responses:
"cacheTTL": {
"stats": 10,
"nodeList": 90,
"nodeDetail": 300,
"analyticsRF": 1800
}
Lower values = fresher data but more server load.
Packet store
| Field | Default | Description |
|---|---|---|
packetStore.maxMemoryMB |
1024 |
Maximum RAM for in-memory packet store |
packetStore.estimatedPacketBytes |
450 |
Estimated bytes per packet (for memory budgeting) |
Timestamps
| Field | Default | Description |
|---|---|---|
timestamps.defaultMode |
"ago" |
Display mode: "ago" (relative) or "absolute" |
timestamps.timezone |
"local" |
"local" or "utc" |
timestamps.formatPreset |
"iso" |
Date format preset |
Live map
| Field | Default | Description |
|---|---|---|
liveMap.propagationBufferMs |
5000 |
How long to buffer observations before animating |
HTTPS
"https": {
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem"
}
Provide cert and key paths to enable HTTPS.
Geographic filtering
"geo_filter": {
"polygon": [[51.55, 3.80], [51.55, 5.90], [50.65, 5.90], [50.65, 3.80]],
"bufferKm": 20
}
Restricts ingestion and API responses to nodes within the polygon plus a buffer margin. Remove the block to disable filtering. Nodes with no GPS fix always pass through.
See Geographic Filtering for the full guide including the visual polygon builder and the prune script for cleaning up historical data.
Home page
The home section customizes the onboarding experience. See config.example.json for the full structure including steps, checklist, and footerLinks.