you
|
e340949253
|
feat: optimize observations table — 478MB → 141MB
Schema v3 migration:
- Replace observer_id TEXT (64-char hex) with observer_idx INTEGER FK
- Drop redundant hash, observer_name, created_at columns
- Store timestamp as epoch integer instead of ISO string
- In-memory dedup Set replaces expensive unique index lookups
- Auto-migration on startup with timestamped backup (never overwrites)
- Detects already-migrated DBs via pragma user_version + column inspection
Fixes:
- disambiguateHops: restore 'known' field dropped during refactor (fba5649)
- Skip MQTT connections when NODE_ENV=test
- e2e test: encodeURIComponent for # channel hashes in URLs
- VACUUM + TRUNCATE checkpoint after migration (not just VACUUM)
- Daily TRUNCATE checkpoint at 2:00 AM UTC to reclaim WAL space
Observability:
- SQLite stats in /api/perf (DB size, WAL size, freelist, row counts, busy pages)
- Rendered in perf dashboard with color-coded thresholds
Tests: 839 pass (89 db + 30 migration + 70 helpers + 200 routes + 34 packet-store + 52 decoder + 255 decoder-spec + 62 filter + 47 e2e)
|
2026-03-25 22:33:39 +00:00 |
|
you
|
629606bbdd
|
feat: optimize observations table schema (v3 migration)
- Replace observer_id TEXT (64-char hex) + observer_name TEXT with observer_idx INTEGER (FK to observers rowid)
- Remove redundant hash TEXT and created_at TEXT columns from observations
- Store timestamp as INTEGER epoch seconds instead of ISO text string
- Auto-migrate old schema on startup: backup DB, migrate data, rebuild indexes, VACUUM
- Migration is safe: backup first, abort on failure, schema_version marker prevents re-runs
- Backward-compatible packets_v view: JOINs observers table, converts epoch→ISO for consumers
- In-memory observer_id→rowid Map for fast lookups during ingestion
- In-memory dedup Set with 5-min TTL to prevent duplicate INSERT attempts
- packet-store.js: detect v3 schema and use appropriate JOIN query
- Tests: 29 migration tests (old→new, idempotency, backup failure, ingestion, dedup)
- Tests: 19 new v3 schema tests in test-db.js (columns, types, view compat, ingestion)
Expected savings on 947K-row prod DB:
- observer_id: 61 bytes → 4 bytes per row (57 bytes saved)
- observer_name: ~15 bytes → 0 (resolved via JOIN)
- hash: 16 bytes → 0 (redundant with transmission_id)
- timestamp: 25 bytes → 4 bytes (21 bytes saved)
- created_at: 25 bytes → 0 (redundant)
- Dedup index: much smaller (integers vs text)
- Estimated ~118 bytes saved per row = ~112MB total + massive index savings
|
2026-03-25 19:13:22 +00:00 |
|