you
629606bbdd
feat: optimize observations table schema (v3 migration)
...
- Replace observer_id TEXT (64-char hex) + observer_name TEXT with observer_idx INTEGER (FK to observers rowid)
- Remove redundant hash TEXT and created_at TEXT columns from observations
- Store timestamp as INTEGER epoch seconds instead of ISO text string
- Auto-migrate old schema on startup: backup DB, migrate data, rebuild indexes, VACUUM
- Migration is safe: backup first, abort on failure, schema_version marker prevents re-runs
- Backward-compatible packets_v view: JOINs observers table, converts epoch→ISO for consumers
- In-memory observer_id→rowid Map for fast lookups during ingestion
- In-memory dedup Set with 5-min TTL to prevent duplicate INSERT attempts
- packet-store.js: detect v3 schema and use appropriate JOIN query
- Tests: 29 migration tests (old→new, idempotency, backup failure, ingestion, dedup)
- Tests: 19 new v3 schema tests in test-db.js (columns, types, view compat, ingestion)
Expected savings on 947K-row prod DB:
- observer_id: 61 bytes → 4 bytes per row (57 bytes saved)
- observer_name: ~15 bytes → 0 (resolved via JOIN)
- hash: 16 bytes → 0 (redundant with transmission_id)
- timestamp: 25 bytes → 4 bytes (21 bytes saved)
- created_at: 25 bytes → 0 (redundant)
- Dedup index: much smaller (integers vs text)
- Estimated ~118 bytes saved per row = ~112MB total + massive index savings
2026-03-25 19:13:22 +00:00
you
3fdad47bfc
Add decoder and packet-store unit tests
...
- test-decoder.js: 52 tests covering all payload types (ADVERT, GRP_TXT, TXT_MSG, ACK, REQ, RESPONSE, ANON_REQ, PATH, TRACE, UNKNOWN), header parsing, path decoding, transport codes, edge cases, validateAdvert, and real packets from the API
- test-packet-store.js: 34 tests covering insert, deduplication, indexing (byHash, byNode, byObserver, advertByObserver), query with filters (type, route, hash, observer, since, until, order), queryGrouped, eviction, findPacketsForNode, getSiblings, countForNode, getTimestamps, getStats
Coverage improvement:
- decoder.js: 73.9% → 85.5% stmts, 41.7% → 89.3% branch, 69.2% → 92.3% funcs
- packet-store.js: 53.9% → 67.5% stmts, 46.6% → 63.9% branch, 50% → 79.2% funcs
- Overall: 37.2% → 40.0% stmts, 43.4% → 56.9% branch, 55.2% → 66.7% funcs
2026-03-24 00:59:41 +00:00
