Commit Graph

7 Commits

Author SHA1 Message Date
Kpa-clawbot
520adcc6ab feat: move stale nodes to inactive_nodes table, fixes #202
- Create inactive_nodes table with identical schema to nodes
- Add retention.nodeDays config (default 7) in Node.js and Go
- On startup: move nodes not seen in N days to inactive_nodes
- Daily timer (24h setInterval / goroutine ticker) repeats the move
- Log 'Moved X nodes to inactive_nodes (not seen in N days)'
- All existing queries unchanged — they only read nodes table
- Add 14 new tests for moveStaleNodes in test-db.js
- Both Node (db.js/server.js) and Go (ingestor/server) implemented

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 22:43:53 -07:00
Kpa-clawbot
8e96b29859 fix: advert_count counts unique transmissions, not observations
advert_count was incremented on every upsertNode call, meaning each
observation of the same ADVERT packet inflated the count. Node N6NU
showed 4191 'adverts' but only had 77 unique ADVERT transmissions.

Changes:
- db.js: Remove advert_count increment from upsertNode SQL. Add
  separate incrementAdvertCount() called only for new transmissions.
  insertTransmission() now returns isNew flag.
- server.js: All three ADVERT processing paths (MQTT format 1,
  companion bridge, API) now check isNew before incrementing.
- cmd/ingestor/db.go: Same fix in Go — UpsertNode no longer
  increments, new IncrementAdvertCount method added.
  InsertTransmission returns (bool, error) with isNew flag.
- cmd/ingestor/main.go: Check isNew before calling IncrementAdvertCount.
- One-time startup migration recalculates advert_count from
  transmissions table (payload_type=4 matching node public_key).

Fixes #200

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 22:31:34 -07:00
Kpa-clawbot
eec6caaa48 fix: stop autoLearnHopNodes from creating phantom nodes, fixes #133
autoLearnHopNodes was creating stub 'repeater' entries in the nodes table
for every unresolved hop prefix. With hash_size=1, this generated thousands
of phantom nodes (6,638 fake repeaters on a ~300-node mesh).

Root cause fix:
- autoLearnHopNodes no longer calls db.upsertNode() for unresolved hops
- Hop prefixes are still cached to avoid repeated DB lookups
- Unresolved hops display as raw hex via hop-resolver (no behavior change)

Cleanup:
- Added db.removePhantomNodes() — deletes nodes with public_key <= 16 chars
  (real MeshCore pubkeys are 64 hex chars / 32 bytes)
- Called at server startup to purge existing phantoms

Tests: 14 new assertions in test-db.js (109 total, all passing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 02:05:31 -07:00
Kpa-clawbot
ac7bd13ae8 fix: filter totalNodes to active nodes (7-day window), fixes #133
The /api/stats endpoint returned totalNodes from SELECT COUNT(*) FROM nodes,
which counts every node ever seen. On long-running instances this climbs to
6800+ for a ~200-400 node mesh.

Changes:
- totalNodes now counts only nodes with last_seen within the last 7 days
- Added totalNodesAllTime field for the full historical count
- Role counts (repeaters, rooms, etc.) also filtered to 7-day window
- Added countActiveNodes and countActiveNodesByRole prepared statements
- Added 6 tests verifying active vs all-time node counting

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 01:56:38 -07:00
you
e340949253 feat: optimize observations table — 478MB → 141MB
Schema v3 migration:
- Replace observer_id TEXT (64-char hex) with observer_idx INTEGER FK
- Drop redundant hash, observer_name, created_at columns
- Store timestamp as epoch integer instead of ISO string
- In-memory dedup Set replaces expensive unique index lookups
- Auto-migration on startup with timestamped backup (never overwrites)
- Detects already-migrated DBs via pragma user_version + column inspection

Fixes:
- disambiguateHops: restore 'known' field dropped during refactor (fba5649)
- Skip MQTT connections when NODE_ENV=test
- e2e test: encodeURIComponent for # channel hashes in URLs
- VACUUM + TRUNCATE checkpoint after migration (not just VACUUM)
- Daily TRUNCATE checkpoint at 2:00 AM UTC to reclaim WAL space

Observability:
- SQLite stats in /api/perf (DB size, WAL size, freelist, row counts, busy pages)
- Rendered in perf dashboard with color-coded thresholds

Tests: 839 pass (89 db + 30 migration + 70 helpers + 200 routes + 34 packet-store + 52 decoder + 255 decoder-spec + 62 filter + 47 e2e)
2026-03-25 22:33:39 +00:00
you
629606bbdd feat: optimize observations table schema (v3 migration)
- Replace observer_id TEXT (64-char hex) + observer_name TEXT with observer_idx INTEGER (FK to observers rowid)
- Remove redundant hash TEXT and created_at TEXT columns from observations
- Store timestamp as INTEGER epoch seconds instead of ISO text string
- Auto-migrate old schema on startup: backup DB, migrate data, rebuild indexes, VACUUM
- Migration is safe: backup first, abort on failure, schema_version marker prevents re-runs
- Backward-compatible packets_v view: JOINs observers table, converts epoch→ISO for consumers
- In-memory observer_id→rowid Map for fast lookups during ingestion
- In-memory dedup Set with 5-min TTL to prevent duplicate INSERT attempts
- packet-store.js: detect v3 schema and use appropriate JOIN query
- Tests: 29 migration tests (old→new, idempotency, backup failure, ingestion, dedup)
- Tests: 19 new v3 schema tests in test-db.js (columns, types, view compat, ingestion)

Expected savings on 947K-row prod DB:
- observer_id: 61 bytes → 4 bytes per row (57 bytes saved)
- observer_name: ~15 bytes → 0 (resolved via JOIN)
- hash: 16 bytes → 0 (redundant with transmission_id)
- timestamp: 25 bytes → 4 bytes (21 bytes saved)
- created_at: 25 bytes → 0 (redundant)
- Dedup index: much smaller (integers vs text)
- Estimated ~118 bytes saved per row = ~112MB total + massive index savings
2026-03-25 19:13:22 +00:00
you
21e7996c98 Extract server-helpers.js and add unit tests for server logic + db.js
- Extract pure/near-pure functions from server.js into server-helpers.js:
  loadConfigFile, loadThemeFile, buildHealthConfig, getHealthMs,
  isHashSizeFlipFlop, computeContentHash, geoDist, deriveHashtagChannelKey,
  buildBreakdown, disambiguateHops, updateHashSizeForPacket, rebuildHashSizeMap,
  requireApiKey

- Add test-server-helpers.js (70 tests) covering all extracted functions
- Add test-db.js (68 tests) covering all db.js exports with temp SQLite DB
- Coverage: 39.97% → 81.3% statements, 56% → 68.5% branches, 65.5% → 89.5% functions
2026-03-24 01:09:03 +00:00