autoLearnHopNodes was creating stub 'repeater' entries in the nodes table
for every unresolved hop prefix. With hash_size=1, this generated thousands
of phantom nodes (6,638 fake repeaters on a ~300-node mesh).
Root cause fix:
- autoLearnHopNodes no longer calls db.upsertNode() for unresolved hops
- Hop prefixes are still cached to avoid repeated DB lookups
- Unresolved hops display as raw hex via hop-resolver (no behavior change)
Cleanup:
- Added db.removePhantomNodes() — deletes nodes with public_key <= 16 chars
(real MeshCore pubkeys are 64 hex chars / 32 bytes)
- Called at server startup to purge existing phantoms
Tests: 14 new assertions in test-db.js (109 total, all passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The /api/stats endpoint returned totalNodes from SELECT COUNT(*) FROM nodes,
which counts every node ever seen. On long-running instances this climbs to
6800+ for a ~200-400 node mesh.
Changes:
- totalNodes now counts only nodes with last_seen within the last 7 days
- Added totalNodesAllTime field for the full historical count
- Role counts (repeaters, rooms, etc.) also filtered to 7-day window
- Added countActiveNodes and countActiveNodesByRole prepared statements
- Added 6 tests verifying active vs all-time node counting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Major refactor of live.js data flow:
- Replaced animatePacket() and animateRealisticPropagation() with
single renderPacketTree(packets, isReplay) function
- All paths use the same function: WS arrival, VCR replay, DB load,
feed card replay button
- VCR fetches use expand=observations to get full observation data
- expandToBufferEntries() extracts per-observer paths from observations
- startReplay() pre-aggregates VCR buffer by hash before playback
- Feed dedup accumulates observation packets for full tree replay
- Longest path shown in feed (scans all observations, not just first)
- Replay button uses full observation set for starburst animation
Server changes:
- WS broadcast includes path_json per observation
- packet-store insert() uses longest path for display (was earliest)
DB changes:
- Removed seed() function and synthetic test data
Not pushed to prod — local testing only.
Migration runs automatically on next startup — drops paths first (FK to
packets), then packets. Removes insertPacket(), insertPath(), all
prepared statements and references to both tables. Server-side type/
observer filtering also removed (client does it in-memory).
Saves ~2M rows (paths) + full packets table worth of disk.
The UNIQUE index on (hash, observer_id, path_json) didn't prevent
duplicates when path_json was NULL because SQLite treats NULL != NULL
for uniqueness. Fixed by:
1. Using COALESCE(path_json, '') in the UNIQUE index expression
2. Adding migration to clean up existing duplicate rows
3. Adding NULL-safe dedup checks in PacketStore load and insert paths
- Create packets_v SQL view joining transmissions+observations to match old packets schema
- Replace all SELECT FROM packets with packets_v in db.js, packet-store.js, server.js
- Update countPackets/countRecentPackets to query observations directly
- Update seed() to use insertTransmission instead of insertPacket
- Remove insertPacket from exports (no longer called)
- Keep packets table schema intact (not dropped yet, pending testing)
SQLite WAL auto-checkpoint (every 1000 pages/4MB) was causing 200ms+
event loop spikes on a 381MB database. This is synchronous I/O that
blocks the Node.js event loop unpredictably.
Fix: disable auto-checkpoint, run PASSIVE (non-blocking) checkpoint
every 5 minutes. PASSIVE won't stall readers or writers — it only
checkpoints pages that aren't currently in use.
~26% of observations were duplicates from multi-broker MQTT ingestion.
Added UNIQUE index to prevent future dupes, INSERT OR IGNORE to skip
silently, and in-memory dedup check in packet-store.
After dedup migration, packet IDs from the legacy 'packets' table differ
from transmission IDs in the 'transmissions' table. URLs using numeric IDs
became invalid after restart when _loadNormalized() assigned different IDs.
Changes:
- All packet URLs now use 16-char hex hashes instead of numeric IDs
(#/packets/HASH instead of #/packet/ID)
- selectPacket() accepts hash parameter, uses hash-based URLs
- Copy Link generates hash-based URLs
- Search results link to hash-based URLs
- /api/packets/:id endpoint accepts both numeric IDs and 16-char hashes
- insert() now calls insertTransmission() to get stable transmission IDs
- Added db.getTransmission() for direct transmission table lookup
- Removed redundant byTransmission map (identical to byHash)
- All byTransmission references replaced with byHash
- GET /api/packets: returns transmissions with observation_count, strip
observations[] by default (use ?expand=observations to include)
- GET /api/packets/🆔 includes observation_count and observations[]
- GET /api/nodes/:pubkey/health: stats.totalTransmissions + totalObservations
(totalPackets kept for backward compat)
- GET /api/nodes/bulk-health: same transmission/observation split
- WebSocket broadcast: includes observation_count
- db.js getStats(): adds totalTransmissions count
- All backward-compatible: old field names preserved alongside new ones
- Add transmissions and observations schema to db.js init
- Add insertTransmission() function: upsert transmission by hash,
always insert observation row
- All 6 pktStore.insert() call sites in server.js now also call
db.insertTransmission() with try/catch (non-fatal on error)
- packet-store.js: add byTransmission Map index (hash → transmission
with observations array) for future M3 query migration
- Existing insertPacket() and all read paths unchanged
- mqttSources[].topics is now an array of topic patterns
- mqttSources[].iataFilter optionally restricts to specific regions
- meshcore/<region>/<id>/status topic parsed for observer metadata:
name, model, firmware, client_version, radio, battery, uptime, noise_floor
- New observer columns with auto-migration for existing DBs
- Status updates don't inflate packet_count (separate updateObserverStatus)
- New route: #/nodes/:pubkey/analytics with Chart.js v4 visualizations
- Activity timeline (bar), SNR trend (line), packet type breakdown (doughnut)
- Observer coverage (horizontal bar), hop distribution (bar)
- Uptime heatmap (7x24 CSS grid, GitHub-style)
- Peer interactions table with links to node details
- Stat cards: availability, signal grade, packets/day, relay %, silence
- Time range selector: 24h / 7d / 30d / All
- Server: GET /api/nodes/:pubkey/analytics with full aggregation in SQLite
- Analytics button added to both sidebar and full-screen node views
