The previous approach matched nodes via data packet hashes seen by
regional observers — but mesh packets propagate everywhere, so nearly
every node matched every region (550/558).
New approach: _advertByObserver index tracks which observers saw each
node's ADVERT packets. ADVERTs are local broadcasts that indicate
physical presence, so they're the correct signal for geographic filtering.
Also fixes role counts to reflect filtered results, not global totals.
~26% of observations were duplicates from multi-broker MQTT ingestion.
Added UNIQUE index to prevent future dupes, INSERT OR IGNORE to skip
silently, and in-memory dedup check in packet-store.
After dedup migration, packet IDs from the legacy 'packets' table differ
from transmission IDs in the 'transmissions' table. URLs using numeric IDs
became invalid after restart when _loadNormalized() assigned different IDs.
Changes:
- All packet URLs now use 16-char hex hashes instead of numeric IDs
(#/packets/HASH instead of #/packet/ID)
- selectPacket() accepts hash parameter, uses hash-based URLs
- Copy Link generates hash-based URLs
- Search results link to hash-based URLs
- /api/packets/:id endpoint accepts both numeric IDs and 16-char hashes
- insert() now calls insertTransmission() to get stable transmission IDs
- Added db.getTransmission() for direct transmission table lookup
- Removed redundant byTransmission map (identical to byHash)
- All byTransmission references replaced with byHash
After dedup, table rows have transmission IDs but getById() maps
observation IDs. Added byTxId index so /api/packets/:id resolves
both observation and transmission IDs correctly.
- Add transmissions and observations schema to db.js init
- Add insertTransmission() function: upsert transmission by hash,
always insert observation row
- All 6 pktStore.insert() call sites in server.js now also call
db.insertTransmission() with try/catch (non-fatal on error)
- packet-store.js: add byTransmission Map index (hash → transmission
with observations array) for future M3 query migration
- Existing insertPacket() and all read paths unchanged
One method resolves name→pubkey, combines byNode index + text search.
Used in: query fast-path, combined-filter path, health endpoint,
analytics endpoint. Bulk-health still uses index-only (perf).
Zero SQLite reads from packets table. Every endpoint that previously
scanned packets now reads from the in-memory PacketStore.
Expected: subpaths from 1.6s to <100ms, topology from 700ms to <50ms,
RF from 270ms to <30ms on cold calls.
- PacketStore loads all packets into memory on startup (~11MB for 27K packets)
- Indexed by id, hash, observer, and node pubkey for fast lookups
- /api/packets, /api/packets/timestamps, /api/packets/:id all served from RAM
- MQTT ingest writes to both RAM + SQLite
- Configurable maxMemoryMB (default 1024MB) in config.json packetStore section
- groupByHash queries computed in-memory
- Packet store stats exposed in /api/perf
- Expected: /api/packets goes from 77ms to <1ms
you