getById() returns null for deduped observations (not stored in byId).
Client filters on m.data.packet being truthy, so all deduped packets
were silently dropped from WS. Fallback to transmission or raw pktData.
Migration runs automatically on next startup — drops paths first (FK to
packets), then packets. Removes insertPacket(), insertPath(), all
prepared statements and references to both tables. Server-side type/
observer filtering also removed (client does it in-memory).
Saves ~2M rows (paths) + full packets table worth of disk.
- Server: support comma-separated type filter values (OR logic)
- Server: add observer_id filtering to /api/packets endpoint
- Client: fix type and observer filters to use OR logic for multi-select
- Client: persist observer and type filter selections to localStorage
- Keys: meshcore-observer-filter, meshcore-type-filter
Previously db.seed() ran unconditionally on startup and would populate
a fresh database with fake test data. Now seeding only triggers when
explicitly requested via --seed CLI flag or SEED_DB=true env var.
The seed functionality remains available for developers:
node server.js --seed
SEED_DB=true node server.js
node db.js (direct run still seeds)
Removed server-side longest-path override in /api/packets/:id that
replaced the transmission's path_json with the longest observation
path. The header should always reflect the first observer's path.
Individual observation paths are available in the observations array.
Device clocks on MeshCore nodes are wildly inaccurate (off by hours or
epoch-near values like 4). The channel messages endpoint was using
sender_timestamp as part of the deduplication key, which could cause
messages to fail deduplication or incorrectly collide.
Changed dedupe key from sender:timestamp to sender:hash, which is the
correct unique identifier for a transmission.
Also added TIMESTAMP-AUDIT.md documenting all device timestamp usage.
- Create packets_v SQL view joining transmissions+observations to match old packets schema
- Replace all SELECT FROM packets with packets_v in db.js, packet-store.js, server.js
- Update countPackets/countRecentPackets to query observations directly
- Update seed() to use insertTransmission instead of insertPacket
- Remove insertPacket from exports (no longer called)
- Keep packets table schema intact (not dropped yet, pending testing)
SQLite WAL auto-checkpoint (every 1000 pages/4MB) was causing 200ms+
event loop spikes on a 381MB database. This is synchronous I/O that
blocks the Node.js event loop unpredictably.
Fix: disable auto-checkpoint, run PASSIVE (non-blocking) checkpoint
every 5 minutes. PASSIVE won't stall readers or writers — it only
checkpoints pages that aren't currently in use.
Previous pre-warm called computeAllSubpaths() synchronously (500ms+)
directly in the setTimeout callback, then sequentially warmed 8 more
endpoints. Any browser request arriving during this 1.5s+ window
waited the full duration (user saw 4816ms for a 3-hop resolve-hops).
Fix: ALL pre-warm now goes through self-HTTP-requests which yield the
event loop between each computation. Delayed to 5s so initial page
load requests complete first (they populate cache on-demand).
Removed the sync computeAllSubpaths() call and inline subpath cache
population — the /api/analytics/subpaths endpoint handles this itself.
Replaced inline resolveHopsInternal (allNodes.filter per hop) with
shared disambiguateHops (prefix-indexed). Also uses _parsedPath cache
and per-request disambig cache. /api/nodes/:pubkey/paths was 560ms cold,
should now be much faster.
200ms gaps meant clients hit cold caches themselves (worse).
Pre-warm should be fast and immediate — start at 100ms after listen,
setImmediate between endpoints to yield but not delay.
setImmediate wasn't enough — each analytics computation blocks for
200-400ms synchronously. Adding a 200ms setTimeout between pre-warm
requests gives pending client requests a window to complete between
the heavy computations.
- 8 separate 'SELECT * FROM nodes' queries replaced with getCachedNodes()
(refreshes every 30s, prefix index built once and reused)
- Region-filtered subpaths + master subpaths use _parsedPath cache
- Eliminates repeated SQLite queries + prefix index rebuilds across
back-to-back analytics endpoint calls
These endpoints were missing from the sequential pre-warm,
causing cold cache hits when clients connect before warm completes.
observers was 3s cold, distance was 600ms cold.
- Topology: replace O(N) allNodes.filter with prefix map + hop cache for resolveHop
- Topology: use _parsedPath cached JSON.parse for path_json (3 call sites)
- Topology: build observer map from already-filtered packets instead of second full scan
- Hash-sizes: prefix map for hop resolution instead of allNodes.find per hop
- Hash-sizes: use _parsedPath and _parsedDecoded cached parses
- Channels: use _parsedDecoded cached parse for decoded_json
- Build prefix map for O(1) hop resolution instead of O(N) linear scan per hop
- Cache resolved hops to avoid re-resolving same hex prefix across packets
- Pre-compute repeater set for O(1) role lookups
- Cache parsed path_json/decoded_json on packet objects (_parsedPath/_parsedDecoded)
The hash_size computation was scanning all 19K+ packets with JSON.parse
on every /api/nodes request, blocking the event loop for hundreds of ms.
Event loop p95 was 236ms, max 1732ms.
Now computed once at startup and updated incrementally on each new packet.
/api/nodes just does a Map.get per node instead of full scan.
Packets array is sorted newest-first. The previous 'last-wins'
approach (unconditional set) gave the OLDEST packet's hash_size.
Switched to first-wins (!has guard) which correctly uses the
newest ADVERT since we iterate newest-first.
Verified: Kpa Roof Solar has 1-byte ADVERTs (old firmware) and
2-byte ADVERTs (new firmware) interleaved. Newest are 2-byte.
Pass 1 correctly uses last-wins for ADVERT packets. But Pass 2
(fallback for nodes without ADVERTs) was also unconditionally
overwriting, so a stale 1-byte non-ADVERT packet would clobber
the correct 2-byte value from Pass 1.
Restored the !hashSizeMap.has() guard on Pass 2 only — it should
only fill gaps, never override ADVERT-derived hash_size.
The hashSizeMap was guarded by !hashSizeMap.has(pk), meaning the oldest
ADVERT determined a node's hash_size permanently. If a node upgraded
firmware from 1-byte to 2-byte hash prefix, the stale value persisted.
Remove the guard so newer packets overwrite older ones (last-seen-wins).
- New config.apiKey field — when set, POST endpoints require X-Api-Key header
- If apiKey not configured, endpoints remain open (dev/local mode)
- GET endpoints and /api/decode (read-only) remain public
- Closes the packet injection attack surface
- 🗺️ button on each top hop row → opens map with from/to markers + line
- 🗺️ button on each top path row → opens map with full multi-hop route
- Server now includes fromPk/toPk in topPaths hops for map resolution
- Uses existing drawPacketRoute() via sessionStorage handoff
- 1000km filter was too generous for LoRa (record ~250km)
- Uhuru kwa watu 📡 ↔ Bay Area hops at 880km were obviously wrong
- Node links in leaderboard now go to #/nodes/:pk (detail) not /analytics
New /api/analytics/distance endpoint that:
- Resolves path hops to nodes with valid GPS coordinates
- Calculates haversine distances between consecutive hops
- Separates stats by link type: R↔R, C↔R, C↔C
- Returns top longest hops, longest paths, category stats, histogram, time series
- Filters out invalid GPS (null, 0/0) and sanity-checks >1000km
- Supports region filtering and caching
New Distance tab in analytics UI with:
- Summary cards (total hops, paths, avg/max distance)
- Link type breakdown table
- Distance histogram
- Average distance over time sparkline
- Top 20 longest hops leaderboard
- Top 10 longest multi-hop paths table
The previous approach matched nodes via data packet hashes seen by
regional observers — but mesh packets propagate everywhere, so nearly
every node matched every region (550/558).
New approach: _advertByObserver index tracks which observers saw each
node's ADVERT packets. ADVERTs are local broadcasts that indicate
physical presence, so they're the correct signal for geographic filtering.
Also fixes role counts to reflect filtered results, not global totals.
Math.min(...arr) and Math.max(...arr) blow the call stack when arr
has tens of thousands of elements. Replaced with simple for-loop
arrMin/arrMax helpers.
- Separate region filtering from SNR filtering in /api/analytics/rf
- totalAllPackets now shows regional observation count (was global)
- Add totalTransmissions (unique hashes in regional set)
- Payload types and packet sizes use all regional data, not just SNR-filtered
- Signal stats (SNR, RSSI, scatter) use SNR-filtered subset
- Handle empty SNR/RSSI arrays gracefully (no Infinity/-Infinity)
The subagent used a non-existent column (sender_key) in the SQL join.
Reverted to the same byObserver + _nodeHashIndex approach used by
bulk-health and network-status endpoints.
The previous approach used pktStore._nodeHashIndex which only tracks
nodes appearing as sender/dest in decoded packet JSON. Most nodes only
send ADVERTs, so they had no entries in _nodeHashIndex and were filtered
out when a region was selected (showing 0 results).
Now uses a direct SQL join between observations and transmissions to find
all sender_keys observed by regional observers, which correctly includes
ADVERT-only nodes.
- Add RegionFilter.regionQueryString() to all API calls in renderSubpaths and renderNodesTab
- Add region filtering to /api/analytics/subpaths (filter packets by regional observer hashes)
- Add region filtering to /api/nodes/bulk-health (filter nodes by regional presence)
- Add region filtering to /api/nodes/network-status (filter node counts by region)
- Add region param to nodes lookup in hash collision tab
- Update cache keys to include region param for proper cache separation
- channel-rainbow.json: 592 pre-computed SHA256-derived keys for common
channel names (cities, topics, ham radio, emergency, etc.)
- server.js: Load rainbow table at startup as lowest-priority key source
- config.example.json: Add #LongFast to hashChannels list
Key derivation verified against MeshCore source: SHA256('#name')[:16bytes].
Rainbow table boosted decryption from ~48% to ~88% in testing.
- Remove composite key scheme (ch_/unk_ prefixes) that broke URL routing
due to # in channel names. Use plain numeric channelHash as key instead.
- All packets with same hash byte go in one bucket; name is set from
first successful decryption.
- Add packet detail renderer for decoded CHAN type showing channel name,
sender, and sender timestamp.
- Update cache buster for packets.js.
The /api/channels endpoint was returning simple numeric hash (e.g. '45') while
/api/channels/:hash/messages was using composite keys (e.g. 'ch_#LongFast',
'unk_45') internally. This mismatch meant no channel ever matched, so all
messages appeared encrypted.
Fix: return the composite key as the hash field from /api/channels so the
frontend passes the correct identifier. Also add encodeURIComponent() to
channel API calls in the frontend since composite keys can contain '#'.
Adds mapDefaults config option with center and zoom properties.
New /api/config/map endpoint serves the defaults. live.js and map.js
fetch the config with fallback to hardcoded Bay Area defaults.
FixesKpa-clawbot/meshcore-analyzer#115
Channels sharing the same hash prefix but with different keys (e.g. #startrek
and #ai-bot both with hash 2d) now display the correct name by keying on the
actual channel name from decryption rather than just the hash byte.
FixesKpa-clawbot/meshcore-analyzer#108
resolve-hops now accepts originLat/originLon params. Forward pass
starts from sender position so first ambiguous hop resolves to the
nearest node to the sender, not the observer.
you