packet.timestamp is first_seen — when the transmission was originally
observed. When multiple observers re-see the same old packet, the
broadcast carries the original (stale) first_seen. For channel list
display, what matters is 'activity happened now', not 'packet was
first seen 10h ago'.
The UNIQUE index on (hash, observer_id, path_json) didn't prevent
duplicates when path_json was NULL because SQLite treats NULL != NULL
for uniqueness. Fixed by:
1. Using COALESCE(path_json, '') in the UNIQUE index expression
2. Adding migration to clean up existing duplicate rows
3. Adding NULL-safe dedup checks in PacketStore load and insert paths
Device clocks on MeshCore nodes are wildly inaccurate (off by hours or
epoch-near values like 4). The channel messages endpoint was using
sender_timestamp as part of the deduplication key, which could cause
messages to fail deduplication or incorrectly collide.
Changed dedupe key from sender:timestamp to sender:hash, which is the
correct unique identifier for a transmission.
Also added TIMESTAMP-AUDIT.md documenting all device timestamp usage.
WS messages from lincomatic bridge lack packet.timestamp, so the
code fell through to payload.sender_timestamp which reflects the
MeshCore device's clock (often wrong). Use current time as fallback.
timeAgo labels were computed once on render and never updated,
showing stale '11h ago' until next WS message triggered re-render.
Added 30s interval to re-render channel list, cleaned up on destroy.
Multiple observers seeing the same packet triggered separate message
entries. Now deduplicates by packet hash — additional observations
increment repeats count and add to observers list. Channel messageCount
also only bumps once per unique packet.
Instead of re-fetching /api/channels and /api/channels/:hash/messages
on every WebSocket event, the channels page now processes WS messages
client-side:
- Extract sender, text, channel, timestamp from WS payload
- Append new messages directly to local messages[] array
- Update channel list entries (lastActivity, lastSender, messageCount)
- Create new channel entries for previously unseen channels
- Deduplicate repeated observations of the same message
API calls now only happen on:
- Initial page load (loadChannels)
- Channel selection (selectChannel)
- Region filter change
This eliminates all polling and WS-triggered re-fetches.
The channels WS handler was calling invalidateApiCache() before
loadChannels()/refreshMessages(), which nuked the cache and forced
network fetches. Combined with the global WS onmessage handler also
invalidating /channels every 5s, this created excessive API traffic
when sitting idle on the channels page.
Changes:
- channels.js: Remove invalidateApiCache calls from WS handler, use
bust:true parameter instead to bypass cache only when WS triggers
- channels.js: Add bust parameter to loadChannels() and refreshMessages()
- app.js: Remove /channels from global WS cache invalidation (channels
page manages its own cache busting via its dedicated WS handler)
Timestamp is decoded from the ADVERT but never persisted to the nodes
table. The validation was rejecting valid nodes with slightly-off clocks
(28h future) and nodes broadcasting timestamp=4. No reason to gate on it.
Expanding a grouped row fired: packets?hash=X&expand=observations,
packets?hash=X&limit=1, and packets/HASH — all returning the same
data. Now uses single /packets/HASH call and passes prefetched data
to selectPacket() to skip redundant fetches.
The WS handler's 250ms debounce fired loadChannels() before the
global 5s cache invalidation timer cleared the stale entry, so
the fetch returned cached data. Now channels.js invalidates its
own cache entries immediately before re-fetching.
packets_v view uses observation IDs, not transmission IDs, so
getPacket(transmissionId) returned null. Skip the DB lookup entirely
and construct the row directly from the incoming packetData object
which already has all needed fields.
packets_v view uses transmission IDs, not packets table IDs.
insertPacket returns a packets table ID which doesn't exist in
packets_v, so getPacket returned null and new packets never got
indexed in memory. Use transmissionId from insertTransmission instead.
Create public/hop-resolver.js that mirrors the server's disambiguateHops()
algorithm (prefix index, forward/backward pass, distance sanity check).
Replace all /api/resolve-hops fetch calls in packets.js with local
HopResolver.resolve() calls. The resolver lazily fetches and caches the
full nodes list via /api/nodes on first use.
The server endpoint is kept as fallback but no longer called by the UI,
eliminating 40+ HTTP requests per session.
- Create packets_v SQL view joining transmissions+observations to match old packets schema
- Replace all SELECT FROM packets with packets_v in db.js, packet-store.js, server.js
- Update countPackets/countRecentPackets to query observations directly
- Update seed() to use insertTransmission instead of insertPacket
- Remove insertPacket from exports (no longer called)
- Keep packets table schema intact (not dropped yet, pending testing)
SQLite WAL auto-checkpoint (every 1000 pages/4MB) was causing 200ms+
event loop spikes on a 381MB database. This is synchronous I/O that
blocks the Node.js event loop unpredictably.
Fix: disable auto-checkpoint, run PASSIVE (non-blocking) checkpoint
every 5 minutes. PASSIVE won't stall readers or writers — it only
checkpoints pages that aren't currently in use.
Previous pre-warm called computeAllSubpaths() synchronously (500ms+)
directly in the setTimeout callback, then sequentially warmed 8 more
endpoints. Any browser request arriving during this 1.5s+ window
waited the full duration (user saw 4816ms for a 3-hop resolve-hops).
Fix: ALL pre-warm now goes through self-HTTP-requests which yield the
event loop between each computation. Delayed to 5s so initial page
load requests complete first (they populate cache on-demand).
Removed the sync computeAllSubpaths() call and inline subpath cache
population — the /api/analytics/subpaths endpoint handles this itself.
Replaced inline resolveHopsInternal (allNodes.filter per hop) with
shared disambiguateHops (prefix-indexed). Also uses _parsedPath cache
and per-request disambig cache. /api/nodes/:pubkey/paths was 560ms cold,
should now be much faster.
200ms gaps meant clients hit cold caches themselves (worse).
Pre-warm should be fast and immediate — start at 100ms after listen,
setImmediate between endpoints to yield but not delay.
setImmediate wasn't enough — each analytics computation blocks for
200-400ms synchronously. Adding a 200ms setTimeout between pre-warm
requests gives pending client requests a window to complete between
the heavy computations.
- 8 separate 'SELECT * FROM nodes' queries replaced with getCachedNodes()
(refreshes every 30s, prefix index built once and reused)
- Region-filtered subpaths + master subpaths use _parsedPath cache
- Eliminates repeated SQLite queries + prefix index rebuilds across
back-to-back analytics endpoint calls
These endpoints were missing from the sequential pre-warm,
causing cold cache hits when clients connect before warm completes.
observers was 3s cold, distance was 600ms cold.
- Topology: replace O(N) allNodes.filter with prefix map + hop cache for resolveHop
- Topology: use _parsedPath cached JSON.parse for path_json (3 call sites)
- Topology: build observer map from already-filtered packets instead of second full scan
- Hash-sizes: prefix map for hop resolution instead of allNodes.find per hop
- Hash-sizes: use _parsedPath and _parsedDecoded cached parses
- Channels: use _parsedDecoded cached parse for decoded_json
- Build prefix map for O(1) hop resolution instead of O(N) linear scan per hop
- Cache resolved hops to avoid re-resolving same hex prefix across packets
- Pre-compute repeater set for O(1) role lookups
- Cache parsed path_json/decoded_json on packet objects (_parsedPath/_parsedDecoded)
The hash_size computation was scanning all 19K+ packets with JSON.parse
on every /api/nodes request, blocking the event loop for hundreds of ms.
Event loop p95 was 236ms, max 1732ms.
Now computed once at startup and updated incrementally on each new packet.
/api/nodes just does a Map.get per node instead of full scan.
Packets array is sorted newest-first. The previous 'last-wins'
approach (unconditional set) gave the OLDEST packet's hash_size.
Switched to first-wins (!has guard) which correctly uses the
newest ADVERT since we iterate newest-first.
Verified: Kpa Roof Solar has 1-byte ADVERTs (old firmware) and
2-byte ADVERTs (new firmware) interleaved. Newest are 2-byte.
Pass 1 correctly uses last-wins for ADVERT packets. But Pass 2
(fallback for nodes without ADVERTs) was also unconditionally
overwriting, so a stale 1-byte non-ADVERT packet would clobber
the correct 2-byte value from Pass 1.
Restored the !hashSizeMap.has() guard on Pass 2 only — it should
only fill gaps, never override ADVERT-derived hash_size.
The hashSizeMap was guarded by !hashSizeMap.has(pk), meaning the oldest
ADVERT determined a node's hash_size permanently. If a node upgraded
firmware from 1-byte to 2-byte hash prefix, the stale value persisted.
Remove the guard so newer packets overwrite older ones (last-seen-wins).
- New config.apiKey field — when set, POST endpoints require X-Api-Key header
- If apiKey not configured, endpoints remain open (dev/local mode)
- GET endpoints and /api/decode (read-only) remain public
- Closes the packet injection attack surface
- 🗺️ button on each top hop row → opens map with from/to markers + line
- 🗺️ button on each top path row → opens map with full multi-hop route
- Server now includes fromPk/toPk in topPaths hops for map resolution
- Uses existing drawPacketRoute() via sessionStorage handoff
- 1000km filter was too generous for LoRa (record ~250km)
- Uhuru kwa watu 📡 ↔ Bay Area hops at 880km were obviously wrong
- Node links in leaderboard now go to #/nodes/:pk (detail) not /analytics
New /api/analytics/distance endpoint that:
- Resolves path hops to nodes with valid GPS coordinates
- Calculates haversine distances between consecutive hops
- Separates stats by link type: R↔R, C↔R, C↔C
- Returns top longest hops, longest paths, category stats, histogram, time series
- Filters out invalid GPS (null, 0/0) and sanity-checks >1000km
- Supports region filtering and caching
New Distance tab in analytics UI with:
- Summary cards (total hops, paths, avg/max distance)
- Link type breakdown table
- Distance histogram
- Average distance over time sparkline
- Top 20 longest hops leaderboard
- Top 10 longest multi-hop paths table
The previous approach matched nodes via data packet hashes seen by
regional observers — but mesh packets propagate everywhere, so nearly
every node matched every region (550/558).
New approach: _advertByObserver index tracks which observers saw each
node's ADVERT packets. ADVERTs are local broadcasts that indicate
physical presence, so they're the correct signal for geographic filtering.
Also fixes role counts to reflect filtered results, not global totals.
Math.min(...arr) and Math.max(...arr) blow the call stack when arr
has tens of thousands of elements. Replaced with simple for-loop
arrMin/arrMax helpers.
Pills look out of place in the dense toolbar. Added { dropdown: true }
option to RegionFilter.init() to force dropdown mode regardless of
region count.
- Add aria-label and aria-haspopup='dialog' to BYOP trigger button
- Add aria-label to close button and textarea
- Add role='status' and aria-live='polite' to result container
- Add role='alert' to error messages for screen reader announcement
- Fix textarea focus style: visible outline instead of outline:none
- Update cache busters
- Separate region filtering from SNR filtering in /api/analytics/rf
- totalAllPackets now shows regional observation count (was global)
- Add totalTransmissions (unique hashes in regional set)
- Payload types and packet sizes use all regional data, not just SNR-filtered
- Signal stats (SNR, RSSI, scatter) use SNR-filtered subset
- Handle empty SNR/RSSI arrays gracefully (no Infinity/-Infinity)
- Fixes empty region dropdown (was populated before async regionMap loaded)
- Now uses multi-select RegionFilter component consistent with other pages
- Loads regions from /api/observers with proper async handling
- Supports multi-region filtering
The subagent used a non-existent column (sender_key) in the SQL join.
Reverted to the same byObserver + _nodeHashIndex approach used by
bulk-health and network-status endpoints.
The previous approach used pktStore._nodeHashIndex which only tracks
nodes appearing as sender/dest in decoded packet JSON. Most nodes only
send ADVERTs, so they had no entries in _nodeHashIndex and were filtered
out when a region was selected (showing 0 results).
Now uses a direct SQL join between observations and transmissions to find
all sender_keys observed by regional observers, which correctly includes
ADVERT-only nodes.
- Add RegionFilter.regionQueryString() to all API calls in renderSubpaths and renderNodesTab
- Add region filtering to /api/analytics/subpaths (filter packets by regional observer hashes)
- Add region filtering to /api/nodes/bulk-health (filter nodes by regional presence)
- Add region filtering to /api/nodes/network-status (filter node counts by region)
- Add region param to nodes lookup in hash collision tab
- Update cache keys to include region param for proper cache separation
you