Previously db.seed() ran unconditionally on startup and would populate
a fresh database with fake test data. Now seeding only triggers when
explicitly requested via --seed CLI flag or SEED_DB=true env var.
The seed functionality remains available for developers:
node server.js --seed
SEED_DB=true node server.js
node db.js (direct run still seeds)
When viewing a specific observation, the URL now includes ?obs=OBSERVER_ID.
Opening such a link auto-expands the group and selects the observation.
Copy Link button includes the obs parameter when an observation is selected.
Removed server-side longest-path override in /api/packets/:id that
replaced the transmission's path_json with the longest observation
path. The header should always reflect the first observer's path.
Individual observation paths are available in the observations array.
When expanding a grouped packet and clicking a child observation row,
the detail pane now shows that observation's observer, SNR/RSSI, path,
and timestamp instead of the parent packet's data.
Child rows use a new 'select-observation' action that builds a synthetic
packet object by overlaying observation-specific fields onto the parent
packet data (no extra API fetch needed).
When a later observation has an earlier timestamp, the transmission's
first_seen was updated but observer_id and path_json were not. This
caused the header row to show the wrong observer and path — whichever
MQTT message arrived first, not whichever observation was actually
earliest.
- Track lastActivityMs (Date.now()) on each channel object instead of ISO lastActivity
- 1s interval iterates channels[] array and updates DOM text only (no re-render)
- Uses data-channel-hash attribute to find time elements after DOM rebuilds
- Simple formatSecondsAgo: <60s→Xs, <3600s→Xm, <86400s→Xh, else Xd
- Seed lastActivityMs from API ISO string on initial load
- WS handler sets lastActivityMs = Date.now() on receipt
- Bump channels.js cache buster
The insert() method had a second code path (for building rows from
packetData) that pushed observations without checking for duplicates.
Added the same dedup check (observer_id + path_json) that exists in
the other insert path and in the load-from-DB path.
packet.timestamp is first_seen — when the transmission was originally
observed. When multiple observers re-see the same old packet, the
broadcast carries the original (stale) first_seen. For channel list
display, what matters is 'activity happened now', not 'packet was
first seen 10h ago'.
The UNIQUE index on (hash, observer_id, path_json) didn't prevent
duplicates when path_json was NULL because SQLite treats NULL != NULL
for uniqueness. Fixed by:
1. Using COALESCE(path_json, '') in the UNIQUE index expression
2. Adding migration to clean up existing duplicate rows
3. Adding NULL-safe dedup checks in PacketStore load and insert paths
Device clocks on MeshCore nodes are wildly inaccurate (off by hours or
epoch-near values like 4). The channel messages endpoint was using
sender_timestamp as part of the deduplication key, which could cause
messages to fail deduplication or incorrectly collide.
Changed dedupe key from sender:timestamp to sender:hash, which is the
correct unique identifier for a transmission.
Also added TIMESTAMP-AUDIT.md documenting all device timestamp usage.
WS messages from lincomatic bridge lack packet.timestamp, so the
code fell through to payload.sender_timestamp which reflects the
MeshCore device's clock (often wrong). Use current time as fallback.
timeAgo labels were computed once on render and never updated,
showing stale '11h ago' until next WS message triggered re-render.
Added 30s interval to re-render channel list, cleaned up on destroy.
Multiple observers seeing the same packet triggered separate message
entries. Now deduplicates by packet hash — additional observations
increment repeats count and add to observers list. Channel messageCount
also only bumps once per unique packet.
Instead of re-fetching /api/channels and /api/channels/:hash/messages
on every WebSocket event, the channels page now processes WS messages
client-side:
- Extract sender, text, channel, timestamp from WS payload
- Append new messages directly to local messages[] array
- Update channel list entries (lastActivity, lastSender, messageCount)
- Create new channel entries for previously unseen channels
- Deduplicate repeated observations of the same message
API calls now only happen on:
- Initial page load (loadChannels)
- Channel selection (selectChannel)
- Region filter change
This eliminates all polling and WS-triggered re-fetches.
The channels WS handler was calling invalidateApiCache() before
loadChannels()/refreshMessages(), which nuked the cache and forced
network fetches. Combined with the global WS onmessage handler also
invalidating /channels every 5s, this created excessive API traffic
when sitting idle on the channels page.
Changes:
- channels.js: Remove invalidateApiCache calls from WS handler, use
bust:true parameter instead to bypass cache only when WS triggers
- channels.js: Add bust parameter to loadChannels() and refreshMessages()
- app.js: Remove /channels from global WS cache invalidation (channels
page manages its own cache busting via its dedicated WS handler)
Timestamp is decoded from the ADVERT but never persisted to the nodes
table. The validation was rejecting valid nodes with slightly-off clocks
(28h future) and nodes broadcasting timestamp=4. No reason to gate on it.
Expanding a grouped row fired: packets?hash=X&expand=observations,
packets?hash=X&limit=1, and packets/HASH — all returning the same
data. Now uses single /packets/HASH call and passes prefetched data
to selectPacket() to skip redundant fetches.
The WS handler's 250ms debounce fired loadChannels() before the
global 5s cache invalidation timer cleared the stale entry, so
the fetch returned cached data. Now channels.js invalidates its
own cache entries immediately before re-fetching.
packets_v view uses observation IDs, not transmission IDs, so
getPacket(transmissionId) returned null. Skip the DB lookup entirely
and construct the row directly from the incoming packetData object
which already has all needed fields.
packets_v view uses transmission IDs, not packets table IDs.
insertPacket returns a packets table ID which doesn't exist in
packets_v, so getPacket returned null and new packets never got
indexed in memory. Use transmissionId from insertTransmission instead.
Create public/hop-resolver.js that mirrors the server's disambiguateHops()
algorithm (prefix index, forward/backward pass, distance sanity check).
Replace all /api/resolve-hops fetch calls in packets.js with local
HopResolver.resolve() calls. The resolver lazily fetches and caches the
full nodes list via /api/nodes on first use.
The server endpoint is kept as fallback but no longer called by the UI,
eliminating 40+ HTTP requests per session.
- Create packets_v SQL view joining transmissions+observations to match old packets schema
- Replace all SELECT FROM packets with packets_v in db.js, packet-store.js, server.js
- Update countPackets/countRecentPackets to query observations directly
- Update seed() to use insertTransmission instead of insertPacket
- Remove insertPacket from exports (no longer called)
- Keep packets table schema intact (not dropped yet, pending testing)
SQLite WAL auto-checkpoint (every 1000 pages/4MB) was causing 200ms+
event loop spikes on a 381MB database. This is synchronous I/O that
blocks the Node.js event loop unpredictably.
Fix: disable auto-checkpoint, run PASSIVE (non-blocking) checkpoint
every 5 minutes. PASSIVE won't stall readers or writers — it only
checkpoints pages that aren't currently in use.
Previous pre-warm called computeAllSubpaths() synchronously (500ms+)
directly in the setTimeout callback, then sequentially warmed 8 more
endpoints. Any browser request arriving during this 1.5s+ window
waited the full duration (user saw 4816ms for a 3-hop resolve-hops).
Fix: ALL pre-warm now goes through self-HTTP-requests which yield the
event loop between each computation. Delayed to 5s so initial page
load requests complete first (they populate cache on-demand).
Removed the sync computeAllSubpaths() call and inline subpath cache
population — the /api/analytics/subpaths endpoint handles this itself.
Replaced inline resolveHopsInternal (allNodes.filter per hop) with
shared disambiguateHops (prefix-indexed). Also uses _parsedPath cache
and per-request disambig cache. /api/nodes/:pubkey/paths was 560ms cold,
should now be much faster.
200ms gaps meant clients hit cold caches themselves (worse).
Pre-warm should be fast and immediate — start at 100ms after listen,
setImmediate between endpoints to yield but not delay.
setImmediate wasn't enough — each analytics computation blocks for
200-400ms synchronously. Adding a 200ms setTimeout between pre-warm
requests gives pending client requests a window to complete between
the heavy computations.
- 8 separate 'SELECT * FROM nodes' queries replaced with getCachedNodes()
(refreshes every 30s, prefix index built once and reused)
- Region-filtered subpaths + master subpaths use _parsedPath cache
- Eliminates repeated SQLite queries + prefix index rebuilds across
back-to-back analytics endpoint calls
These endpoints were missing from the sequential pre-warm,
causing cold cache hits when clients connect before warm completes.
observers was 3s cold, distance was 600ms cold.
- Topology: replace O(N) allNodes.filter with prefix map + hop cache for resolveHop
- Topology: use _parsedPath cached JSON.parse for path_json (3 call sites)
- Topology: build observer map from already-filtered packets instead of second full scan
- Hash-sizes: prefix map for hop resolution instead of allNodes.find per hop
- Hash-sizes: use _parsedPath and _parsedDecoded cached parses
- Channels: use _parsedDecoded cached parse for decoded_json
- Build prefix map for O(1) hop resolution instead of O(N) linear scan per hop
- Cache resolved hops to avoid re-resolving same hex prefix across packets
- Pre-compute repeater set for O(1) role lookups
- Cache parsed path_json/decoded_json on packet objects (_parsedPath/_parsedDecoded)
The hash_size computation was scanning all 19K+ packets with JSON.parse
on every /api/nodes request, blocking the event loop for hundreds of ms.
Event loop p95 was 236ms, max 1732ms.
Now computed once at startup and updated incrementally on each new packet.
/api/nodes just does a Map.get per node instead of full scan.
Packets array is sorted newest-first. The previous 'last-wins'
approach (unconditional set) gave the OLDEST packet's hash_size.
Switched to first-wins (!has guard) which correctly uses the
newest ADVERT since we iterate newest-first.
Verified: Kpa Roof Solar has 1-byte ADVERTs (old firmware) and
2-byte ADVERTs (new firmware) interleaved. Newest are 2-byte.
Pass 1 correctly uses last-wins for ADVERT packets. But Pass 2
(fallback for nodes without ADVERTs) was also unconditionally
overwriting, so a stale 1-byte non-ADVERT packet would clobber
the correct 2-byte value from Pass 1.
Restored the !hashSizeMap.has() guard on Pass 2 only — it should
only fill gaps, never override ADVERT-derived hash_size.
The hashSizeMap was guarded by !hashSizeMap.has(pk), meaning the oldest
ADVERT determined a node's hash_size permanently. If a node upgraded
firmware from 1-byte to 2-byte hash prefix, the stale value persisted.
Remove the guard so newer packets overwrite older ones (last-seen-wins).
- New config.apiKey field — when set, POST endpoints require X-Api-Key header
- If apiKey not configured, endpoints remain open (dev/local mode)
- GET endpoints and /api/decode (read-only) remain public
- Closes the packet injection attack surface
- 🗺️ button on each top hop row → opens map with from/to markers + line
- 🗺️ button on each top path row → opens map with full multi-hop route
- Server now includes fromPk/toPk in topPaths hops for map resolution
- Uses existing drawPacketRoute() via sessionStorage handoff
- 1000km filter was too generous for LoRa (record ~250km)
- Uhuru kwa watu 📡 ↔ Bay Area hops at 880km were obviously wrong
- Node links in leaderboard now go to #/nodes/:pk (detail) not /analytics
you