Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create inactive_nodes table with identical schema to nodes
- Add retention.nodeDays config (default 7) in Node.js and Go
- On startup: move nodes not seen in N days to inactive_nodes
- Daily timer (24h setInterval / goroutine ticker) repeats the move
- Log 'Moved X nodes to inactive_nodes (not seen in N days)'
- All existing queries unchanged — they only read nodes table
- Add 14 new tests for moveStaleNodes in test-db.js
- Both Node (db.js/server.js) and Go (ingestor/server) implemented
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
advert_count was incremented on every upsertNode call, meaning each
observation of the same ADVERT packet inflated the count. Node N6NU
showed 4191 'adverts' but only had 77 unique ADVERT transmissions.
Changes:
- db.js: Remove advert_count increment from upsertNode SQL. Add
separate incrementAdvertCount() called only for new transmissions.
insertTransmission() now returns isNew flag.
- server.js: All three ADVERT processing paths (MQTT format 1,
companion bridge, API) now check isNew before incrementing.
- cmd/ingestor/db.go: Same fix in Go — UpsertNode no longer
increments, new IncrementAdvertCount method added.
InsertTransmission returns (bool, error) with isNew flag.
- cmd/ingestor/main.go: Check isNew before calling IncrementAdvertCount.
- One-time startup migration recalculates advert_count from
transmissions table (payload_type=4 matching node public_key).
Fixes#200
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The /api/observers handler assumed byObserver arrays were sorted
newest-first and used an early break when hitting an old timestamp.
In reality, byObserver is only roughly DESC from the initial DB load;
live-ingested observations are appended at the end (oldest-to-newest).
After ~1 hour of uptime, the first element is old, the break fires
immediately, and every observer returns packetsLastHour=0.
Fix: full scan without break — the array is not uniformly sorted.
The endpoint is cached so performance is unaffected.
fixes#182
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Node.js: reads version from package.json, commit from .git-commit file
or git rev-parse --short HEAD at runtime, with unknown fallback.
Go: uses -ldflags build-time variables (Version, Commit) with fallback
to .git-commit file and git command at runtime.
Dockerfile: copies .git-commit if present (CI bakes it before build).
Dockerfile.go: passes APP_VERSION and GIT_COMMIT as build args to ldflags.
deploy.yml: writes GITHUB_SHA to .git-commit before docker build steps.
docker-compose.yml: passes build args to Go staging build.
Tests updated to verify version and commit fields in both endpoints.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both backends now return an 'engine' field ('node' or 'go') in
/api/stats and /api/health responses so the frontend can display
which backend is running.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When multiple nodes share the same hash prefix (e.g. 1CC4 and 1C82 both
starting with 1C under 1-byte hash_size), updatePathSeenTimestamps() was
non-deterministically picking the first DB match, keeping dead nodes alive
on the map. Now resolveUniquePrefixMatch() only resolves prefixes that
match exactly one node. Ambiguous prefixes are cached in a negative-cache
set to avoid repeated DB queries.
- Extract resolveUniquePrefixMatch() used by both autoLearnHopNodes and
updatePathSeenTimestamps (DRY)
- Add ambiguousHopPrefixes negative cache (Set)
- LIMIT 2 in the uniqueness query to detect collisions efficiently
- 3 new regression tests: ambiguous prefix, unique prefix, 1-byte
collision scenario (204 -> 207 tests)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
autoLearnHopNodes was creating stub 'repeater' entries in the nodes table
for every unresolved hop prefix. With hash_size=1, this generated thousands
of phantom nodes (6,638 fake repeaters on a ~300-node mesh).
Root cause fix:
- autoLearnHopNodes no longer calls db.upsertNode() for unresolved hops
- Hop prefixes are still cached to avoid repeated DB lookups
- Unresolved hops display as raw hex via hop-resolver (no behavior change)
Cleanup:
- Added db.removePhantomNodes() — deletes nodes with public_key <= 16 chars
(real MeshCore pubkeys are 64 hex chars / 32 bytes)
- Called at server startup to purge existing phantoms
Tests: 14 new assertions in test-db.js (109 total, all passing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The /api/stats endpoint returned totalNodes from SELECT COUNT(*) FROM nodes,
which counts every node ever seen. On long-running instances this climbs to
6800+ for a ~200-400 node mesh.
Changes:
- totalNodes now counts only nodes with last_seen within the last 7 days
- Added totalNodesAllTime field for the full historical count
- Role counts (repeaters, rooms, etc.) also filtered to 7-day window
- Added countActiveNodes and countActiveNodesByRole prepared statements
- Added 6 tests verifying active vs all-time node counting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add hash field back to observation objects in packet-store.js (both
_loadNormalized and insert paths) — it's only 16 chars, negligible
memory vs the big fields raw_hex + decoded_json
- Fix /api/analytics/signal: look up raw_hex from transmission via
byTxId for packet size calculation
- Fix /api/observers/:id/analytics: enrich obsPackets so payload_type
and decoded_json are available for type breakdown and node buckets
- Endpoints /api/nodes/bulk-health, /api/nodes/network-status, and
/api/analytics/subpaths now work because observations carry hash
All 625 tests pass (unit + integration).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Repeaters that actively relay packets showed stale 'last seen' times
because last_seen only updates on adverts (every 12h) and last_heard
only tracked sender/recipient appearances, not relay hops.
- Add lastPathSeenMap: full pubkey → ISO timestamp for path hop sightings
- updatePathSeenTimestamps() resolves hop prefixes via hopPrefixToKey cache
- /api/nodes uses max(pktStore timestamp, path hop timestamp) for last_heard
- 4 new tests: hop-only nodes, stale vs fresh, pktStore priority, cache invalidation
Major refactor of live.js data flow:
- Replaced animatePacket() and animateRealisticPropagation() with
single renderPacketTree(packets, isReplay) function
- All paths use the same function: WS arrival, VCR replay, DB load,
feed card replay button
- VCR fetches use expand=observations to get full observation data
- expandToBufferEntries() extracts per-observer paths from observations
- startReplay() pre-aggregates VCR buffer by hash before playback
- Feed dedup accumulates observation packets for full tree replay
- Longest path shown in feed (scans all observations, not just first)
- Replay button uses full observation set for starburst animation
Server changes:
- WS broadcast includes path_json per observation
- packet-store insert() uses longest path for display (was earliest)
DB changes:
- Removed seed() function and synthetic test data
Not pushed to prod — local testing only.
- server.js disambiguateHops() now delegates to server-helpers.js
(was a full copy of the same algorithm, ~70 lines removed)
- live.js resolveHopPositions() now delegates to shared HopResolver
(was a standalone reimplementation, ~50 lines removed)
- HopResolver.init() called when live page loads/updates node data
- Net -106 lines, same behavior, single source of truth
All unit tests pass (241). E2E 13/16 (3 pre-existing Chromium crashes).
- Install nyc for Istanbul instrumentation
- Add scripts/instrument-frontend.sh to instrument public/*.js
- Add scripts/collect-frontend-coverage.js to extract window.__coverage__
- Add scripts/combined-coverage.sh for combined server+frontend coverage
- Make server.js serve public-instrumented/ when COVERAGE=1 is set
- Add test:full-coverage npm script
- Add public-instrumented/ and .nyc_output/ to .gitignore
test-server-routes.js destructures { cache, pktStore, db } but these
weren't in module.exports. Also adds require.main guard so server
doesn't listen when imported by tests.
Replace duplicated function definitions in server.js with imports from
server-helpers.js. Functions replaced: loadConfigFile, loadThemeFile,
buildHealthConfig, getHealthMs, isHashSizeFlipFlop, computeContentHash,
geoDist, deriveHashtagChannelKey, buildBreakdown, updateHashSizeForPacket,
rebuildHashSizeMap, requireApiKey, CONFIG_PATHS, THEME_PATHS.
disambiguateHops kept in server.js due to behavioral differences in the
distance sanity check (server version nulls lat/lon on unreliable hops
and adds ambiguous field in output mapping).
server.js: 3201 → 3001 lines (-200 lines, -224 deletions/+24 insertions)
All tests pass (unit, e2e, frontend).
Server now computes last_heard from in-memory packet store (all traffic
types) and includes it in /api/nodes response. Client prefers last_heard
over DB last_seen for display, sort, filter, and status calculation.
Fixes inconsistency where list showed '5d ago' but side pane showed
'26m ago' for the same node.
ADV_TYPE_ROOM=3 (0b0011) was misread as chat+repeater because decoder
treated lower nibble as individual bits. Now correctly: type & 0x0F as
enum (0=none, 1=chat, 2=repeater, 3=room, 4=sensor).
Includes startup backfill: scans all adverts and fixes any node roles
in the DB that were incorrectly set to 'repeater' when they should be
'room'. Logs count of fixed nodes on startup.
A node going 1B→2B and staying 2B is a firmware upgrade, not a bug.
Only flag as inconsistent when hash sizes flip-flop (2+ transitions in
the chronological advert sequence). Single transition = clean upgrade.
Tracks all hash_size values seen per node. If a node has sent adverts
with different hash sizes, flags it as hash_size_inconsistent with a
yellow ⚠️ badge on both side pane and detail page. Tooltip mentions
likely firmware bug (pre-1.14.1). Stats row shows all sizes seen.
Some nodes emit adverts with varying path byte values (e.g. 0x00 and 0x40).
Taking the first/newest was unreliable. Now takes the maximum hash_size
seen across all adverts for each node.
- Search order: app dir first (next to config.json), then data/ dir
- Startup log: '[theme] Loaded from ...' or 'No theme.json found. Place it next to config.json'
- README updated: 'put it next to config.json' instead of confusing data/ path
- Both loadConfigFile() and loadThemeFile() check data/ dir first, then app dir
- Theme endpoint re-reads both files on every request — edit the file, refresh the page
- No container restart, no symlinks, no extra mounts needed
- Just edit /app/data/theme.json (or config.json) and it's live
- Server reads from theme.json (separate from config.json), hot-loads on every request
- POST /api/config/theme writes theme.json directly — no manual file editing
- GET /api/config/theme now merges: defaults → config.json → theme.json
- Also returns themeDark and typeColors (were missing from API)
- Customizer: replaced 'merge into config.json' instructions with Save/Load Server buttons
- JSON export moved to collapsible details section
- theme.json added to .gitignore (instance-specific)
Add GET /api/config/theme endpoint serving branding, theme colors,
node colors, and home page content from config.json with sensible
defaults so unconfigured instances look identical to before.
Client-side (app.js):
- Fetch theme config on page load, before first render
- Override CSS variables from theme.* on document root
- Override ROLE_COLORS/ROLE_STYLE from nodeColors.*
- Replace nav brand text, logo, favicon from branding.*
- Store config in window.SITE_CONFIG for other pages
Home page (home.js):
- Hero title/subtitle from config.home
- Steps and checklist from config.home
- Footer links from config.home.footerLinks
- Chooser welcome text uses configured siteName
Config example updated with all available theme options.
No default appearance changes — all overrides are optional.
Without sender GPS (channel texts etc), the forward pass had no
anchor and just took candidates[0] — random order. Now regional
candidates are sorted by distKm to observer IATA center before
disambiguation. Closest to region center = default pick.
- packets.js: obsName() now shows IATA code next to observer name, e.g. 'EW-SFC-DR01 (SFO)'
- packets.js: hop conflicts in field table show distance (e.g. '37km')
- nodes.js: both full and sidebar detail views show 'Regions: SJC, OAK, SFO' badges and per-observer IATA
- live.js: node detail panel shows regions in 'Heard By' heading and per-observer IATA
- server.js: /api/nodes/:pubkey/health now returns iata field for each observer
- Bump cache busters
HopResolver now mirrors server-side layered regional filtering:
- init() accepts observers list + IATA coords
- resolve() accepts observerId, looks up IATA, filters candidates
by haversine distance (300km radius) to IATA center
- Candidates include regional, filterMethod, distKm fields
- Packet detail view passes observer_id to resolve()
New endpoint: GET /api/iata-coords returns airport coordinates
for client-side use.
Fixes: conflict badges showing "0 conflicts" in packet detail
because client-side resolver had no regional filtering.
Layer 1 (GPS, bridge-proof): Nodes with lat/lon are checked via
haversine distance to the observer IATA center. Only nodes within
300km are considered regional. Bridged WA nodes appearing in SJC
MQTT feeds are correctly rejected because their GPS coords are
1100km+ from SJC.
Layer 2 (observer-based, fallback): Nodes without GPS fall back to
_advertByObserver index — were they seen by a regional observer?
Less precise but still useful for nodes that never sent ADVERTs
with coordinates.
Layer 3: Global fallback, flagged.
New module: iata-coords.js with 60+ IATA airport coordinates +
haversine distance function.
API response now includes filterMethod (geo/observer/none) and
distKm per conflict candidate.
Tests: 22 unit tests (haversine, boundaries, cross-regional
collision sim, layered fallback, bridge rejection).
1-byte (and 2-byte) hop IDs match many nodes globally. Previously
resolve-hops picked candidates from anywhere, causing cross-regional
false paths (e.g. Eugene packet showing Vancouver repeaters).
Fix: Use observer IATA to determine packet region. Filter candidates
to nodes seen by observers in the same IATA region via the existing
_advertByObserver index. Fall back to global only if zero regional
candidates exist (flagged as globalFallback).
API changes to /api/resolve-hops response:
- conflicts[]: all candidates with regional flag per hop
- totalGlobal/totalRegional: candidate counts
- globalFallback: true when no regional candidates found
- region: packet IATA region in top-level response
UI changes:
- Conflict count badge (⚠3) instead of bare ⚠
- Tooltip shows regional vs global candidates
- Unreliable hops shown with strikethrough + opacity
- Global fallback hops shown with red dashed underline
New #/audio-lab page for understanding and debugging audio sonification.
Server: GET /api/audio-lab/buckets — returns representative packets
bucketed by type (up to 8 per type spanning size range).
Client: Left sidebar with collapsible type sections, right panel with:
- Controls: Play, Loop, Speed (0.25x-4x), BPM, Volume, Voice select
- Packet Data: type, sizes, hops, obs count, hex dump with sampled
bytes highlighted
- Sound Mapping: computed instrument, scale, filter, volume, voices,
pan — shows exactly why it sounds the way it does
- Note Sequence: table of sampled bytes → MIDI → freq → duration → gap
- Byte Visualizer: bar chart of payload bytes, sampled ones colored
Enables MeshAudio automatically on first play. Mobile responsive.
Secondary broadcast paths (ADVERT, GRP_TXT, TXT_MSG, TRACE, API)
were missing hash field. Without hash, realistic mode's buffer
check (if pkt.hash) failed and packets fell through to
animatePacket individually — causing duplicate feed items and
duplicate sonification.
Also added missing addFeedItem call in animateRealisticPropagation
so the feed shows consolidated entries in realistic mode.
getById() returns null for deduped observations (not stored in byId).
Client filters on m.data.packet being truthy, so all deduped packets
were silently dropped from WS. Fallback to transmission or raw pktData.
Migration runs automatically on next startup — drops paths first (FK to
packets), then packets. Removes insertPacket(), insertPath(), all
prepared statements and references to both tables. Server-side type/
observer filtering also removed (client does it in-memory).
Saves ~2M rows (paths) + full packets table worth of disk.
- Server: support comma-separated type filter values (OR logic)
- Server: add observer_id filtering to /api/packets endpoint
- Client: fix type and observer filters to use OR logic for multi-select
- Client: persist observer and type filter selections to localStorage
- Keys: meshcore-observer-filter, meshcore-type-filter
Previously db.seed() ran unconditionally on startup and would populate
a fresh database with fake test data. Now seeding only triggers when
explicitly requested via --seed CLI flag or SEED_DB=true env var.
The seed functionality remains available for developers:
node server.js --seed
SEED_DB=true node server.js
node db.js (direct run still seeds)
Removed server-side longest-path override in /api/packets/:id that
replaced the transmission's path_json with the longest observation
path. The header should always reflect the first observer's path.
Individual observation paths are available in the observations array.
Device clocks on MeshCore nodes are wildly inaccurate (off by hours or
epoch-near values like 4). The channel messages endpoint was using
sender_timestamp as part of the deduplication key, which could cause
messages to fail deduplication or incorrectly collide.
Changed dedupe key from sender:timestamp to sender:hash, which is the
correct unique identifier for a transmission.
Also added TIMESTAMP-AUDIT.md documenting all device timestamp usage.
- Create packets_v SQL view joining transmissions+observations to match old packets schema
- Replace all SELECT FROM packets with packets_v in db.js, packet-store.js, server.js
- Update countPackets/countRecentPackets to query observations directly
- Update seed() to use insertTransmission instead of insertPacket
- Remove insertPacket from exports (no longer called)
- Keep packets table schema intact (not dropped yet, pending testing)
SQLite WAL auto-checkpoint (every 1000 pages/4MB) was causing 200ms+
event loop spikes on a 381MB database. This is synchronous I/O that
blocks the Node.js event loop unpredictably.
Fix: disable auto-checkpoint, run PASSIVE (non-blocking) checkpoint
every 5 minutes. PASSIVE won't stall readers or writers — it only
checkpoints pages that aren't currently in use.
