Match the C++ firmware wire format (Packet::writeTo/readFrom):
1. Field order: transport codes are parsed BEFORE path_length byte,
matching firmware's header → transport_codes → path_len → path → payload
2. ACK payload: just 4-byte CRC checksum, not dest+src+ackHash.
Firmware createAck() writes only ack_crc (4 bytes).
3. TRACE payload: tag(4) + authCode(4) + flags(1) + pathData,
matching firmware createTrace() and onRecvPacket() TRACE handler.
4. ADVERT features: parse feat1 (0x20) and feat2 (0x40) optional
2-byte fields between location and name, matching AdvertDataBuilder
and AdvertDataParser in the firmware.
5. Transport code naming: code1/code2 instead of nextHop/lastHop,
matching firmware's transport_codes[0]/transport_codes[1] naming.
Fixes applied to both cmd/ingestor/decoder.go and cmd/server/decoder.go.
Tests updated to match new behavior.
Server defaults to 6060, ingestor to 6061. Removed shared PPROF_PORT
env var. Bind failure logs warning instead of log.Fatal killing the process.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fresh Go installs failed with 'no such table: packets_v' because the
ingestor created tables but never the VIEW that the Go server queries.
Add DROP VIEW IF EXISTS + CREATE VIEW packets_v to applySchema(), using
the v3 definition (observer_idx → observers.rowid JOIN). The view is
rebuilt on every startup to stay current with any definition changes.
Add tests: verify view exists after OpenStore, and verify it returns
correct observer_id/observer_name via the LEFT JOIN.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add net/http/pprof support to both Go server (default port 6060) and
ingestor (default port 6061). Profiling is off by default — only
starts the pprof HTTP listener when ENABLE_PPROF=true.
PPROF_PORT env var overrides the default port for each binary.
Enable on staging-go in docker-compose with exposed ports 6060/6061.
Not enabled on prod.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Observability:
- Add DBStats struct with atomic counters for tx_inserted, tx_dupes,
obs_inserted, node_upserts, observer_upserts, write_errors
- Log SQLite config on startup (busy_timeout, max_open_conns, journal)
- Periodic stats logging every 5 minutes + final stats on shutdown
- Instrument all write paths with counter increments
Tests:
- TestConcurrentWrites: 20 goroutines × 50 writes (1000 total) with
interleaved InsertTransmission + UpsertNode + UpsertObserver calls.
Verifies zero errors and data integrity under concurrent load.
- TestDBStats: verifies counter accuracy for inserts, duplicates,
upserts, and that LogStats does not panic
Three changes to eliminate concurrent write collisions:
1. Add _busy_timeout=5000 to ingestor SQLite DSN (matches server)
- SQLite will wait up to 5s for the write lock instead of
immediately returning SQLITE_BUSY
2. Set SetMaxOpenConns(1) on ingestor DB connection pool
- Serializes all DB access at the Go sql.DB level
- Prevents multiple goroutines from opening overlapping writes
3. Change SetOrderMatters(false) to SetOrderMatters(true)
- MQTT handlers now run sequentially per client
- Eliminates concurrent handler execution that caused
overlapping multi-statement write flows
Root cause: concurrent MQTT handlers (SetOrderMatters=false) each
performed multiple separate writes (transmission lookup/insert,
observation insert, node upsert, observer upsert) without transactions
or connection limits. SQLite only permits one writer at a time, so
under bursty MQTT traffic the ingestor was competing with itself.
UpsertNode only updates name/role/lat/lon/last_seen. The advert_count
field is modified exclusively by IncrementAdvertCount, which is called
separately in the MQTT handler. The test incorrectly expected count=2
after two UpsertNode calls; the correct value is 0 (the schema default).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create inactive_nodes table with identical schema to nodes
- Add retention.nodeDays config (default 7) in Node.js and Go
- On startup: move nodes not seen in N days to inactive_nodes
- Daily timer (24h setInterval / goroutine ticker) repeats the move
- Log 'Moved X nodes to inactive_nodes (not seen in N days)'
- All existing queries unchanged — they only read nodes table
- Add 14 new tests for moveStaleNodes in test-db.js
- Both Node (db.js/server.js) and Go (ingestor/server) implemented
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
advert_count was incremented on every upsertNode call, meaning each
observation of the same ADVERT packet inflated the count. Node N6NU
showed 4191 'adverts' but only had 77 unique ADVERT transmissions.
Changes:
- db.js: Remove advert_count increment from upsertNode SQL. Add
separate incrementAdvertCount() called only for new transmissions.
insertTransmission() now returns isNew flag.
- server.js: All three ADVERT processing paths (MQTT format 1,
companion bridge, API) now check isNew before incrementing.
- cmd/ingestor/db.go: Same fix in Go — UpsertNode no longer
increments, new IncrementAdvertCount method added.
InsertTransmission returns (bool, error) with isNew flag.
- cmd/ingestor/main.go: Check isNew before calling IncrementAdvertCount.
- One-time startup migration recalculates advert_count from
transmissions table (payload_type=4 matching node public_key).
Fixes#200
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The #197 decryption fix added channelKeys parameter to decodePayload and
DecodePacket, but the test call sites were malformed:
- DecodePacket(hex, nil + stringExpr) → nil concatenated with string (type error)
- decodePayload(type, make([]byte, N, nil)) → nil used as make capacity (type error)
Fixed to:
- DecodePacket(hex + stringExpr, nil) → string concat then nil channelKeys
- decodePayload(type, make([]byte, N), nil) → proper 3-arg call
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After decryption produces text, validate it's printable UTF-8.
If it contains more than 2 non-printable characters (excluding
newline/tab), mark as decryption_failed with text: null.
Applied to both Node (decoder.js) and Go (cmd/ingestor/decoder.go)
decoders. Added tests for garbage and valid text in both.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#184: Strip non-printable chars (<0x20 except tab/newline) from ADVERT
names in Go server decoder, Go ingestor decoder, and Node decoder.js.
#185: Add visual (N) badge next to node names when multiple nodes share
the same display name (case-insensitive). Shows in list, side pane, and
full detail page with 'also known as' links to other keys.
#186: Add packetsLast24h field to /api/stats response.
#187#188: Cache runtime.ReadMemStats() with 5s TTL in Go server.
#189: Temporarily patch HTMLCanvasElement.prototype.getContext during
L.heatLayer().addTo(map) to pass { willReadFrequently: true }, preventing
Chrome console warning about canvas readback performance.
Tests: 10 new tests for buildDupNameMap + dupNameBadge (143 total frontend).
Cache busters bumped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go ingestor never had channel decryption — GRP_TXT packets were stored
with raw encrypted data while Node.js decoded them successfully.
Changes:
- decoder.go: Add decryptChannelMessage() implementing MeshCore channel
crypto (HMAC-SHA256 MAC verification + AES-128-ECB decryption), matching
the algorithm in @michaelhart/meshcore-decoder. Update decodeGrpTxt(),
decodePayload(), and DecodePacket() to accept and pass channel keys.
Add Payload fields: ChannelHashHex, DecryptionStatus, Channel, Text,
Sender, SenderTimestamp.
- config.go: Add ChannelKeysPath and ChannelKeys fields to Config struct.
- main.go: Add loadChannelKeys() that loads channel-rainbow.json (same
file used by Node.js server) from beside the config file, with env var
and config overrides. Pass loaded keys through the decoder pipeline.
- decoder_test.go: Add 14 channel decryption tests covering valid
decryption, MAC failure, wrong key, no-sender messages, bracket
sender exclusion, key iteration, channelHashHex formatting, and
decryption status states. Cross-validated against Node.js output.
- Update all DecodePacket/decodePayload/decodeGrpTxt/handleMessage call
sites in test files to pass the new channelKeys parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#165 — build_time in API: already implemented (BuildTime ldflags in
Dockerfile.go, main.go, StatsResponse, HealthResponse)
#168 — subpaths API slow: cache (subpathCache with TTL) and invalidation
already in place; verified working
#169 — distance API slow: cache (distCache with TTL) and invalidation
already in place; verified working
#170 — audio-lab/buckets: in-memory store path already implemented,
matching Node.js pktStore.packets iteration with type grouping and
size-distributed sampling
#171 — channels stale latest message: add companion bridge handling to
Go ingestor for meshcore/message/channel/<n> and meshcore/message/direct/<id>
MQTT topics. Stores decoded channel messages with type CHAN in decoded_json,
enabling the channels endpoint to find them. Also handles direct messages.
#172 — packets page not live-updating: add missing direction field to WS
broadcast packet map for full parity with txToMap/Node.js fullPacket shape.
WS broadcast shape verified correct (type, data.packet structure, timestamp,
payload_type, observer_id all present).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Paho MQTT client uses tcp:// and ssl:// schemes, not mqtt:// and mqtts://.
Also properly configure TLS for mqtts connections with InsecureSkipVerify
when rejectUnauthorized is false.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
First step of Go rewrite — separates MQTT ingestion from the Node.js
web server. Single static binary (no CGO) that connects to MQTT
brokers, decodes MeshCore packets, and writes to the shared SQLite DB.
Ported from JS:
- decoder.js → decoder.go (header, path, all payload types, adverts)
- computeContentHash → Go (SHA-256, path-independent)
- db.js v3 schema → db.go (transmissions, observations, nodes, observers)
- server.js MQTT logic → main.go (multi-broker, reconnect, IATA filter)
25 Go tests passing (golden fixtures from production + schema compat).
No existing JS files modified.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
you