#156: The Go WebSocket broadcast from IngestNewFromDB was missing the
'decoded' field (with header.payloadTypeName) that live.js needs to
display packet types. Added decoded object with payloadTypeName
resolution, plus observer_id, observer_name, snr, rssi, path_json,
and observation_count fields to match the Node.js broadcast shape.
#157-160: Analytics endpoints (hash-sizes, topology, channels) were
recomputing everything on every request. Added:
- TTL caching (15s) for topology, hash-sizes, and channels endpoints
(matching the existing RF cache pattern)
- Cached node list + prefix map shared across analytics (30s TTL)
- Lazy-cached parsed path JSON on StoreTx (parse once, read many)
- Cache invalidation on new data ingestion
- Global payloadTypeNames map (avoids per-call allocation)
Fixes#156, fixes#157, fixes#158, fixes#159, fixes#160
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- db.go: Add freelistMB (PRAGMA freelist_count * page_size) and walPages
(PRAGMA wal_checkpoint(PASSIVE)) to GetDBSizeStats
- store.go: Add advertByObserver count to GetPerfStoreStats indexes
(count distinct pubkeys with ADVERT observations)
- db.go: Add getObservationsForTransmissions helper; enrich
GetRecentTransmissionsForNode results with observations array,
_parsedPath, and _parsedDecoded
- db_test.go: Add second ADVERT with different hash_size to seed data
so hash_sizes_seen is populated; enrich decoded_json with full
ADVERT fields; update count assertions for new seed row
fixes#151, fixes#152
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Frontend reads ps.inMemory.toLocaleString() which crashed because
the Go response was missing inMemory, sqliteOnly, maxPackets, maxMB,
evicted, inserts, queries fields. Added all + atomic counters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend perf dashboard crashes with toLocaleString error because
the Go /api/perf endpoint was missing fields that Node returns:
- status: "ok"
- uptimeHuman: formatted "Xh Ym" string
- websocket: { clients: N } via hub.ClientCount()
The Go /api/health endpoint already had all three fields.
fixes#150
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix#148: channels endpoint returned null for msgLengths when no
decrypted messages exist. Initialize msgLengths as make([]int, 0)
in store path and guard channels slice in DB fallback path.
- Fix#149: nodes endpoint always returned hash_size=null and
hash_size_inconsistent=false. Add GetNodeHashSizeInfo() to
PacketStore that scans advert packets to compute per-node hash
size, flip-flop detection, and sizes_seen. Enrich nodes in both
handleNodes and handleNodeDetail with computed hash data.
fixes#148, fixes#149
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go's /api/health was missing the eventLoop object that Node.js provides.
The perf.js frontend reads health.eventLoop.p95Ms which crashed with
'Cannot read properties of undefined' when served by the Go server.
Adds eventLoop field using GC pause data from runtime.MemStats.PauseNs
(last 256 pauses) to compute p50Ms, p95Ms, p99Ms, currentLagMs, maxLagMs
— matching the Node.js response shape exactly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go analytics topology endpoint was counting every unique hop string
from packet paths (including unresolved 1-byte hex prefixes) as a unique
node, inflating the count from ~540 to 6502. Now resolves each hop via
the prefix map and deduplicates by public key, matching the Node.js
behavior.
fixes#146
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- /api/perf: add goRuntime (heap, GC, goroutines, CPU), packetStore
stats (totalLoaded, observations, index sizes, estimatedMB),
sqlite stats (dbSizeMB, walSizeMB, row counts), real RF cache
hit/miss tracking, and endpoint sorting by total time spent
- /api/health: add memory.heapMB, goRuntime (goroutines, gcPauses,
numCPU), real packetStore packet count and estimatedMB, real
cache stats from RF cache; remove hardcoded-zero eventLoop
- store.go: add cacheHits/cacheMisses tracking in GetAnalyticsRF,
GetPerfStoreStats() and GetCacheStats() methods
- db.go: add path field to DB struct, GetDBSizeStats() for file
sizes and row counts
- Tests: verify new fields in health/perf endpoints, add
TestGetDBSizeStats, wire up PacketStore in test server setup
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement all analytics endpoints from in-memory PacketStore instead of
returning stubs/empty data. Each handler now matches the Node.js response
shape field-by-field.
Endpoints fixed:
- /api/analytics/topology (#135): full hop distribution, top repeaters,
top pairs, hops-vs-SNR, per-observer reachability, cross-observer
comparison, best path analysis
- /api/analytics/distance (#137): haversine distance computation,
category stats (R↔R, C↔R, C↔C), distance histogram, top hops/paths,
distance over time
- /api/analytics/hash-sizes (#136): hash size distribution from raw_hex
path byte parsing, hourly breakdown, top hops, multi-byte node tracking
- /api/analytics/hash-issues (#138): hash-sizes data now populated so
frontend collision tab can compute inconsistent sizes and collision risk
- /api/analytics/route-patterns (#134): subpaths and subpath-detail now
compute from in-memory store with hop resolution
- /api/nodes/bulk-health (#140): switched from N per-node SQL queries to
in-memory PacketStore lookups with observer stats
- /api/channels (#142): response shape already correct via GetChannels;
analytics/channels now returns topSenders, channelTimeline, msgLengths
- /api/analytics/channels: full channel analytics with sender tracking,
timeline, and message length distribution
All handlers fall back to DB/stubs when store is nil (test compat).
All 42+ existing Go tests pass. go vet clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previous approach invalidated cache on every ingest (every 1s with live
mesh data). Now uses TTL-only expiry (15s). Separate cache mutex avoids
data race with main store RWMutex.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Node.js: reads version from package.json, commit from .git-commit file
or git rev-parse --short HEAD at runtime, with unknown fallback.
Go: uses -ldflags build-time variables (Version, Commit) with fallback
to .git-commit file and git command at runtime.
Dockerfile: copies .git-commit if present (CI bakes it before build).
Dockerfile.go: passes APP_VERSION and GIT_COMMIT as build args to ldflags.
deploy.yml: writes GITHUB_SHA to .git-commit before docker build steps.
docker-compose.yml: passes build args to Go staging build.
Tests updated to verify version and commit fields in both endpoints.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cache the computed RF analytics result for 15 seconds.
1.2M observation scan takes ~140ms; cached response <1ms.
Cache invalidated when new packets are ingested.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move per-transmission work (hash indexing, type resolution, packet sizes)
outside the per-observation loop. Cache SNR dereference, pre-resolve type
name once per transmission. Reduces redundant map lookups from 1.2M to 52K.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both backends now return an 'engine' field ('node' or 'go') in
/api/stats and /api/health responses so the frontend can display
which backend is running.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add byPayloadType index to PacketStore for O(1) type-5 lookups
- Channels scan reduced from 52K to ~17K packets (3x fewer iterations)
- Use struct-based JSON decoding (avoids map[string]interface{} allocations)
- Pre-allocate snrVals/rssiVals/scatterAll with capacity hints for 1.2M obs
- Remove second-pass time.Parse loop (1.2M calls) in RF analytics
Track min/max timestamps as strings during first pass instead
- Index also populated during IngestNewFromDB for new packets
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace SQLite-backed handlers for /api/channels, /api/channels/:hash/messages,
/api/analytics/rf, and /api/analytics/channels with in-memory PacketStore queries.
Before (SQLite via packets_v VIEW on 1.2M rows):
/api/channels 7.2s
/api/channels/:hash/msgs 8.2s
/api/analytics/rf 4.2s
After (in-memory scan of ~50K transmissions):
Target: all under 100ms
Three new PacketStore methods:
- GetChannels(region) — filters payload_type 5 + decoded type CHAN
- GetChannelMessages(hash, limit, offset) — deduplicates by sender+hash
- GetAnalyticsRF(region) — full RF stats with histograms, scatter, per-type SNR
All handlers fall back to DB queries when store is nil (test compat).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Streams transmissions + observations from SQLite at startup into
5 indexed in-memory structures. QueryPackets and QueryGroupedPackets
now serve from RAM (<10ms) instead of hitting SQLite (2.3s).
- store.go: PacketStore with byHash, byTxID, byObsID, byObserver, byNode indexes
- main.go: create + load store at startup
- routes.go: dispatch to store for packet/stats endpoints
- websocket.go: poller ingests new transmissions into store
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Paho MQTT client uses tcp:// and ssl:// schemes, not mqtt:// and mqtts://.
Also properly configure TLS for mqtts connections with InsecureSkipVerify
when rejectUnauthorized is false.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend sends ISO timestamps to filter by observation time.
Go was filtering by transmission first_seen which missed packets
with recent observations but old first_seen. Now converts ISO to
unix epoch and queries the observations table directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Node.js upgrades WS at /, Go was only at /ws. Now the static file
handler checks for Upgrade header first and routes to WebSocket.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
First step of Go rewrite — separates MQTT ingestion from the Node.js
web server. Single static binary (no CGO) that connects to MQTT
brokers, decodes MeshCore packets, and writes to the shared SQLite DB.
Ported from JS:
- decoder.js → decoder.go (header, path, all payload types, adverts)
- computeContentHash → Go (SHA-256, path-independent)
- db.js v3 schema → db.go (transmissions, observations, nodes, observers)
- server.js MQTT logic → main.go (multi-broker, reconnect, IATA filter)
25 Go tests passing (golden fixtures from production + schema compat).
No existing JS files modified.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
