#156: The Go WebSocket broadcast from IngestNewFromDB was missing the
'decoded' field (with header.payloadTypeName) that live.js needs to
display packet types. Added decoded object with payloadTypeName
resolution, plus observer_id, observer_name, snr, rssi, path_json,
and observation_count fields to match the Node.js broadcast shape.
#157-160: Analytics endpoints (hash-sizes, topology, channels) were
recomputing everything on every request. Added:
- TTL caching (15s) for topology, hash-sizes, and channels endpoints
(matching the existing RF cache pattern)
- Cached node list + prefix map shared across analytics (30s TTL)
- Lazy-cached parsed path JSON on StoreTx (parse once, read many)
- Cache invalidation on new data ingestion
- Global payloadTypeNames map (avoids per-call allocation)
Fixes#156, fixes#157, fixes#158, fixes#159, fixes#160
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- db.go: Add freelistMB (PRAGMA freelist_count * page_size) and walPages
(PRAGMA wal_checkpoint(PASSIVE)) to GetDBSizeStats
- store.go: Add advertByObserver count to GetPerfStoreStats indexes
(count distinct pubkeys with ADVERT observations)
- db.go: Add getObservationsForTransmissions helper; enrich
GetRecentTransmissionsForNode results with observations array,
_parsedPath, and _parsedDecoded
- db_test.go: Add second ADVERT with different hash_size to seed data
so hash_sizes_seen is populated; enrich decoded_json with full
ADVERT fields; update count assertions for new seed row
fixes#151, fixes#152
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Frontend reads ps.inMemory.toLocaleString() which crashed because
the Go response was missing inMemory, sqliteOnly, maxPackets, maxMB,
evicted, inserts, queries fields. Added all + atomic counters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix#148: channels endpoint returned null for msgLengths when no
decrypted messages exist. Initialize msgLengths as make([]int, 0)
in store path and guard channels slice in DB fallback path.
- Fix#149: nodes endpoint always returned hash_size=null and
hash_size_inconsistent=false. Add GetNodeHashSizeInfo() to
PacketStore that scans advert packets to compute per-node hash
size, flip-flop detection, and sizes_seen. Enrich nodes in both
handleNodes and handleNodeDetail with computed hash data.
fixes#148, fixes#149
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go analytics topology endpoint was counting every unique hop string
from packet paths (including unresolved 1-byte hex prefixes) as a unique
node, inflating the count from ~540 to 6502. Now resolves each hop via
the prefix map and deduplicates by public key, matching the Node.js
behavior.
fixes#146
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- /api/perf: add goRuntime (heap, GC, goroutines, CPU), packetStore
stats (totalLoaded, observations, index sizes, estimatedMB),
sqlite stats (dbSizeMB, walSizeMB, row counts), real RF cache
hit/miss tracking, and endpoint sorting by total time spent
- /api/health: add memory.heapMB, goRuntime (goroutines, gcPauses,
numCPU), real packetStore packet count and estimatedMB, real
cache stats from RF cache; remove hardcoded-zero eventLoop
- store.go: add cacheHits/cacheMisses tracking in GetAnalyticsRF,
GetPerfStoreStats() and GetCacheStats() methods
- db.go: add path field to DB struct, GetDBSizeStats() for file
sizes and row counts
- Tests: verify new fields in health/perf endpoints, add
TestGetDBSizeStats, wire up PacketStore in test server setup
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement all analytics endpoints from in-memory PacketStore instead of
returning stubs/empty data. Each handler now matches the Node.js response
shape field-by-field.
Endpoints fixed:
- /api/analytics/topology (#135): full hop distribution, top repeaters,
top pairs, hops-vs-SNR, per-observer reachability, cross-observer
comparison, best path analysis
- /api/analytics/distance (#137): haversine distance computation,
category stats (R↔R, C↔R, C↔C), distance histogram, top hops/paths,
distance over time
- /api/analytics/hash-sizes (#136): hash size distribution from raw_hex
path byte parsing, hourly breakdown, top hops, multi-byte node tracking
- /api/analytics/hash-issues (#138): hash-sizes data now populated so
frontend collision tab can compute inconsistent sizes and collision risk
- /api/analytics/route-patterns (#134): subpaths and subpath-detail now
compute from in-memory store with hop resolution
- /api/nodes/bulk-health (#140): switched from N per-node SQL queries to
in-memory PacketStore lookups with observer stats
- /api/channels (#142): response shape already correct via GetChannels;
analytics/channels now returns topSenders, channelTimeline, msgLengths
- /api/analytics/channels: full channel analytics with sender tracking,
timeline, and message length distribution
All handlers fall back to DB/stubs when store is nil (test compat).
All 42+ existing Go tests pass. go vet clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previous approach invalidated cache on every ingest (every 1s with live
mesh data). Now uses TTL-only expiry (15s). Separate cache mutex avoids
data race with main store RWMutex.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cache the computed RF analytics result for 15 seconds.
1.2M observation scan takes ~140ms; cached response <1ms.
Cache invalidated when new packets are ingested.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move per-transmission work (hash indexing, type resolution, packet sizes)
outside the per-observation loop. Cache SNR dereference, pre-resolve type
name once per transmission. Reduces redundant map lookups from 1.2M to 52K.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add byPayloadType index to PacketStore for O(1) type-5 lookups
- Channels scan reduced from 52K to ~17K packets (3x fewer iterations)
- Use struct-based JSON decoding (avoids map[string]interface{} allocations)
- Pre-allocate snrVals/rssiVals/scatterAll with capacity hints for 1.2M obs
- Remove second-pass time.Parse loop (1.2M calls) in RF analytics
Track min/max timestamps as strings during first pass instead
- Index also populated during IngestNewFromDB for new packets
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace SQLite-backed handlers for /api/channels, /api/channels/:hash/messages,
/api/analytics/rf, and /api/analytics/channels with in-memory PacketStore queries.
Before (SQLite via packets_v VIEW on 1.2M rows):
/api/channels 7.2s
/api/channels/:hash/msgs 8.2s
/api/analytics/rf 4.2s
After (in-memory scan of ~50K transmissions):
Target: all under 100ms
Three new PacketStore methods:
- GetChannels(region) — filters payload_type 5 + decoded type CHAN
- GetChannelMessages(hash, limit, offset) — deduplicates by sender+hash
- GetAnalyticsRF(region) — full RF stats with histograms, scatter, per-type SNR
All handlers fall back to DB queries when store is nil (test compat).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Streams transmissions + observations from SQLite at startup into
5 indexed in-memory structures. QueryPackets and QueryGroupedPackets
now serve from RAM (<10ms) instead of hitting SQLite (2.3s).
- store.go: PacketStore with byHash, byTxID, byObsID, byObserver, byNode indexes
- main.go: create + load store at startup
- routes.go: dispatch to store for packet/stats endpoints
- websocket.go: poller ingests new transmissions into store
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
