#165 — build_time in API: already implemented (BuildTime ldflags in
Dockerfile.go, main.go, StatsResponse, HealthResponse)
#168 — subpaths API slow: cache (subpathCache with TTL) and invalidation
already in place; verified working
#169 — distance API slow: cache (distCache with TTL) and invalidation
already in place; verified working
#170 — audio-lab/buckets: in-memory store path already implemented,
matching Node.js pktStore.packets iteration with type grouping and
size-distributed sampling
#171 — channels stale latest message: add companion bridge handling to
Go ingestor for meshcore/message/channel/<n> and meshcore/message/direct/<id>
MQTT topics. Stores decoded channel messages with type CHAN in decoded_json,
enabling the channels endpoint to find them. Also handles direct messages.
#172 — packets page not live-updating: add missing direction field to WS
broadcast packet map for full parity with txToMap/Node.js fullPacket shape.
WS broadcast shape verified correct (type, data.packet structure, timestamp,
payload_type, observer_id all present).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
db.GetChannels() queried packets_v (observation-level rows) ordered by
observation timestamp and always overwrote lastMessage. When an older
message had a later re-observation, it would overwrite the correct
latest message with stale data.
Fix: query transmissions table directly (one row per unique message)
ordered by first_seen. This ensures lastMessage always reflects the
most recently sent message, not the most recently observed one.
Also fix db.GetChannelMessages() to use first_seen ordering with
schema-aware queries (v2/v3), and add missing distCache/subpathCache
invalidation on packet ingestion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use s.store (in-memory PacketStore) instead of direct packets_v SQL query,
matching how the Node.js handler iterates pktStore.packets. This fixes the
endpoint returning empty buckets when packets_v view is missing or the DB
query fails silently.
Key changes:
- Group by decoded_json.type first, fall back to payloadTypeNames
- Evenly-spaced sampling (up to 8 per type) sorted by raw_hex length
- Use actual ObservationCount instead of hardcoded 1
- Reuse payloadTypeNames from store.go instead of duplicating
- Retain DB fallback path when store is nil
fixes#170
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Added decision: Proto fixture capture in CI runs against prod (stable reference)
- Added decision: Protobuf API contract architecture (single source of truth for frontend/backend interfaces)
- Merged 2 directives from spawn manifest processing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 15s TTL cache to GetAnalyticsSubpaths with composite key (region|minLen|maxLen|limit),
matching the existing cache pattern used by RF, topology, hash, channel, and distance analytics.
Cache hits return instantly vs 900ms+ computation. fixes#168
- Add BuildTime to /api/stats and /api/health responses, injected via ldflags at build time.
Dockerfile.go now accepts BUILD_TIME build arg. fixes#165
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The proto validation infrastructure was added in commit e70ba44 but used
an invalid --syntax_check flag. Changed to use --descriptor_set_out=/dev/null
which validates syntax without generating files.
Proto validation flow (now complete):
1. go-test job: verify .proto files compile (syntax check) ✅
2. deploy-node job: validate protos match prod API responses ✅
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go server's WebSocket broadcast included first_seen but not
timestamp in the nested packet object. The frontend packets.js
filters on m.data.packet and reads p.timestamp for row insertion
and sorting. Without this field, live-updating silently failed
(rows inserted with undefined latest, breaking display).
Mirrors the pattern already used in txToMap() (store.go:1168)
which correctly emits both first_seen and timestamp.
Also updates websocket_test.go to assert timestamp presence
in broadcast data to prevent regression.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The build-node job was failing with 'node: not found' because it
runs scripts/validate.sh (which uses 'node -c' for syntax checking)
but didn't have the actions/setup-node@v4 step.
Added Node.js 22 setup before the validate step to match the pattern
used in other jobs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The distance analytics endpoint was recomputing haversine distances on
every request (~1.22s). Add a 15s TTL cache following the same pattern
used by RF, topology, hash-sizes, and channels analytics endpoints.
Include distCache in cache stats size calculation.
fixes#169
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously only captured 19 simple endpoints. Now captures all 33:
- 19 simple endpoints (stats, health, nodes, etc.)
- 14 dynamic ID endpoints (node-detail, packet-detail, etc.)
Dynamic ID resolution:
- Extracts real pubkey from /api/nodes for node detail endpoints
- Extracts real hash from /api/packets for packet-detail
- Extracts real observer ID from /api/observers for observer endpoints
- Gracefully skips fixtures if DB is empty (no data yet)
WebSocket capture:
- Uses node -e with ws module to capture one live WS message
- Falls back gracefully if no live packets available
The validator already handles missing fixtures without failing, so this
will work even when staging container has no data yet.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Added a CI step that:
- Refreshes Node fixtures from the staging container after deployment
- Runs tools/validate-protos.py to validate proto definitions match actual API responses
- Fails the pipeline if proto drift is detected
This ensures nobody can merge a Node change that breaks the Go proto contract
without updating the .proto definitions.
The step runs after the Node staging healthcheck, capturing fresh responses
from 19 API endpoints (stats, health, nodes, analytics/*, config/*, etc.).
Endpoints requiring parameters (node-detail, packet-detail) use existing
fixtures and aren't auto-refreshed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validates all 33 captured Node fixtures against the 10 .proto files.
Parses proto message definitions, maps each fixture to its response
message, and checks field presence, types, and structural shape.
Mismatches found (for Hicks to fix):
1. analytics-channels.json: ChannelAnalyticsSummary.hash is int in
fixture but proto says string
2. analytics-rf.json: PayloadTypeEntry.type is null in fixture but
proto says non-optional int32
3. bulk-health.json: API returns bare array with flat node fields;
proto nests them in BulkHealthEntry.node (structural mismatch)
4. node-analytics.json: activityTimeline uses 'bucket' key but
TimeBucket proto expects 'label'
5. observer-analytics.json: recentPackets are Observation-shaped
(have transmission_id) but proto says repeated Transmission
6. packet-detail.json: ByteRange has 'color' field not in proto
7. websocket-message.json: DecodedResult missing transportCodes,
raw fields; DecodedHeader missing routeTypeName; DecodedPayload
is flat (not oneof-wrapped); WSPacketData.packet is Observation-
shaped, not Transmission-shaped
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go staging packets page wasn't live-updating because the deployed
binary was stale (built before the #162 fix). Rebuilding from current
source fixed the issue — broadcasts now fire correctly.
Added two permanent diagnostic log lines:
- [poller] IngestNewFromDB: logs when new transmissions are found
- [broadcast] sending N packets to M clients: logs each broadcast batch
These log lines make it easy to verify the broadcast pipeline is working
and would have caught this stale-deployment issue immediately.
Verified on VM: WS clients receive packets with nested 'packet' field,
all Go tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Capture real responses from all 32 REST endpoints + 1 WebSocket
message from the production MeshCore Analyzer instance. Fixtures
include nodes, packets, observers, channels, analytics, config,
and health endpoints with real IDs substituted.
Stored in proto/testdata/node-fixtures/ for Go port contract testing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the exact response shape, query parameters, and type information
for every endpoint in server.js. This is the authoritative contract that
both Node.js and Go backends must conform to.
Covers:
- All 30+ REST endpoints with full JSON response schemas
- WebSocket message envelope and data shapes
- Shared object shapes (Packet, Observation, DecodedHeader, DecodedPath)
- Query parameter documentation with types and defaults
- Null rules, pagination conventions, error response format
- Frontend consumer matrix (which page reads which WS fields)
- Payload type and route type reference tables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- build-node depends only on node-test
- build-go depends only on go-test
- deploy-node depends only on build-node
- deploy-go depends only on build-go
- publish job waits for both deploy-node and deploy-go to complete
- Badges and deployment summary moved to final publish step
Result: Go staging no longer waits for Node tests to complete.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go staging now deploys immediately after build completes, in parallel
with Node staging. Both test suites still gate the build job.
Before:
go-test + node-test → build → deploy-node → deploy-go
After:
go-test + node-test → build → deploy-node (parallel)
deploy-go (parallel)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend packets.js filters WS messages with m.data?.packet and
extracts m.data.packet for live rendering. Node's server.js includes
a packet sub-object (packet: fullPacket) in the broadcast data, but
Go's IngestNewFromDB built the data flat without a nested packet field.
This caused the Go staging packets page to never live-update via WS
even though messages were being sent — they were silently filtered out
by packets.js.
Fix: build the packet fields map separately, then create the broadcast
map with both top-level fields (for live.js) and nested packet (for
packets.js). Also fixes the fallback DB-direct poller path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env var for Node.js 20 deprecation
- Add cache-dependency-path for go.sum files in cmd/server and cmd/ingestor
- Add if-no-files-found: ignore to go-badges upload-artifact step
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Split the monolithic 3-job pipeline (go-build, test, deploy) into 5
focused jobs that each do ONE thing:
go-test - Go Build & Test (coverage badges, runs on ubuntu-latest)
node-test - Node.js Tests (backend + Playwright E2E, coverage)
build - Build Docker Images (Node + Go, badge publishing)
deploy-node - Deploy Node Staging (port 81, healthcheck, smoke test)
deploy-go - Deploy Go Staging (port 82, healthcheck, smoke test)
Dependency chain: go-test + node-test (parallel) -> build -> deploy-node -> deploy-go
Every step now has a human-readable name describing exactly what it does.
Job names include emoji for visual scanning on GitHub Actions.
All existing functionality preserved - just reorganized for clarity.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#156: The Go WebSocket broadcast from IngestNewFromDB was missing the
'decoded' field (with header.payloadTypeName) that live.js needs to
display packet types. Added decoded object with payloadTypeName
resolution, plus observer_id, observer_name, snr, rssi, path_json,
and observation_count fields to match the Node.js broadcast shape.
#157-160: Analytics endpoints (hash-sizes, topology, channels) were
recomputing everything on every request. Added:
- TTL caching (15s) for topology, hash-sizes, and channels endpoints
(matching the existing RF cache pattern)
- Cached node list + prefix map shared across analytics (30s TTL)
- Lazy-cached parsed path JSON on StoreTx (parse once, read many)
- Cache invalidation on new data ingestion
- Global payloadTypeNames map (avoids per-call allocation)
Fixes#156, fixes#157, fixes#158, fixes#159, fixes#160
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- db.go: Add freelistMB (PRAGMA freelist_count * page_size) and walPages
(PRAGMA wal_checkpoint(PASSIVE)) to GetDBSizeStats
- store.go: Add advertByObserver count to GetPerfStoreStats indexes
(count distinct pubkeys with ADVERT observations)
- db.go: Add getObservationsForTransmissions helper; enrich
GetRecentTransmissionsForNode results with observations array,
_parsedPath, and _parsedDecoded
- db_test.go: Add second ADVERT with different hash_size to seed data
so hash_sizes_seen is populated; enrich decoded_json with full
ADVERT fields; update count assertions for new seed row
fixes#151, fixes#152
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When engine=go, the perf page now renders Go-specific runtime stats
(goroutines, GC collections, GC pause times, heap breakdown, CPUs)
instead of the misleading Node.js Event Loop metrics. Falls back to
the existing Node UI when engine is not 'go' or goRuntime data is
missing. Includes color-coded GC pause thresholds.
fixes#153
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Frontend reads ps.inMemory.toLocaleString() which crashed because
the Go response was missing inMemory, sqliteOnly, maxPackets, maxMB,
evicted, inserts, queries fields. Added all + atomic counters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend perf dashboard crashes with toLocaleString error because
the Go /api/perf endpoint was missing fields that Node returns:
- status: "ok"
- uptimeHuman: formatted "Xh Ym" string
- websocket: { clients: N } via hub.ClientCount()
The Go /api/health endpoint already had all three fields.
fixes#150
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix#148: channels endpoint returned null for msgLengths when no
decrypted messages exist. Initialize msgLengths as make([]int, 0)
in store path and guard channels slice in DB fallback path.
- Fix#149: nodes endpoint always returned hash_size=null and
hash_size_inconsistent=false. Add GetNodeHashSizeInfo() to
PacketStore that scans advert packets to compute per-node hash
size, flip-flop detection, and sizes_seen. Enrich nodes in both
handleNodes and handleNodeDetail with computed hash data.
fixes#148, fixes#149
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go's /api/health was missing the eventLoop object that Node.js provides.
The perf.js frontend reads health.eventLoop.p95Ms which crashed with
'Cannot read properties of undefined' when served by the Go server.
Adds eventLoop field using GC pause data from runtime.MemStats.PauseNs
(last 256 pauses) to compute p50Ms, p95Ms, p99Ms, currentLagMs, maxLagMs
— matching the Node.js response shape exactly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go analytics topology endpoint was counting every unique hop string
from packet paths (including unresolved 1-byte hex prefixes) as a unique
node, inflating the count from ~540 to 6502. Now resolves each hop via
the prefix map and deduplicates by public key, matching the Node.js
behavior.
fixes#146
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- /api/perf: add goRuntime (heap, GC, goroutines, CPU), packetStore
stats (totalLoaded, observations, index sizes, estimatedMB),
sqlite stats (dbSizeMB, walSizeMB, row counts), real RF cache
hit/miss tracking, and endpoint sorting by total time spent
- /api/health: add memory.heapMB, goRuntime (goroutines, gcPauses,
numCPU), real packetStore packet count and estimatedMB, real
cache stats from RF cache; remove hardcoded-zero eventLoop
- store.go: add cacheHits/cacheMisses tracking in GetAnalyticsRF,
GetPerfStoreStats() and GetCacheStats() methods
- db.go: add path field to DB struct, GetDBSizeStats() for file
sizes and row counts
- Tests: verify new fields in health/perf endpoints, add
TestGetDBSizeStats, wire up PacketStore in test server setup
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
