- #178: Use strftime ISO 8601 format instead of datetime() for observation
timestamps in all SQL queries (v3 + v2 views). Add normalizeTimestamp()
helper for non-v3 paths that may store space-separated timestamps.
- #179: Strip internal fields (decoded_json, direction, payload_type,
raw_hex, route_type, score, created_at) from ObservationResp. Only
expose id, transmission_id, observer_id, observer_name, snr, rssi,
path_json, timestamp — matching Node.js parity.
- #180: Remove _parsedDecoded and _parsedPath from node detail
recentAdverts response. These internal/computed fields were leaking
to the API. Updated golden shapes.json accordingly.
- #181: Use mux route template (GetPathTemplate) for perf stats path
normalization, converting {param} to :param for Node.js parity.
Fallback to hex regex for unmatched routes. Compile regexes once at
package level instead of per-request.
fixes#178, fixes#179, fixes#180, fixes#181
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Deduplicated protobuf contract + fixture directive into single entry.
Protobuf API contract is now single source of truth for all frontend/backend interfaces, with fixture capture running against prod (stable).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go ingestor never had channel decryption — GRP_TXT packets were stored
with raw encrypted data while Node.js decoded them successfully.
Changes:
- decoder.go: Add decryptChannelMessage() implementing MeshCore channel
crypto (HMAC-SHA256 MAC verification + AES-128-ECB decryption), matching
the algorithm in @michaelhart/meshcore-decoder. Update decodeGrpTxt(),
decodePayload(), and DecodePacket() to accept and pass channel keys.
Add Payload fields: ChannelHashHex, DecryptionStatus, Channel, Text,
Sender, SenderTimestamp.
- config.go: Add ChannelKeysPath and ChannelKeys fields to Config struct.
- main.go: Add loadChannelKeys() that loads channel-rainbow.json (same
file used by Node.js server) from beside the config file, with env var
and config overrides. Pass loaded keys through the decoder pipeline.
- decoder_test.go: Add 14 channel decryption tests covering valid
decryption, MAC failure, wrong key, no-sender messages, bracket
sender exclusion, key iteration, channelHashHex formatting, and
decryption status states. Cross-validated against Node.js output.
- Update all DecodePacket/decodePayload/decodeGrpTxt/handleMessage call
sites in test files to pass the new channelKeys parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Automated script that compares all 13 major API endpoints between
Go staging (meshcore-staging-go) and Node prod (meshcore-prod)
containers. Uses python3 for JSON field diffing and reports
MATCH/PARTIAL/MISMATCH per endpoint.
Usage: scp to server then run, or pipe via ssh.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The env var was overriding the config and forcing Go staging to only
connect to its own empty local mosquitto, missing all external data.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The poller only queried WHERE t.id > sinceID, which missed new
observations added to transmissions already in the store. The trace
page was correct because it always queries the DB directly.
Add IngestNewObservations() that polls observations by o.id watermark,
adds them to existing StoreTx entries, re-picks best observation, and
invalidates analytics caches. The Poller now tracks both lastTxID and
lastObsID watermarks.
Includes tests for v3, v2, dedup, best-path re-pick, and
GetMaxObservationID.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Change .traces-page max-width from 1000px to 95vw via CSS variable
(--trace-max-width) and center with margin: 0 auto
- Increase SVG path graph column spacing from 140px to 200px so nodes
and labels don't overlap
- Bump cache busters
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#165 — build_time in API: already implemented (BuildTime ldflags in
Dockerfile.go, main.go, StatsResponse, HealthResponse)
#168 — subpaths API slow: cache (subpathCache with TTL) and invalidation
already in place; verified working
#169 — distance API slow: cache (distCache with TTL) and invalidation
already in place; verified working
#170 — audio-lab/buckets: in-memory store path already implemented,
matching Node.js pktStore.packets iteration with type grouping and
size-distributed sampling
#171 — channels stale latest message: add companion bridge handling to
Go ingestor for meshcore/message/channel/<n> and meshcore/message/direct/<id>
MQTT topics. Stores decoded channel messages with type CHAN in decoded_json,
enabling the channels endpoint to find them. Also handles direct messages.
#172 — packets page not live-updating: add missing direction field to WS
broadcast packet map for full parity with txToMap/Node.js fullPacket shape.
WS broadcast shape verified correct (type, data.packet structure, timestamp,
payload_type, observer_id all present).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
db.GetChannels() queried packets_v (observation-level rows) ordered by
observation timestamp and always overwrote lastMessage. When an older
message had a later re-observation, it would overwrite the correct
latest message with stale data.
Fix: query transmissions table directly (one row per unique message)
ordered by first_seen. This ensures lastMessage always reflects the
most recently sent message, not the most recently observed one.
Also fix db.GetChannelMessages() to use first_seen ordering with
schema-aware queries (v2/v3), and add missing distCache/subpathCache
invalidation on packet ingestion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use s.store (in-memory PacketStore) instead of direct packets_v SQL query,
matching how the Node.js handler iterates pktStore.packets. This fixes the
endpoint returning empty buckets when packets_v view is missing or the DB
query fails silently.
Key changes:
- Group by decoded_json.type first, fall back to payloadTypeNames
- Evenly-spaced sampling (up to 8 per type) sorted by raw_hex length
- Use actual ObservationCount instead of hardcoded 1
- Reuse payloadTypeNames from store.go instead of duplicating
- Retain DB fallback path when store is nil
fixes#170
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Added decision: Proto fixture capture in CI runs against prod (stable reference)
- Added decision: Protobuf API contract architecture (single source of truth for frontend/backend interfaces)
- Merged 2 directives from spawn manifest processing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 15s TTL cache to GetAnalyticsSubpaths with composite key (region|minLen|maxLen|limit),
matching the existing cache pattern used by RF, topology, hash, channel, and distance analytics.
Cache hits return instantly vs 900ms+ computation. fixes#168
- Add BuildTime to /api/stats and /api/health responses, injected via ldflags at build time.
Dockerfile.go now accepts BUILD_TIME build arg. fixes#165
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The proto validation infrastructure was added in commit e70ba44 but used
an invalid --syntax_check flag. Changed to use --descriptor_set_out=/dev/null
which validates syntax without generating files.
Proto validation flow (now complete):
1. go-test job: verify .proto files compile (syntax check) ✅
2. deploy-node job: validate protos match prod API responses ✅
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go server's WebSocket broadcast included first_seen but not
timestamp in the nested packet object. The frontend packets.js
filters on m.data.packet and reads p.timestamp for row insertion
and sorting. Without this field, live-updating silently failed
(rows inserted with undefined latest, breaking display).
Mirrors the pattern already used in txToMap() (store.go:1168)
which correctly emits both first_seen and timestamp.
Also updates websocket_test.go to assert timestamp presence
in broadcast data to prevent regression.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The build-node job was failing with 'node: not found' because it
runs scripts/validate.sh (which uses 'node -c' for syntax checking)
but didn't have the actions/setup-node@v4 step.
Added Node.js 22 setup before the validate step to match the pattern
used in other jobs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The distance analytics endpoint was recomputing haversine distances on
every request (~1.22s). Add a 15s TTL cache following the same pattern
used by RF, topology, hash-sizes, and channels analytics endpoints.
Include distCache in cache stats size calculation.
fixes#169
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously only captured 19 simple endpoints. Now captures all 33:
- 19 simple endpoints (stats, health, nodes, etc.)
- 14 dynamic ID endpoints (node-detail, packet-detail, etc.)
Dynamic ID resolution:
- Extracts real pubkey from /api/nodes for node detail endpoints
- Extracts real hash from /api/packets for packet-detail
- Extracts real observer ID from /api/observers for observer endpoints
- Gracefully skips fixtures if DB is empty (no data yet)
WebSocket capture:
- Uses node -e with ws module to capture one live WS message
- Falls back gracefully if no live packets available
The validator already handles missing fixtures without failing, so this
will work even when staging container has no data yet.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Added a CI step that:
- Refreshes Node fixtures from the staging container after deployment
- Runs tools/validate-protos.py to validate proto definitions match actual API responses
- Fails the pipeline if proto drift is detected
This ensures nobody can merge a Node change that breaks the Go proto contract
without updating the .proto definitions.
The step runs after the Node staging healthcheck, capturing fresh responses
from 19 API endpoints (stats, health, nodes, analytics/*, config/*, etc.).
Endpoints requiring parameters (node-detail, packet-detail) use existing
fixtures and aren't auto-refreshed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validates all 33 captured Node fixtures against the 10 .proto files.
Parses proto message definitions, maps each fixture to its response
message, and checks field presence, types, and structural shape.
Mismatches found (for Hicks to fix):
1. analytics-channels.json: ChannelAnalyticsSummary.hash is int in
fixture but proto says string
2. analytics-rf.json: PayloadTypeEntry.type is null in fixture but
proto says non-optional int32
3. bulk-health.json: API returns bare array with flat node fields;
proto nests them in BulkHealthEntry.node (structural mismatch)
4. node-analytics.json: activityTimeline uses 'bucket' key but
TimeBucket proto expects 'label'
5. observer-analytics.json: recentPackets are Observation-shaped
(have transmission_id) but proto says repeated Transmission
6. packet-detail.json: ByteRange has 'color' field not in proto
7. websocket-message.json: DecodedResult missing transportCodes,
raw fields; DecodedHeader missing routeTypeName; DecodedPayload
is flat (not oneof-wrapped); WSPacketData.packet is Observation-
shaped, not Transmission-shaped
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go staging packets page wasn't live-updating because the deployed
binary was stale (built before the #162 fix). Rebuilding from current
source fixed the issue — broadcasts now fire correctly.
Added two permanent diagnostic log lines:
- [poller] IngestNewFromDB: logs when new transmissions are found
- [broadcast] sending N packets to M clients: logs each broadcast batch
These log lines make it easy to verify the broadcast pipeline is working
and would have caught this stale-deployment issue immediately.
Verified on VM: WS clients receive packets with nested 'packet' field,
all Go tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Capture real responses from all 32 REST endpoints + 1 WebSocket
message from the production MeshCore Analyzer instance. Fixtures
include nodes, packets, observers, channels, analytics, config,
and health endpoints with real IDs substituted.
Stored in proto/testdata/node-fixtures/ for Go port contract testing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the exact response shape, query parameters, and type information
for every endpoint in server.js. This is the authoritative contract that
both Node.js and Go backends must conform to.
Covers:
- All 30+ REST endpoints with full JSON response schemas
- WebSocket message envelope and data shapes
- Shared object shapes (Packet, Observation, DecodedHeader, DecodedPath)
- Query parameter documentation with types and defaults
- Null rules, pagination conventions, error response format
- Frontend consumer matrix (which page reads which WS fields)
- Payload type and route type reference tables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- build-node depends only on node-test
- build-go depends only on go-test
- deploy-node depends only on build-node
- deploy-go depends only on build-go
- publish job waits for both deploy-node and deploy-go to complete
- Badges and deployment summary moved to final publish step
Result: Go staging no longer waits for Node tests to complete.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go staging now deploys immediately after build completes, in parallel
with Node staging. Both test suites still gate the build job.
Before:
go-test + node-test → build → deploy-node → deploy-go
After:
go-test + node-test → build → deploy-node (parallel)
deploy-go (parallel)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend packets.js filters WS messages with m.data?.packet and
extracts m.data.packet for live rendering. Node's server.js includes
a packet sub-object (packet: fullPacket) in the broadcast data, but
Go's IngestNewFromDB built the data flat without a nested packet field.
This caused the Go staging packets page to never live-update via WS
even though messages were being sent — they were silently filtered out
by packets.js.
Fix: build the packet fields map separately, then create the broadcast
map with both top-level fields (for live.js) and nested packet (for
packets.js). Also fixes the fallback DB-direct poller path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env var for Node.js 20 deprecation
- Add cache-dependency-path for go.sum files in cmd/server and cmd/ingestor
- Add if-no-files-found: ignore to go-badges upload-artifact step
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Split the monolithic 3-job pipeline (go-build, test, deploy) into 5
focused jobs that each do ONE thing:
go-test - Go Build & Test (coverage badges, runs on ubuntu-latest)
node-test - Node.js Tests (backend + Playwright E2E, coverage)
build - Build Docker Images (Node + Go, badge publishing)
deploy-node - Deploy Node Staging (port 81, healthcheck, smoke test)
deploy-go - Deploy Go Staging (port 82, healthcheck, smoke test)
Dependency chain: go-test + node-test (parallel) -> build -> deploy-node -> deploy-go
Every step now has a human-readable name describing exactly what it does.
Job names include emoji for visual scanning on GitHub Actions.
All existing functionality preserved - just reorganized for clarity.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#156: The Go WebSocket broadcast from IngestNewFromDB was missing the
'decoded' field (with header.payloadTypeName) that live.js needs to
display packet types. Added decoded object with payloadTypeName
resolution, plus observer_id, observer_name, snr, rssi, path_json,
and observation_count fields to match the Node.js broadcast shape.
#157-160: Analytics endpoints (hash-sizes, topology, channels) were
recomputing everything on every request. Added:
- TTL caching (15s) for topology, hash-sizes, and channels endpoints
(matching the existing RF cache pattern)
- Cached node list + prefix map shared across analytics (30s TTL)
- Lazy-cached parsed path JSON on StoreTx (parse once, read many)
- Cache invalidation on new data ingestion
- Global payloadTypeNames map (avoids per-call allocation)
Fixes#156, fixes#157, fixes#158, fixes#159, fixes#160
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- db.go: Add freelistMB (PRAGMA freelist_count * page_size) and walPages
(PRAGMA wal_checkpoint(PASSIVE)) to GetDBSizeStats
- store.go: Add advertByObserver count to GetPerfStoreStats indexes
(count distinct pubkeys with ADVERT observations)
- db.go: Add getObservationsForTransmissions helper; enrich
GetRecentTransmissionsForNode results with observations array,
_parsedPath, and _parsedDecoded
- db_test.go: Add second ADVERT with different hash_size to seed data
so hash_sizes_seen is populated; enrich decoded_json with full
ADVERT fields; update count assertions for new seed row
fixes#151, fixes#152
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When engine=go, the perf page now renders Go-specific runtime stats
(goroutines, GC collections, GC pause times, heap breakdown, CPUs)
instead of the misleading Node.js Event Loop metrics. Falls back to
the existing Node UI when engine is not 'go' or goRuntime data is
missing. Includes color-coded GC pause thresholds.
fixes#153
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Frontend reads ps.inMemory.toLocaleString() which crashed because
the Go response was missing inMemory, sqliteOnly, maxPackets, maxMB,
evicted, inserts, queries fields. Added all + atomic counters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend perf dashboard crashes with toLocaleString error because
the Go /api/perf endpoint was missing fields that Node returns:
- status: "ok"
- uptimeHuman: formatted "Xh Ym" string
- websocket: { clients: N } via hub.ClientCount()
The Go /api/health endpoint already had all three fields.
fixes#150
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
