Remove DB fallback paths from all route handlers. The in-memory
PacketStore now handles all packet/node/analytics queries. Handlers
return empty results or 404 when no store is available instead of
falling back to direct DB queries.
- Remove else-DB branches from handlePacketDetail, handleNodeHealth,
handleNodeAnalytics, handleBulkHealth, handlePacketTimestamps, etc.
- Remove unused DB methods (GetPacketByHash, GetTransmissionByID,
GetPacketByID, GetObservationsForHash, GetTimestamps, GetNodeHealth,
GetNodeAnalytics, GetBulkHealth, etc.)
- Remove packets_v VIEW creation from schema
- Update tests for new behavior (no-store returns 404/empty, not 500)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The glob trick COPY .git-commi[t] only works with BuildKit.
manage.sh uses legacy docker build. Just create a default via RUN.
Commit hash comes through --build-arg ldflags anyway.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Server defaults to 6060, ingestor to 6061. Removed shared PPROF_PORT
env var. Bind failure logs warning instead of log.Fatal killing the process.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fresh Go installs failed with 'no such table: packets_v' because the
ingestor created tables but never the VIEW that the Go server queries.
Add DROP VIEW IF EXISTS + CREATE VIEW packets_v to applySchema(), using
the v3 definition (observer_idx → observers.rowid JOIN). The view is
rebuilt on every startup to stay current with any definition changes.
Add tests: verify view exists after OpenStore, and verify it returns
correct observer_id/observer_name via the LEFT JOIN.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add net/http/pprof support to both Go server (default port 6060) and
ingestor (default port 6061). Profiling is off by default — only
starts the pprof HTTP listener when ENABLE_PPROF=true.
PPROF_PORT env var overrides the default port for each binary.
Enable on staging-go in docker-compose with exposed ports 6060/6061.
Not enabled on prod.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SQLite stores these as REAL on some instances. Go *int scan silently
fails, dropping the entire observer row (404 on detail, missing from list).
Reported for YC-Base-Repeater and YC-Work-Repeater.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Observability:
- Add DBStats struct with atomic counters for tx_inserted, tx_dupes,
obs_inserted, node_upserts, observer_upserts, write_errors
- Log SQLite config on startup (busy_timeout, max_open_conns, journal)
- Periodic stats logging every 5 minutes + final stats on shutdown
- Instrument all write paths with counter increments
Tests:
- TestConcurrentWrites: 20 goroutines × 50 writes (1000 total) with
interleaved InsertTransmission + UpsertNode + UpsertObserver calls.
Verifies zero errors and data integrity under concurrent load.
- TestDBStats: verifies counter accuracy for inserts, duplicates,
upserts, and that LogStats does not panic
Three changes to eliminate concurrent write collisions:
1. Add _busy_timeout=5000 to ingestor SQLite DSN (matches server)
- SQLite will wait up to 5s for the write lock instead of
immediately returning SQLITE_BUSY
2. Set SetMaxOpenConns(1) on ingestor DB connection pool
- Serializes all DB access at the Go sql.DB level
- Prevents multiple goroutines from opening overlapping writes
3. Change SetOrderMatters(false) to SetOrderMatters(true)
- MQTT handlers now run sequentially per client
- Eliminates concurrent handler execution that caused
overlapping multi-statement write flows
Root cause: concurrent MQTT handlers (SetOrderMatters=false) each
performed multiple separate writes (transmission lookup/insert,
observation insert, node upsert, observer upsert) without transactions
or connection limits. SQLite only permits one writer at a time, so
under bursty MQTT traffic the ingestor was competing with itself.
#210: Add role="img" aria-label to 9 Chart.js canvases in node-analytics.js
and observer-detail.js with descriptive labels.
#211: Add scope="col" to all <th> elements across analytics.js, audio-lab.js,
compare.js, node-analytics.js, nodes.js, observer-detail.js, observers.js,
and packets.js (40+ headers).
#212: Add aria-label to packet filter input and time window select in
packets.js. Add for/id associations to all customize.js inputs: branding,
theme colors, node/type colors, heatmap sliders, onboarding fields, and
export controls.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#203: Live page node detail panel becomes a bottom-sheet on mobile
(width:100%, bottom:0, max-height:60vh, rounded top corners).
#204: Perf page reduces padding to 12px, perf-cards stack in 2-col
grid, tables get smaller font/padding on mobile.
#205: Nodes table hides Public Key column on mobile via .col-pubkey
class (same pattern as packets page .col-region/.col-rpt).
Cache busters bumped in index.html.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build args ensure version badge shows correctly. Health timeout
bumped from 20s to 90s for Go store loading time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Changed all 'Node.js' references to generic 'Server' in:
- verify_health() - health check messages
- show_container_status() - stats display comment
- cmd_status() - service health output
The Go backend runs behind Caddy just like the Node version did,
so the health checks via docker exec localhost:3000 remain correct.
Only the messaging needed updating.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- app.js: render engine badge with .engine-badge span (was plain text)
- test: fix #pktRight waitForSelector to use state:'attached' (hidden by detail-collapsed)
- test: fix map heat persist race — wait for async init to restore checkbox state
- test: fix live heat persist race — test via localStorage set+reload instead of click
- test: fix live matrix toggle race — wait for Leaflet tiles before clicking
- test: increase packet detail timeouts for remote server resilience
- test: make close-button test self-contained (navigate if #pktRight missing)
- bump cache busters
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Problem 1 (Go staging timeout): Increased healthcheck from 60s to 120s to allow 50K+ packets to load into memory.
Problem 2 (Node staging timeout): Added forced cleanup of stale containers, volumes, and ports before starting staging containers to prevent conflicts.
Problem 3 (Proto validation WS timeout): Made WebSocket message capture non-blocking using timeout command. If no live packets are available, it now skips with a warning instead of failing the entire proto validation pipeline.
Problem 4 (Playwright E2E failures): Added forced cleanup of stale server on port 13581 before starting test server, plus better diagnostics on failure.
All health checks now include better logging (tail 50 instead of 30 lines) for debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UpsertNode only updates name/role/lat/lon/last_seen. The advert_count
field is modified exclusively by IncrementAdvertCount, which is called
separately in the MQTT handler. The test incorrectly expected count=2
after two UpsertNode calls; the correct value is 0 (the schema default).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Lead with performance stats and Go architecture. Update project
structure to reflect two-process model (Go server + Go ingestor).
Remove Node.js-specific sections (npm install, node server.js).
Keep screenshots, features, quick start, and deployment docs.
Add developer section with 380 Go tests + 150+ Node tests + E2E.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Go server is production-ready. Users upgrading via git pull + manage.sh
get Go automatically. No flags, no engine selection, no decision needed.
- Dockerfile (was Dockerfile.go) — Go multi-stage build
- Dockerfile.node — archived Node.js build for rollback
- docker-compose staging-go now builds from Dockerfile
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- User directive: Soft-delete nodes (inactive flag instead of deletion)
- Merged copilot-directive-soft-delete-nodes.md into Active Decisions section
- Removed processed inbox file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace expensive per-request distance computation (1.2s cold) with
precomputed distance index built during Load() and incrementally
updated on IngestNewFromDB/IngestNewObservations.
- Add distHopRecord/distPathRecord types for precomputed hop distances
- buildDistanceIndex() iterates all packets once during Load(), computing
haversine distances and storing results in distHops/distPaths slices
- computeDistancesForTx() handles per-packet distance computation,
shared between full rebuild and incremental ingest
- IngestNewFromDB appends distance records for new packets (no rebuild)
- IngestNewObservations triggers full rebuild only if paths changed
- computeAnalyticsDistance() now aggregates from precomputed records
instead of re-iterating all packets with JSON parsing + haversine
Cold request path: ~10-20ms (filter + sort precomputed records)
vs previous: ~1.2s (iterate 30K+ packets, parse JSON, resolve hops,
compute haversine for each).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Channels with garbage-decrypted names (pre-#197 data still in DB) are now
filtered at the API level using the same non-printable character heuristic
from #197. Applied in both Node.js server.js and Go server (store.go, db.go).
No data is deleted — only filtered from API responses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KpaBap