## Summary
- When `groupByHash=true`, each group only carries its representative
(best-path) `observer_id`. The client-side filter was checking only that
field, silently dropping groups that were seen by the selected observer
but had a different representative.
- `loadPackets` now passes the `observer` param to the server so
`filterPackets`/`buildGroupedWhere` do the correct "any observation
matches" check.
- Client-side observer filter in `renderTableRows` is skipped for
grouped mode (server already filtered correctly).
- Both `db.go` and `store.go` observer filtering extended to support
comma-separated IDs (multi-select UI).
## Test plan
- [ ] Set an observer filter on the Packets screen with grouping enabled
— all groups that have **any** observation from the selected observer(s)
should appear, not just groups where that observer is the representative
- [ ] Multi-select two observers — groups seen by either should appear
- [ ] Toggle to flat (ungrouped) mode — per-observation filter still
works correctly
- [ ] Existing grouped packets tests pass: `cd cmd/server && go test
./...`
Fixes#464🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
## Summary
- `indexByNode` was calling `json.Unmarshal` for every packet during
`Load()` and `IngestNewFromDB()`, even channel messages and other
payloads that can never contain node pubkey fields
- All three target fields (`"pubKey"`, `"destPubKey"`, `"srcPubKey"`)
share the common substring `"ubKey"` — added a `strings.Contains`
pre-check that skips the JSON parse entirely for packets that don't
match
- At 30K+ packets on startup, this eliminates the majority of
`json.Unmarshal` calls in `indexByNode` (channel messages, status
packets, etc. all bypass it)
## Test plan
- [x] 5 new subtests in `TestIndexByNodePreCheck`: ADVERT with pubKey
indexed, destPubKey indexed, channel message skipped, empty JSON
skipped, duplicate hash deduped
- [x] All existing Go tests pass
- [x] Deploy to staging and verify node-filtered packet queries still
work correctly
Closes#376🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
## Summary
The in-memory `PacketStore` had **no eviction or aging** — it grew
unbounded until OOM killed the process. At ~3K packets/hour and ~5KB per
packet (not the 450 bytes previously estimated), an 8GB VM would OOM in
a few days.
## Changes
### Time-based eviction
- Configurable via `config.json`: `"packetStore": { "retentionHours": 24
}`
- Packets older than the retention window are evicted from the head of
the sorted slice
### Memory-based cap
- Configurable via `"packetStore": { "maxMemoryMB": 1024 }`
- Hard ceiling — evicts oldest packets when estimated memory exceeds the
cap
### Index cleanup
When a `StoreTx` is evicted, ALL associated data is removed from:
- `byHash`, `byTxID`, `byObsID`, `byObserver`, `byNode`, `byPayloadType`
- `nodeHashes`, `distHops`, `distPaths`, `spIndex`
### Periodic execution
- Background ticker runs eviction every 60 seconds
- Analytics caches and hash size cache are invalidated after eviction
### Stats fixes
- `estimatedMB` now uses ~5KB/packet + ~500B/observation (was 430B +
200B)
- `evicted` counter reflects actual evictions (was hardcoded to 0)
- Removed fake `maxPackets: 2386092` and `maxMB: 1024` from stats
### Config example
```json
{
"packetStore": {
"retentionHours": 24,
"maxMemoryMB": 1024
}
}
```
Both values default to 0 (unlimited) for backward compatibility.
## Tests
- 7 new tests in `eviction_test.go` covering time-based, memory-based,
index cleanup, thread safety, config parsing, and no-op when disabled
- All existing tests pass unchanged
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
SQLite :memory: databases create separate databases per connection.
When the connection pool opens multiple connections (e.g. poller goroutine
vs main test goroutine), tables created on one connection are invisible
to others. Setting MaxOpenConns(1) ensures all queries use the same
in-memory database, fixing TestPollerBroadcastsMultipleObservations.
IngestNewFromDB now broadcasts one message per observation (not per
transmission). IngestNewObservations also broadcasts late arrivals.
Tests verify multi-observer packets produce multiple WS messages.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* refactor: remove all packets_v SQL fallbacks — store handles all queries
Remove DB fallback paths from all route handlers. The in-memory
PacketStore now handles all packet/node/analytics queries. Handlers
return empty results or 404 when no store is available instead of
falling back to direct DB queries.
- Remove else-DB branches from handlePacketDetail, handleNodeHealth,
handleNodeAnalytics, handleBulkHealth, handlePacketTimestamps, etc.
- Remove unused DB methods (GetPacketByHash, GetTransmissionByID,
GetPacketByID, GetObservationsForHash, GetTimestamps, GetNodeHealth,
GetNodeAnalytics, GetBulkHealth, etc.)
- Remove packets_v VIEW creation from schema
- Update tests for new behavior (no-store returns 404/empty, not 500)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address PR #220 review comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: KpaBap <kpabap@gmail.com>
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The subpaths analytics endpoint iterated ALL packets on every cold query,
taking ~900ms. The TTL cache only masked the problem.
Fix: maintain a precomputed raw-hop subpath index (map[string]int) that
is built once during Load() and incrementally updated during
IngestNewFromDB() and IngestNewObservations().
At query time the fast path iterates only unique raw subpaths (typically
a few thousand entries) instead of all packets (30K+), resolves hop
prefixes to names, and merges counts. Region-filtered queries still
fall back to the O(N) path since they require per-transmission observer
checks.
Expected cold-hit improvement: ~900ms → <5ms for the common no-region
case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IngestNewFromDB was appending new transmissions to byPayloadType slices,
breaking the newest-first ordering established by Load(). This caused
GetChannelMessages (which iterates backwards assuming newest-first) to
place newly ingested messages at the wrong position, making them invisible
when returning the latest messages from the tail.
Changed append to prepend, matching the existing s.packets prepend pattern
on line 881. Added regression test.
fixes#198
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- #178: Use strftime ISO 8601 format instead of datetime() for observation
timestamps in all SQL queries (v3 + v2 views). Add normalizeTimestamp()
helper for non-v3 paths that may store space-separated timestamps.
- #179: Strip internal fields (decoded_json, direction, payload_type,
raw_hex, route_type, score, created_at) from ObservationResp. Only
expose id, transmission_id, observer_id, observer_name, snr, rssi,
path_json, timestamp — matching Node.js parity.
- #180: Remove _parsedDecoded and _parsedPath from node detail
recentAdverts response. These internal/computed fields were leaking
to the API. Updated golden shapes.json accordingly.
- #181: Use mux route template (GetPathTemplate) for perf stats path
normalization, converting {param} to :param for Node.js parity.
Fallback to hex regex for unmatched routes. Compile regexes once at
package level instead of per-request.
fixes#178, fixes#179, fixes#180, fixes#181
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The poller only queried WHERE t.id > sinceID, which missed new
observations added to transmissions already in the store. The trace
page was correct because it always queries the DB directly.
Add IngestNewObservations() that polls observations by o.id watermark,
adds them to existing StoreTx entries, re-picks best observation, and
invalidates analytics caches. The Poller now tracks both lastTxID and
lastObsID watermarks.
Includes tests for v3, v2, dedup, best-path re-pick, and
GetMaxObservationID.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The frontend packets.js filters WS messages with m.data?.packet and
extracts m.data.packet for live rendering. Node's server.js includes
a packet sub-object (packet: fullPacket) in the broadcast data, but
Go's IngestNewFromDB built the data flat without a nested packet field.
This caused the Go staging packets page to never live-update via WS
even though messages were being sent — they were silently filtered out
by packets.js.
Fix: build the packet fields map separately, then create the broadcast
map with both top-level fields (for live.js) and nested packet (for
packets.js). Also fixes the fallback DB-direct poller path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fixes#154: Go analytics channels showed single 'ch?' because
channelHash is a JSON number (from decoder.js) but the Go struct
declared it as string. json.Unmarshal failed on every packet.
Changed to interface{} with proper type conversion. Also fixed
chKey to use hash (not name) for grouping, matching Node.js.
fixes#155: uniqueNodes in topology analytics used hop resolution
count (phantom hops inflated it). Both Node.js and Go now use
db.getStats().totalNodes (7-day active window), matching /api/stats.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
efiten