## Summary
Surfaces transport route types in the packets view by adding a **"T"
badge** next to the payload type badge for packets with
`TRANSPORT_FLOOD` (route type 0) or `TRANSPORT_DIRECT` (route type 3)
routes.
This helps mesh analysis — communities can quickly identify transported
packets and gain insights into scope usage adoption.
Closes#241
## What Changed
### Frontend (`public/`)
- **app.js**: Added `isTransportRoute(rt)` and `transportBadge(rt)`
helper functions that render a `<span class="badge
badge-transport">T</span>` badge with the full route type name as a
tooltip
- **packets.js**: Applied `transportBadge()` in all three packet row
render paths:
- Flat (ungrouped) packet rows
- Grouped packet header rows
- Grouped packet child rows
- **style.css**: Added `.badge-transport` class with amber styling and
CSS variable support (`--transport-badge-bg`, `--transport-badge-fg`)
for theme customization
### Backend (`cmd/server/`)
- **decoder_test.go**: Added 6 new tests covering:
- `TestDecodeHeader_TransportFlood` — verifies route type 0 decodes as
TRANSPORT_FLOOD
- `TestDecodeHeader_TransportDirect` — verifies route type 3 decodes as
TRANSPORT_DIRECT
- `TestDecodeHeader_Flood` — verifies route type 1 (non-transport)
decodes correctly
- `TestIsTransportRoute` — verifies the helper identifies transport vs
non-transport routes
- `TestDecodePacket_TransportFloodHasCodes` — verifies transport codes
are extracted from T_FLOOD packets
- `TestDecodePacket_FloodHasNoCodes` — verifies FLOOD packets have no
transport codes
## Visual
In the packets table Type column, transport packets now show:
```
[Channel Msg] [T] ← transport packet
[Channel Msg] ← normal flood packet
```
The "T" badge has an amber color scheme and shows the full route type
name on hover.
## Tests
- All Go tests pass (`cmd/server` and `cmd/ingestor`)
- All frontend tests pass (`test-packet-filter.js`, `test-aging.js`,
`test-frontend-helpers.js`)
- Cache busters bumped in `index.html`
---------
Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Fix: unprotect /api/decode from API key auth
Fixes#304
### Problem
PR #283 applied `requireAPIKey` to all POST endpoints including
`/api/decode`. But BYOP decode is a stateless read-only decoder — it
never writes to the database. Users see "write endpoints disabled" when
trying to decode packets.
### Fix
- Removed `requireAPIKey` wrapper from `/api/decode` in
`cmd/server/routes.go`
- Updated auth tests to use `/api/perf/reset` (actual write endpoint)
instead of `/api/decode`
- Added tests proving `/api/decode` works without API key, even when
apiKey is configured or empty
### Note
Decoder consolidation (`internal/decoder/` shared package) is tracked
separately and not included here to keep the PR clean.
### Tests
- `cd cmd/server && go test ./...` ✅
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary
- show relative build age next to the commit hash in the nav stats
version badge (e.g. `abc1234 (3h ago)`)
- use `stats.buildTime` from `/api/stats` and existing `timeAgo()`
formatting in `public/app.js`
- keep behavior unchanged when `buildTime` is missing/unknown
## What changed
- updated `formatVersionBadge()` signature to accept `buildTime`
- appended a `build-age` span after the commit link when `buildTime` is
valid
- passed `stats.buildTime` from `updateNavStats()`
- updated frontend helper tests for the new function signature
- added regression tests for build-age rendering/skip behavior
- bumped cache busters in `public/index.html`
## API check
- verified Go server already exposes `buildTime` on `/api/stats` and
`/api/health` via `cmd/server/routes.go`
- no backend API changes required
## Tests
- `node test-frontend-helpers.js`
- `node test-packet-filter.js`
- `node test-aging.js`
All passed locally.
## Browser validation
- Not run in this environment (no browser session available).
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pass region through channel message routes, apply DB/store filtering, normalize IATA at read and write boundaries, and add regression coverage for routes/server/ingestor.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Problem
Repeaters with 2-byte adverts occasionally appear as 1-byte on the map
and in stats.
**Root cause:** `computeNodeHashSizeInfo()` sets `HashSize` by
overwriting on every packet (`ni.HashSize = hs`), so the last advert
processed wins — regardless of how many previous packets correctly
showed 2-byte.
When a node sends an ADVERT directly (no relay hops), the path byte
encodes `hashCount=0`. Some firmware sets the full path byte to `0x00`
in this case, which decodes as `hashSize=1` even if the node normally
uses 2-byte hashes. If this packet happens to be the last one iterated,
the node shows as 1-byte.
## Fix
Compute the **mode** (most frequent hash size) across all observed
adverts instead of using the last-seen value. On a tie, prefer the
larger value.
```go
counts := make(map[int]int, len(ni.AllSizes))
for _, hs := range ni.Seq {
counts[hs]++
}
best, bestCount := 1, 0
for hs, cnt := range counts {
if cnt > bestCount || (cnt == bestCount && hs > best) {
best = hs
bestCount = cnt
}
}
ni.HashSize = best
```
A node with 4× hashSize=2 and 1× hashSize=1 now correctly reports
`HashSize=2`.
## Test
`TestGetNodeHashSizeInfoDominant`: seeds 5 adverts (4× 2-byte, 1×
1-byte) and asserts `HashSize=2`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Adds `GeoFilter` struct to `Config` in `cmd/server/config.go` so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by the
Go backend
- Adds `GET /api/config/geo-filter` endpoint in `cmd/server/routes.go`
returning the polygon + bufferKm to the frontend
- Restores the blue polygon overlay (solid inner + dashed buffer zone)
on the **Map** page (`public/map.js`)
- Restores the same overlay on the **Live** page (`public/live.js`),
toggled via the "Mesh live area" checkbox
## Test plan
- [x] `GET /api/config/geo-filter` returns `{ polygon: [...], bufferKm:
N }` when configured
- [x] `GET /api/config/geo-filter` returns `{ polygon: null, bufferKm: 0
}` when not configured
- [x] Map page shows blue polygon overlay when `geo_filter.polygon` is
set in config
- [x] Live page shows same overlay, checkbox state shared via
localStorage
- [x] Checkbox is hidden when no polygon is configured
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- added API key middleware for write routes in cmd/server/routes.go
- protected all current non-GET API routes (POST /api/packets, POST
/api/perf/reset, POST /api/decode)
- middleware enforces X-API-Key against cfg.APIKey and returns 401 JSON
error on missing/wrong key
- preserves backward compatibility: if piKey is empty, requests pass
through
- added startup warning log in cmd/server/main.go when no API key is
configured:
- [security] WARNING: no apiKey configured — write endpoints are
unprotected
- added route tests for missing/wrong/correct key and empty-apiKey
compatibility
## Validation
- cd cmd/server && go test ./... ✅
## Notes
- config.example.json already contains piKey, so no changes were
required.
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes#276
## Root cause
TRACE packets store hop IDs in the payload (bytes 9+) rather than in the
header path field. The header path field is overloaded in TRACE packets
to carry RSSI values instead of repeater IDs (as noted in the issue
comments). This meant `Path.Hops` was always empty for TRACE packets —
the raw bytes ended up as an opaque `PathData` hex string with no
structure.
The hashSize encoded in the header path byte (bits 6–7) is still valid
for TRACE and is used to split the payload path bytes into individual
hop prefixes.
## Fix
After decoding a TRACE payload, if `PathData` is non-empty, parse it
into individual hops using `path.HashSize`:
```go
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
path.Hops = append(path.Hops, ...)
}
}
}
```
Applied to both `cmd/ingestor/decoder.go` and `cmd/server/decoder.go`.
## Verification
Packet from the issue: `260001807dca00000000007d547d`
| | Before | After |
|---|---|---|
| `Path.Hops` | `[]` | `["7D", "54", "7D"]` |
| `Path.HashCount` | `0` | `3` |
New test `TestDecodeTracePathParsing` covers this exact packet.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
The in-memory `PacketStore` had **no eviction or aging** — it grew
unbounded until OOM killed the process. At ~3K packets/hour and ~5KB per
packet (not the 450 bytes previously estimated), an 8GB VM would OOM in
a few days.
## Changes
### Time-based eviction
- Configurable via `config.json`: `"packetStore": { "retentionHours": 24
}`
- Packets older than the retention window are evicted from the head of
the sorted slice
### Memory-based cap
- Configurable via `"packetStore": { "maxMemoryMB": 1024 }`
- Hard ceiling — evicts oldest packets when estimated memory exceeds the
cap
### Index cleanup
When a `StoreTx` is evicted, ALL associated data is removed from:
- `byHash`, `byTxID`, `byObsID`, `byObserver`, `byNode`, `byPayloadType`
- `nodeHashes`, `distHops`, `distPaths`, `spIndex`
### Periodic execution
- Background ticker runs eviction every 60 seconds
- Analytics caches and hash size cache are invalidated after eviction
### Stats fixes
- `estimatedMB` now uses ~5KB/packet + ~500B/observation (was 430B +
200B)
- `evicted` counter reflects actual evictions (was hardcoded to 0)
- Removed fake `maxPackets: 2386092` and `maxMB: 1024` from stats
### Config example
```json
{
"packetStore": {
"retentionHours": 24,
"maxMemoryMB": 1024
}
}
```
Both values default to 0 (unlimited) for backward compatibility.
## Tests
- 7 new tests in `eviction_test.go` covering time-based, memory-based,
index cleanup, thread safety, config parsing, and no-op when disabled
- All existing tests pass unchanged
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
## Problem
The RF analytics `packetsPerHour` chart was counting **observations**
instead of **unique transmissions** per hour. With ~34 observations per
transmission on average, the chart showed ~5,645 packets/hr instead of
the correct ~163/hr.
**Evidence from prod API:**
- `packetsPerHour` total: 1,580,620 (sum of all hourly counts)
- `totalPackets`: 45,764
- That's a ~34× inflation — exactly the observations-per-transmission
ratio
## Root Cause
In `store.go`, the `hourBuckets[hr]++` counter was inside the
observations loop (both regional and non-regional paths). Other counters
like `packetSizes` and `typeBuckets` already deduplicate by hash —
`hourBuckets` was the only one that didn't.
## Fix
Added a `seenHourHash` map (keyed by `hash|hour`) to deduplicate. Each
unique transmission is counted once per hour bucket, matching how packet
sizes and payload types already work.
Both the regional observer path and the non-regional path are fixed. The
legacy path (transmissions without observations) was already correct
since it iterates per-transmission.
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
## Summary
Adds `distributionByRepeaters` to the `/api/analytics/hash-sizes`
endpoint in the **Go server**.
### Problem
PR #263 implemented this feature in the deprecated Node.js server
(server.js). All backend changes should go in the Go server at
`cmd/server/`.
### Solution
- For each hash size (1, 2, 3), count how many unique repeaters (nodes)
advertise packets with that hash size
- Uses the existing `byNode` map already computed in
`computeAnalyticsHashSizes()`
- Added to both the live response and the empty/fallback response in
routes.go
- Frontend changes from PR #263 (`public/analytics.js`) already render
this field — no frontend changes needed
### Response shape
```json
{
"distributionByRepeaters": { "1": 42, "2": 7, "3": 2 },
...existing fields...
}
```
### Testing
- All Go server tests pass
- Replaces PR #263 (which modified the wrong server)
Closes#263
---------
Co-authored-by: you <you@example.com>
Change healthThresholds config from milliseconds to hours for readability.
Config keys: infraDegradedHours, infraSilentHours, nodeDegradedHours, nodeSilentHours.
Defaults: infra degraded 24h, silent 72h; node degraded 1h, silent 24h.
- Config stored in hours, converted to ms at comparison time
- /api/config/client sends ms to frontend (backward compatible)
- Frontend tooltips use dynamic thresholds instead of hardcoded strings
- Added healthThresholds section to config.example.json
- Updated Go and Node.js servers, tests
The poller's Start() calls GetMaxTransmissionID() to initialize its cursor.
When the test goroutine inserts data between go poller.Start() and the
actual GetMaxTransmissionID() call, the poller's cursor skips past the
test data and never broadcasts it, causing a timeout.
Adding a 100ms sleep after go poller.Start() ensures the poller has
initialized its cursors before the test inserts new data.
SQLite :memory: databases create separate databases per connection.
When the connection pool opens multiple connections (e.g. poller goroutine
vs main test goroutine), tables created on one connection are invisible
to others. Setting MaxOpenConns(1) ensures all queries use the same
in-memory database, fixing TestPollerBroadcastsMultipleObservations.
IngestNewFromDB now broadcasts one message per observation (not per
transmission). IngestNewObservations also broadcasts late arrivals.
Tests verify multi-observer packets produce multiple WS messages.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match the C++ firmware wire format (Packet::writeTo/readFrom):
1. Field order: transport codes are parsed BEFORE path_length byte,
matching firmware's header → transport_codes → path_len → path → payload
2. ACK payload: just 4-byte CRC checksum, not dest+src+ackHash.
Firmware createAck() writes only ack_crc (4 bytes).
3. TRACE payload: tag(4) + authCode(4) + flags(1) + pathData,
matching firmware createTrace() and onRecvPacket() TRACE handler.
4. ADVERT features: parse feat1 (0x20) and feat2 (0x40) optional
2-byte fields between location and name, matching AdvertDataBuilder
and AdvertDataParser in the firmware.
5. Transport code naming: code1/code2 instead of nextHop/lastHop,
matching firmware's transport_codes[0]/transport_codes[1] naming.
Fixes applied to both cmd/ingestor/decoder.go and cmd/server/decoder.go.
Tests updated to match new behavior.
- /api/stats: 10s server-side cache — was running 5 SQLite COUNT queries
on every call, taking ~1500ms with 28 concurrent WS clients polling every 15s
- GetNodeHashSizeInfo: 15s cache — was doing a full O(n) scan + JSON unmarshal
of all advert packets in memory on every /nodes request, taking ~1200ms
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Load() SQL: keep o.timestamp DESC (consistent with IngestNewFromDB) so
pickBestObservation tie-breaking is identical on both load paths
- GetTimestamps: scan from tail instead of head (was breaking on first item
assuming it was the newest, now correctly reads from newest end)
- QueryMultiNodePackets: apply same DESC/ASC tail-read pagination as
QueryPackets (was sorting for ASC and assuming DESC as-is)
- GetNodeHealth recentPackets: read from tail to return 20 newest items
(was reading from head = 20 oldest items)
- Remove stale "Prepend (newest first)" comments, replace with accurate
"oldest-first; new items go to tail" wording
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
s.packets and s.byPayloadType[t] were prepended on every new packet
to maintain newest-first order, copying the entire slice each time.
With 2-3M packets in memory this meant ~24MB of pointer copies per
ingest cycle, causing sustained high CPU and GC pressure.
Fix: store both slices oldest-first (append to tail). Load() SQL
changed to ASC ordering. QueryPackets DESC pagination now reads from
the tail in O(page_size) with no sort; GetChannelMessages switches
from reverse-iteration to forward-iteration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: remove all packets_v SQL fallbacks — store handles all queries
Remove DB fallback paths from all route handlers. The in-memory
PacketStore now handles all packet/node/analytics queries. Handlers
return empty results or 404 when no store is available instead of
falling back to direct DB queries.
- Remove else-DB branches from handlePacketDetail, handleNodeHealth,
handleNodeAnalytics, handleBulkHealth, handlePacketTimestamps, etc.
- Remove unused DB methods (GetPacketByHash, GetTransmissionByID,
GetPacketByID, GetObservationsForHash, GetTimestamps, GetNodeHealth,
GetNodeAnalytics, GetBulkHealth, etc.)
- Remove packets_v VIEW creation from schema
- Update tests for new behavior (no-store returns 404/empty, not 500)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: address PR #220 review comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: KpaBap <kpabap@gmail.com>
Server defaults to 6060, ingestor to 6061. Removed shared PPROF_PORT
env var. Bind failure logs warning instead of log.Fatal killing the process.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sensor nodes embed telemetry (battery_mv, temperature_c) in their advert
appdata after the null-terminated name. This commit adds decoding and
storage for both the Go ingestor and Node.js backend.
Changes:
- decoder.go/decoder.js: Parse telemetry bytes from advert appdata
(battery_mv as uint16 LE millivolts, temperature_c as int16 LE /100)
- db.go/db.js: Add battery_mv INTEGER and temperature_c REAL columns
to nodes and inactive_nodes tables, with migration for existing DBs
- main.go/server.js: Update node telemetry on advert processing
- server db.go: Include battery_mv/temperature_c in node API responses
- Tests: Decoder telemetry tests (positive, negative temp, no telemetry),
DB migration test, node telemetry update test, server API shape tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add net/http/pprof support to both Go server (default port 6060) and
ingestor (default port 6061). Profiling is off by default — only
starts the pprof HTTP listener when ENABLE_PPROF=true.
PPROF_PORT env var overrides the default port for each binary.
Enable on staging-go in docker-compose with exposed ports 6060/6061.
Not enabled on prod.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SQLite stores these as REAL on some instances. Go *int scan silently
fails, dropping the entire observer row (404 on detail, missing from list).
Reported for YC-Base-Repeater and YC-Work-Repeater.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace expensive per-request distance computation (1.2s cold) with
precomputed distance index built during Load() and incrementally
updated on IngestNewFromDB/IngestNewObservations.
- Add distHopRecord/distPathRecord types for precomputed hop distances
- buildDistanceIndex() iterates all packets once during Load(), computing
haversine distances and storing results in distHops/distPaths slices
- computeDistancesForTx() handles per-packet distance computation,
shared between full rebuild and incremental ingest
- IngestNewFromDB appends distance records for new packets (no rebuild)
- IngestNewObservations triggers full rebuild only if paths changed
- computeAnalyticsDistance() now aggregates from precomputed records
instead of re-iterating all packets with JSON parsing + haversine
Cold request path: ~10-20ms (filter + sort precomputed records)
vs previous: ~1.2s (iterate 30K+ packets, parse JSON, resolve hops,
compute haversine for each).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Channels with garbage-decrypted names (pre-#197 data still in DB) are now
filtered at the API level using the same non-printable character heuristic
from #197. Applied in both Node.js server.js and Go server (store.go, db.go).
No data is deleted — only filtered from API responses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The subpaths analytics endpoint iterated ALL packets on every cold query,
taking ~900ms. The TTL cache only masked the problem.
Fix: maintain a precomputed raw-hop subpath index (map[string]int) that
is built once during Load() and incrementally updated during
IngestNewFromDB() and IngestNewObservations().
At query time the fast path iterates only unique raw subpaths (typically
a few thousand entries) instead of all packets (30K+), resolves hop
prefixes to names, and merges counts. Region-filtered queries still
fall back to the O(N) path since they require per-transmission observer
checks.
Expected cold-hit improvement: ~900ms → <5ms for the common no-region
case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create inactive_nodes table with identical schema to nodes
- Add retention.nodeDays config (default 7) in Node.js and Go
- On startup: move nodes not seen in N days to inactive_nodes
- Daily timer (24h setInterval / goroutine ticker) repeats the move
- Log 'Moved X nodes to inactive_nodes (not seen in N days)'
- All existing queries unchanged — they only read nodes table
- Add 14 new tests for moveStaleNodes in test-db.js
- Both Node (db.js/server.js) and Go (ingestor/server) implemented
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. Update golden shapes.json goRuntime keys to match new struct fields
(goroutines, heapAllocMB, heapSysMB, etc. replacing heapMB, sysMB, etc.)
2. Fix analytics_hash_sizes hourly element shape — use explicit keys instead
of dynamicKeys to avoid flaky validation when map iteration picks 'hour'
string value against number valueShape
3. Update TestPerfEndpoint to check new goRuntime field names
4. Guard +Inf in handlePerf: use safeAvg() instead of raw division that
produces infinity when endpoint count is 0
5. Fix TestBroadcastMarshalError: use func(){} in map instead of chan int
to avoid channel-related marshal errors in test output
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IngestNewFromDB was appending new transmissions to byPayloadType slices,
breaking the newest-first ordering established by Load(). This caused
GetChannelMessages (which iterates backwards assuming newest-first) to
place newly ingested messages at the wrong position, making them invisible
when returning the latest messages from the tail.
Changed append to prepend, matching the existing s.packets prepend pattern
on line 881. Added regression test.
fixes#198
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#195 — /api/nodes/:pubkey/analytics was hitting SQL (packets_v view)
for all queries. Added store.GetNodeAnalytics(pubkey, days) that uses
the byNode[pubkey] index + text search through decoded_json, computing
all analytics (timeline, SNR trend, type breakdown, observer coverage,
hop distribution, peer interactions, uptime heatmap, computed stats)
entirely in-memory. Route handler now uses store path when available,
falling back to SQL only when store is nil.
#196 — recentPackets from /api/nodes/:pubkey/health were missing the
_parsedPath field that Node.js includes (lazy-cached parsed path_json
array). Added _parsedPath to txToMap() output using txGetParsedPath(),
matching the Node.js packet shape.
fixes#195, fixes#196
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Frontend reads goroutines/pauseTotalMs/lastPauseMs/heapAllocMB/heapSysMB/
heapInuseMB/heapIdleMB/numCPU but Go was returning heapMB/sysMB/numGoroutine/
gcPauseMs. All showed as undefined.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#191: Hash collision matrix now filters to role=repeater only (routing-relevant)
#192: expand=observations in /api/packets now returns full observation details (txToMap includes observations, stripped by default)
#193: /api/nodes/:pubkey/health uses in-memory PacketStore when available instead of slow SQL queries
#194: goRuntime (heapMB, sysMB, numGoroutine, numGC, gcPauseMs) restored in /api/perf response
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#184: Strip non-printable chars (<0x20 except tab/newline) from ADVERT
names in Go server decoder, Go ingestor decoder, and Node decoder.js.
#185: Add visual (N) badge next to node names when multiple nodes share
the same display name (case-insensitive). Shows in list, side pane, and
full detail page with 'also known as' links to other keys.
#186: Add packetsLast24h field to /api/stats response.
#187#188: Cache runtime.ReadMemStats() with 5s TTL in Go server.
#189: Temporarily patch HTMLCanvasElement.prototype.getContext during
L.heatLayer().addTo(map) to pass { willReadFrequently: true }, preventing
Chrome console warning about canvas readback performance.
Tests: 10 new tests for buildDupNameMap + dupNameBadge (143 total frontend).
Cache busters bumped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- #178: Use strftime ISO 8601 format instead of datetime() for observation
timestamps in all SQL queries (v3 + v2 views). Add normalizeTimestamp()
helper for non-v3 paths that may store space-separated timestamps.
- #179: Strip internal fields (decoded_json, direction, payload_type,
raw_hex, route_type, score, created_at) from ObservationResp. Only
expose id, transmission_id, observer_id, observer_name, snr, rssi,
path_json, timestamp — matching Node.js parity.
- #180: Remove _parsedDecoded and _parsedPath from node detail
recentAdverts response. These internal/computed fields were leaking
to the API. Updated golden shapes.json accordingly.
- #181: Use mux route template (GetPathTemplate) for perf stats path
normalization, converting {param} to :param for Node.js parity.
Fallback to hex regex for unmatched routes. Compile regexes once at
package level instead of per-request.
fixes#178, fixes#179, fixes#180, fixes#181
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The poller only queried WHERE t.id > sinceID, which missed new
observations added to transmissions already in the store. The trace
page was correct because it always queries the DB directly.
Add IngestNewObservations() that polls observations by o.id watermark,
adds them to existing StoreTx entries, re-picks best observation, and
invalidates analytics caches. The Poller now tracks both lastTxID and
lastObsID watermarks.
Includes tests for v3, v2, dedup, best-path re-pick, and
GetMaxObservationID.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#165 — build_time in API: already implemented (BuildTime ldflags in
Dockerfile.go, main.go, StatsResponse, HealthResponse)
#168 — subpaths API slow: cache (subpathCache with TTL) and invalidation
already in place; verified working
#169 — distance API slow: cache (distCache with TTL) and invalidation
already in place; verified working
#170 — audio-lab/buckets: in-memory store path already implemented,
matching Node.js pktStore.packets iteration with type grouping and
size-distributed sampling
#171 — channels stale latest message: add companion bridge handling to
Go ingestor for meshcore/message/channel/<n> and meshcore/message/direct/<id>
MQTT topics. Stores decoded channel messages with type CHAN in decoded_json,
enabling the channels endpoint to find them. Also handles direct messages.
#172 — packets page not live-updating: add missing direction field to WS
broadcast packet map for full parity with txToMap/Node.js fullPacket shape.
WS broadcast shape verified correct (type, data.packet structure, timestamp,
payload_type, observer_id all present).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
db.GetChannels() queried packets_v (observation-level rows) ordered by
observation timestamp and always overwrote lastMessage. When an older
message had a later re-observation, it would overwrite the correct
latest message with stale data.
Fix: query transmissions table directly (one row per unique message)
ordered by first_seen. This ensures lastMessage always reflects the
most recently sent message, not the most recently observed one.
Also fix db.GetChannelMessages() to use first_seen ordering with
schema-aware queries (v2/v3), and add missing distCache/subpathCache
invalidation on packet ingestion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use s.store (in-memory PacketStore) instead of direct packets_v SQL query,
matching how the Node.js handler iterates pktStore.packets. This fixes the
endpoint returning empty buckets when packets_v view is missing or the DB
query fails silently.
Key changes:
- Group by decoded_json.type first, fall back to payloadTypeNames
- Evenly-spaced sampling (up to 8 per type) sorted by raw_hex length
- Use actual ObservationCount instead of hardcoded 1
- Reuse payloadTypeNames from store.go instead of duplicating
- Retain DB fallback path when store is nil
fixes#170
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 15s TTL cache to GetAnalyticsSubpaths with composite key (region|minLen|maxLen|limit),
matching the existing cache pattern used by RF, topology, hash, channel, and distance analytics.
Cache hits return instantly vs 900ms+ computation. fixes#168
- Add BuildTime to /api/stats and /api/health responses, injected via ldflags at build time.
Dockerfile.go now accepts BUILD_TIME build arg. fixes#165
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Go server's WebSocket broadcast included first_seen but not
timestamp in the nested packet object. The frontend packets.js
filters on m.data.packet and reads p.timestamp for row insertion
and sorting. Without this field, live-updating silently failed
(rows inserted with undefined latest, breaking display).
Mirrors the pattern already used in txToMap() (store.go:1168)
which correctly emits both first_seen and timestamp.
Also updates websocket_test.go to assert timestamp presence
in broadcast data to prevent regression.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The distance analytics endpoint was recomputing haversine distances on
every request (~1.22s). Add a 15s TTL cache following the same pattern
used by RF, topology, hash-sizes, and channels analytics endpoints.
Include distCache in cache stats size calculation.
fixes#169
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
