feat(node-analytics): hop-count statistics per node (#1812) (#2021)

## Summary
Adds per-node hop-count statistics so repeater operators can choose
`flood.max`, `flood.max.unscoped` and `flood.max.advert` from what their
node actually sees.

- New endpoint `GET /api/nodes/{pubkey}/hop_analytics?days=N`
(`cmd/server/routes.go:299`, `cmd/server/node_hop_analytics.go:312`),
separate from `/analytics` as requested in the issue.
- New card "Hop Count at This Node" on the node analytics page
(`public/node-hop-analytics.js`, wired at
`public/node-analytics.js:130,174`): histogram of hop counts with a box
plot on the same x axis, filters `flood.max` (default),
`flood.max.advert`, `flood.max.unscoped`, driven by the existing range
picker.
- The existing "Hop Distribution" chart is unchanged: it shows path
length at the observer, a different quantity.
- No `direct` tag, although the issue lists one: for DIRECT packets the
path is the remaining route and no flood limit applies, so there is no
hop count to report.

## Hop count definition (firmware 0679dbef)
- `src/helpers/RoutingPolicy.h:15-21`: limits compare
`getPathHashCount()`; `.unscoped` applies to route type FLOOD, `.advert`
to adverts.
- `src/Mesh.cpp:344-350`: `routeRecvPacket` checks with n hashes in the
path, then writes its own hash at index n. So hops = the node's
zero-based index in the path, no +1.
- `src/Mesh.cpp:265-285`: a node forwards a flood once;
`src/Mesh.cpp:651,680`: an originator never forwards its own flood.
- DIRECT packets are excluded: their path is the remaining route
(`src/Mesh.cpp:78-103,334-341`).

Response: `{timeRange, packets: [{hash, timestamp, hops, tags}],
ambiguous}`. Tags: `flood`, `scoped` or `unscoped`, `advert`. Documented
in `docs/api-spec.md:679` and `cmd/server/openapi.go:90`.

## Attribution
`cmd/server/node_hop_analytics.go:198-309`. The result depends only on
the observed paths, the prefix map and the neighbor graph, so it is the
same after a restart as after live ingest.

- Every observation of every flood packet in the window is read.
`byNode` holds the server resolver's pick at ingest and other picks
after a cold load; `byPathHop` indexes only each packet's longest path,
which for a busy relay often runs through another branch of the flood.
- A packet counts when the node's prefix sits at exactly one index
across its observations, and either the node is the only relay candidate
for that prefix (`prefixMap.relayCandidates`,
`cmd/server/store.go:6795`), or the hop resolves to the node under the
ingestor's strict rule (`cmd/ingestor/path_resolver.go:143-214`) in at
least one observation and to another node in none. Strict rule: earlier
hops identified without a tiebreak, exactly one candidate adjacent in
`neighbor_edges` to the previous hop (the originator for hop 0 of an
advert), nodes already on the path excluded.
- The server resolver's tiebreaks (affinity, GPS distance, advert count,
pubkey order) are not used.
- Everything else with the node's prefix goes to `ambiguous`. In
practice that is most packets with a colliding 1-byte path hash.

On a read-only 7-day dump of a 1,669-node mesh DB, for one busy
repeater: 23,081 packets attributed, 11,437 ambiguous. Taking candidates
from `byPathHop` instead gave 9,995 attributed, with the histogram mode
moved from 2 to 3-5 hops.

## Performance
Scans `s.packets` under the read lock, no SQL per packet. Per
observation: one substring test for the node's first prefix byte; the
hop scan only for observations containing it; the strict walk only for
colliding prefixes, with per-request caches for candidates and
adjacency. `BenchmarkNodeHopPackets` models one 7-day request at that
scale (73,782 flood packets, 1,430,280 observations): 44-87 ms/op, 13.4
MB, 40 allocs on a throttling laptop.

Response size for that repeater over 7 days: about 23k entries, 2.3 MB
JSON, 375 KB gzipped. `hash` and `timestamp` are 61% of the raw and 91%
of the gzipped bytes; they stay because the issue asks for them so a
client can join entries to packets and bin by time.

## Tests
- Go: `cmd/server/node_hop_analytics_test.go`: 12 unit tests, a
live-ingest test through `IngestNewFromDB` (a colliding prefix without
independent attribution goes to `ambiguous`, not to the node the
resolver picked), live ingest versus cold load of the same DB, route
test, benchmark. 15 mutations of the attribution logic each fail a test.
- JS: `test-node-hop-analytics.js` (filters, histogram, quartiles and
whiskers with a fixture that separates 1.5 IQR from 3 IQR, render),
registered in `test-all.sh` and `.github/workflows/deploy.yml`.
- `gofmt`, `go vet ./...`, `go test ./...` in `cmd/server`,
`scripts/check-css-vars.js` pass.

## Staging validation
Build `c646310f` (this PR's review follow-up together with the other
open follow-ups), after a container restart and full load, on a busy
Belgian repeater:

- `hop_analytics?days=7`: 23,302 packets, 11,548 ambiguous, median 4,
adverts never above hop 7 (matching the firmware default
`flood_max_advert = 8`, `examples/simple_repeater/MyMesh.cpp:922`), 1.2
s. The first version reported 23,035 packets and 86 ambiguous in 534 ms,
because it trusted the resolver's pick for colliding prefixes.
- The card rendered on the first version with no console errors; the
rework does not touch the frontend beyond a test fixture.

## Not verified
- Response time and lock hold for 30 days on the busiest node on a
14-day store.
- Server relay candidates exclude companions and listeners while the
ingestor's prefix index does not, so a few strict attributions can
differ from the ingestor's persisted `resolved_path`.
- Identical numbers across a second container restart were shown in a Go
test, not repeated on staging.
- Dark theme, phone width, and switching the range picker in the
browser.
- Filter state is not reflected in the URL hash (the range picker is not
either).

Fixes #1812

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
efiten
2026-09-13 19:59:56 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 2d4019f719
commit a059299588
14 changed files with 1260 additions and 50 deletions
+1
View File
@@ -35,6 +35,7 @@
"MAX_HOP_DIST": "readonly",
"MeshAudio": "readonly",
"MeshConfigReady": "readonly",
"NodeHopAnalytics": "readonly",
"PAYLOAD_COLORS": "readonly",
"PAYLOAD_TYPES": "readonly",
"PERF_SLOW_MS": "readonly",
+1 -12
View File
@@ -183,18 +183,7 @@ func resolvePathForObsColdLoad(pathJSON, observerID string, tx *StoreTx, pm *pre
// observation_count_fallback would still pick a winner for
// ambiguous prefixes, which is exactly what we must NOT do.
// Hence the explicit candidate-count check here.
h := strings.ToLower(hop)
candidates := pm.m[h]
if len(pm.nonRelay) > 0 && len(candidates) > 0 {
filtered := candidates[:0:0]
for j := range candidates {
if _, isListener := pm.nonRelay[strings.ToLower(candidates[j].PublicKey)]; isListener {
continue
}
filtered = append(filtered, candidates[j])
}
candidates = filtered
}
candidates := pm.relayCandidates(hop)
if len(candidates) == 1 {
pk := strings.ToLower(candidates[0].PublicKey)
resolved[i] = &pk
+331
View File
@@ -0,0 +1,331 @@
package main
import (
"strings"
"time"
)
// Per-node hop count statistics (issue #1812).
//
// A repeater decides whether to forward a flood with
// isFloodHopLimitExceeded (firmware src/helpers/RoutingPolicy.h:15-21),
// comparing getPathHashCount() (src/Packet.h:80, path_len & 63) against
// flood.max, flood.max.unscoped (ROUTE_TYPE_FLOOD only) and flood.max.advert
// (PAYLOAD_TYPE_ADVERT only). Mesh::routeRecvPacket (src/Mesh.cpp:344-350)
// runs that check with n hashes in the path and then writes its own hash at
// index n. So the node's zero-based index in an observed flood path is the hop
// count its flood.max check saw. Firmware refs: meshcore-dev/MeshCore 0679dbef.
//
// Hash rules the attribution relies on:
// - A node forwards a given flood once (wasSeen/markSeen before
// routeRecvPacket, e.g. src/Mesh.cpp:265-285 for adverts), so it sits at
// one index in every observation that contains it. A prefix match at two
// different indices means another node shares the prefix.
// - All hops of one packet share one hash size (src/Packet.h:79), so every
// match in a packet is the same prefix string.
// - An originator marks its own flood seen before sending
// (Mesh::sendFlood, src/Mesh.cpp:651 and :680), so it never appears in
// that path.
// - DIRECT paths are the remaining route and shrink at every hop
// (removeSelfFromPath, src/Mesh.cpp:334-341); the flood.max limits only
// apply to floods (examples/simple_repeater/MyMesh.cpp:436-437). DIRECT
// packets carry no hop count for this purpose and are skipped.
var (
hopTagsUnscoped = []string{"flood", "unscoped"}
hopTagsScoped = []string{"flood", "scoped"}
hopTagsUnscopedAdvert = []string{"flood", "unscoped", "advert"}
hopTagsScopedAdvert = []string{"flood", "scoped", "advert"}
)
// nextPathHop returns the hop string starting after offset pos of a path_json
// array and the offset after it, or ok=false when no hop is left. path_json
// is a JSON array of hex hop strings, which carry no escapes, so hops are read
// between quotes without unmarshalling and without allocating: the scan runs
// under s.mu.RLock for every observation in the window.
func nextPathHop(p string, pos int) (hop string, next int, ok bool) {
open := strings.IndexByte(p[pos:], '"')
if open < 0 {
return "", pos, false
}
start := pos + open + 1
n := strings.IndexByte(p[start:], '"')
if n < 0 {
return "", pos, false
}
return p[start : start+n], start + n + 1, true
}
// hopAttributor decides whether a path hop is a given node without using the
// server resolver's tiebreaks (affinity score, GPS distance, advert count,
// pubkey order), which always name a winner among colliding prefixes.
// Candidates are prefixMap.relayCandidates; adjacency is the neighbor graph,
// which the server loads from the ingestor's neighbor_edges, the same table
// the ingestor's resolver reads. Caches live for one request.
type hopAttributor struct {
pm *prefixMap
graph *NeighborGraph
cands map[string][]string
adj map[string]map[string]struct{}
seen []string
}
func newHopAttributor(pm *prefixMap, graph *NeighborGraph) *hopAttributor {
return &hopAttributor{pm: pm, graph: graph, cands: map[string][]string{}, adj: map[string]map[string]struct{}{}}
}
// candidates returns the lowercase pubkeys of the relay candidates for a hop.
// The cache is keyed by the hop as given, so wire-case hops are looked up
// without lowercasing a copy.
func (a *hopAttributor) candidates(hop string) []string {
c, ok := a.cands[hop]
if !ok {
for _, n := range a.pm.relayCandidates(hop) {
c = append(c, strings.ToLower(n.PublicKey))
}
a.cands[hop] = c
}
return c
}
func (a *hopAttributor) adjacent(anchor, pk string) bool {
nbrs, ok := a.adj[anchor]
if !ok {
nbrs = map[string]struct{}{}
if a.graph != nil {
for _, e := range a.graph.Neighbors(anchor) {
if e.Ambiguous || e.NodeA == "" || e.NodeB == "" {
continue
}
other := e.NodeA
if other == anchor {
other = e.NodeB
}
nbrs[other] = struct{}{}
}
}
a.adj[anchor] = nbrs
}
_, ok = nbrs[pk]
return ok
}
// strictHopAt resolves the hop at index idx of path p the way the ingestor
// does (cmd/ingestor/path_resolver.go resolvePathWithContext): walking from
// hop 0, a hop with one candidate resolves to it, a hop with several resolves
// only when exactly one of them is a graph neighbor of the previous resolved
// hop (the advert originator for hop 0), and nodes already on the path are
// excluded. An unresolved hop breaks the chain for the next one.
//
// ok is false when p has no hop idx or that hop is not prefix. Otherwise pk is
// the lowercase pubkey the hop resolves to, or "" when it does not resolve,
// and end is the offset in p after hop idx.
func (a *hopAttributor) strictHopAt(p string, idx int, prefix, origin string) (pk string, end int, ok bool) {
anchor := origin
a.seen = a.seen[:0]
if anchor != "" {
a.seen = append(a.seen, anchor)
}
pos := 0
for i := 0; i <= idx; i++ {
hop, next, found := nextPathHop(p, pos)
if !found || (i == idx && !strings.EqualFold(hop, prefix)) {
return "", 0, false
}
pos = next
cands := a.candidates(hop)
match, survivors := "", 0
switch {
case len(cands) == 1:
if c := cands[0]; !a.onPath(c) {
match, survivors = c, 1
}
case len(cands) > 1 && anchor != "" && a.graph != nil:
for _, c := range cands {
if !a.onPath(c) && a.adjacent(anchor, c) {
match = c
survivors++
}
}
}
if survivors != 1 {
match = ""
}
if i == idx {
return match, pos, true
}
anchor = match
if match != "" {
a.seen = append(a.seen, match)
}
}
return "", 0, false
}
func (a *hopAttributor) onPath(pk string) bool {
for _, s := range a.seen {
if s == pk {
return true
}
}
return false
}
// computeNodeHopPackets returns one entry per flood packet in txs, first seen
// after fromISO, that the node forwarded, and how many packets carry the node's
// prefix at one index without being attributable to it.
//
// A packet counts when the node's prefix sits at exactly one index across all
// its observations, and that hop is the node independently of the server
// resolver's pick: either the node is the only relay candidate for the prefix,
// or, for a colliding prefix, at least one observation resolves that hop to the
// node under the ingestor's strict neighbor rule (hopAttributor.strictHopAt)
// and none resolves it to another node. Everything else with the node's prefix
// is counted as ambiguous and left out. The answer depends only on the
// observed paths, the prefix map and the neighbor graph, so it is the same
// after a restart as after live ingest.
//
// Every observation is read, not an index: byNode holds the resolver's pick at
// ingest and other picks after a cold load, and byPathHop only indexes each
// packet's longest path, which for a busy relay often runs through another
// branch of the flood (on a live 7-day sample it held 9,995 of the 23,081
// attributable packets of one repeater, skewed towards higher hop counts).
//
// txs must be deduplicated. Cost is linear in the observations of the flood
// packets in the window: a substring test per observation, the hop scan only
// for observations that contain the node's first prefix byte, and the strict
// walk only for colliding prefixes.
func computeNodeHopPackets(pubkey string, txs []*StoreTx, fromISO string, pm *prefixMap, graph *NeighborGraph) ([]NodeHopPacket, int) {
lowerPK := strings.ToLower(pubkey)
// Every hop hash of the node, whatever its size, starts with its first
// pubkey byte, and path hops are written in upper case
// (internal/packetpath/path.go:50; none of 1,495,712 live observations of
// 7 days had a lower-case hex digit), so an observation without that byte
// after an opening quote holds no hop of the node.
quotedFirstByte := `"` + strings.ToUpper(lowerPK[:2])
packets := make([]NodeHopPacket, 0)
ambiguous := 0
attr := newHopAttributor(pm, graph)
onlyCandidate := map[string]bool{}
for _, tx := range txs {
if tx.RouteType == nil || (*tx.RouteType != RouteFlood && *tx.RouteType != RouteTransportFlood) || tx.FirstSeen <= fromISO {
continue
}
idx, prefix, conflict := -1, "", false
for _, obs := range tx.Observations {
p := obs.PathJSON
if !strings.Contains(p, quotedFirstByte) {
continue
}
for i, pos := 0, 0; ; i++ {
hop, next, ok := nextPathHop(p, pos)
if !ok {
break
}
pos = next
if len(hop) == 0 || len(hop) > len(lowerPK) || !strings.EqualFold(hop, lowerPK[:len(hop)]) {
continue
}
if idx < 0 {
idx, prefix = i, lowerPK[:len(hop)]
} else if i != idx {
conflict = true
}
}
}
if idx < 0 {
continue
}
origin := ""
if tx.DecodedJSON != "" && strings.Contains(tx.DecodedJSON, "ubKey") {
origin = strings.ToLower(extractFromNode(tx))
if origin == lowerPK {
continue
}
}
if conflict {
ambiguous++
continue
}
advert := tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT
only, known := onlyCandidate[prefix]
if !known {
cands := attr.candidates(prefix)
only = len(cands) == 1 && cands[0] == lowerPK
onlyCandidate[prefix] = only
}
if !only && !attributeByNeighbor(attr, tx, idx, prefix, advert, origin, lowerPK) {
ambiguous++
continue
}
tags := hopTagsUnscoped
switch {
case *tx.RouteType == RouteTransportFlood && advert:
tags = hopTagsScopedAdvert
case *tx.RouteType == RouteTransportFlood:
tags = hopTagsScoped
case advert:
tags = hopTagsUnscopedAdvert
}
packets = append(packets, NodeHopPacket{Hash: tx.Hash, Timestamp: tx.FirstSeen, Hops: idx, Tags: tags})
}
return packets, ambiguous
}
// attributeByNeighbor reports whether the hop at idx, which carries prefix, is
// the node under the strict neighbor rule in at least one observation and
// another node in none. The ingestor anchors hop 0 on the originator only for
// adverts (cmd/ingestor/db.go FromPubkey), and so does this. An observation
// whose path text equals the previous walked one up to hop idx reuses its
// result.
func attributeByNeighbor(attr *hopAttributor, tx *StoreTx, idx int, prefix string, advert bool, origin, lowerPK string) bool {
if !advert {
origin = ""
}
self := false
walked, pk := "", ""
for _, obs := range tx.Observations {
if walked == "" || !strings.HasPrefix(obs.PathJSON, walked) {
r, end, ok := attr.strictHopAt(obs.PathJSON, idx, prefix, origin)
if !ok {
continue
}
walked, pk = obs.PathJSON[:end], r
}
switch pk {
case "":
case lowerPK:
self = true
default:
return false
}
}
return self
}
// GetNodeHopAnalytics returns the hop count at this node for every flood packet
// it forwarded in the last days. Returns nil for an unknown node.
func (s *PacketStore) GetNodeHopAnalytics(pubkey string, days int) (*NodeHopAnalyticsResponse, error) {
node, err := s.db.GetNodeByPubkey(pubkey)
if err != nil || node == nil {
return nil, err
}
now := time.Now()
fromISO := now.Add(-time.Duration(days) * 24 * time.Hour).Format(time.RFC3339)
s.mu.RLock()
_, pm := s.getCachedNodesAndPM()
packets, ambiguous := computeNodeHopPackets(pubkey, s.packets, fromISO, pm, s.graph.Load())
s.mu.RUnlock()
return &NodeHopAnalyticsResponse{
TimeRange: TimeRangeResp{From: fromISO, To: now.Format(time.RFC3339), Days: days},
Packets: packets,
Ambiguous: ambiguous,
}, nil
}
+504
View File
@@ -0,0 +1,504 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"reflect"
"sort"
"testing"
"time"
"github.com/gorilla/mux"
)
// Issue #1812: per-node hop count, as the repeater's own flood.max check sees
// it. The firmware compares getPathHashCount() (the number of hashes already in
// the path when the packet arrives) against the limits before appending its own
// hash, so the target's zero-based index in an observed flood path is exactly
// that count.
const (
hopTarget = "ab12cd34ef567890"
hopOther = "77aa88bb99cc0011"
hopCollider = "ab99ffee00112233" // shares the 1-byte prefix "ab" with hopTarget
hopListener = "ab12ffff00000000" // shares the 2-byte prefix "ab12", listener only
hopBridge = "88cc000000000000" // neighbor of hopCollider only
hopBoth = "55dd000000000000" // neighbor of hopTarget and hopCollider
hopTwinA = "99aa000000000000" // shares the 1-byte prefix "99" with hopTwinB
hopTwinB = "99bb000000000000"
hopTimestamp = "2026-09-10T10:00:00Z"
)
func hopTx(id int, hash string, routeType, payloadType int, paths ...string) *StoreTx {
rt, pt := routeType, payloadType
tx := &StoreTx{ID: id, Hash: hash, FirstSeen: hopTimestamp, RouteType: &rt, PayloadType: &pt}
for i, p := range paths {
tx.Observations = append(tx.Observations, &StoreObs{ID: id*10 + i, TransmissionID: id, PathJSON: p})
}
return tx
}
func hopPM() *prefixMap {
return buildPrefixMap([]nodeInfo{
{PublicKey: hopTarget, Role: "repeater"},
{PublicKey: hopOther, Role: "repeater"},
{PublicKey: hopCollider, Role: "repeater"},
{PublicKey: hopBridge, Role: "repeater"},
{PublicKey: hopBoth, Role: "repeater"},
{PublicKey: hopTwinA, Role: "repeater"},
{PublicKey: hopTwinB, Role: "repeater"},
})
}
// hopGraph: hopOther and hopTwinB neighbor hopTarget, hopBridge neighbors
// hopCollider, hopBoth neighbors both, and hopOther and hopBoth neighbor both
// twins.
func hopGraph() *NeighborGraph {
g := NewNeighborGraph()
now := time.Now()
for _, e := range [][2]string{
{hopOther, hopTarget},
{hopTwinB, hopTarget},
{hopBridge, hopCollider},
{hopBoth, hopTarget},
{hopBoth, hopCollider},
{hopOther, hopTwinA},
{hopOther, hopTwinB},
{hopBoth, hopTwinA},
{hopBoth, hopTwinB},
} {
g.upsertEdge(e[0], e[1], "", "obs", nil, now)
}
return g
}
func hopsOf(packets []NodeHopPacket) []int {
out := make([]int, 0, len(packets))
for _, p := range packets {
out = append(out, p.Hops)
}
return out
}
func hashesOf(packets []NodeHopPacket) []string {
out := make([]string, 0, len(packets))
for _, p := range packets {
out = append(out, p.Hash)
}
return out
}
func TestNodeHopPackets_IndexIsHopCountAtDifferentPositions(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["AB12"]`),
hopTx(2, "h2", RouteFlood, PayloadGRP_TXT, `["77AA","AB12"]`),
hopTx(3, "h3", RouteTransportFlood, PayloadGRP_TXT, `["77AA","77AA","77AA","AB12","77AA"]`),
}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if got, want := hopsOf(packets), []int{0, 1, 3}; !reflect.DeepEqual(got, want) {
t.Fatalf("hops = %v, want %v (zero-based index of the node in the path, no +1)", got, want)
}
if ambiguous != 0 {
t.Errorf("ambiguous = %d, want 0", ambiguous)
}
if packets[1].Hash != "h2" || packets[1].Timestamp != hopTimestamp {
t.Errorf("packet[1] = %+v, want hash h2 and timestamp %s", packets[1], hopTimestamp)
}
}
func TestNodeHopPackets_Tags(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "unscoped", RouteFlood, PayloadGRP_TXT, `["AB12"]`),
hopTx(2, "scoped", RouteTransportFlood, PayloadGRP_TXT, `["AB12"]`),
hopTx(3, "advert", RouteFlood, PayloadADVERT, `["AB12"]`),
hopTx(4, "scoped-advert", RouteTransportFlood, PayloadADVERT, `["AB12"]`),
}
packets, _ := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
want := map[string][]string{
"unscoped": {"flood", "unscoped"},
"scoped": {"flood", "scoped"},
"advert": {"flood", "unscoped", "advert"},
"scoped-advert": {"flood", "scoped", "advert"},
}
if len(packets) != len(want) {
t.Fatalf("got %d packets, want %d", len(packets), len(want))
}
for _, p := range packets {
if !reflect.DeepEqual(p.Tags, want[p.Hash]) {
t.Errorf("%s tags = %v, want %v", p.Hash, p.Tags, want[p.Hash])
}
}
}
func TestNodeHopPackets_DirectRoutesExcluded(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "direct", RouteDirect, PayloadTXT_MSG, `["77AA","AB12"]`),
hopTx(2, "tdirect", RouteTransportDirect, PayloadTXT_MSG, `["AB12"]`),
}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if len(packets) != 0 || ambiguous != 0 {
t.Fatalf("direct packets must be skipped: packets=%v ambiguous=%d", packets, ambiguous)
}
}
func TestNodeHopPackets_OneEventPerPacketAcrossObservations(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["77AA","AB12"]`, `["77AA","AB12","77AA"]`, `["77AA"]`),
}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if got := hopsOf(packets); !reflect.DeepEqual(got, []int{1}) {
t.Fatalf("hops = %v, want [1] (one event per packet hash)", got)
}
if ambiguous != 0 {
t.Errorf("ambiguous = %d, want 0", ambiguous)
}
}
func TestNodeHopPackets_NotInPathIsNeitherCountedNorAmbiguous(t *testing.T) {
txs := []*StoreTx{hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["77AA"]`, `[]`)}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if len(packets) != 0 || ambiguous != 0 {
t.Fatalf("packets=%v ambiguous=%d, want none", packets, ambiguous)
}
}
func TestNodeHopPackets_OriginatorNeverForwardsOwnFlood(t *testing.T) {
tx := hopTx(1, "own-advert", RouteFlood, PayloadADVERT, `["AB12"]`)
tx.DecodedJSON = `{"pubKey":"` + hopTarget + `"}`
packets, ambiguous := computeNodeHopPackets(hopTarget, []*StoreTx{tx}, "", hopPM(), hopGraph())
if len(packets) != 0 || ambiguous != 0 {
t.Fatalf("own advert: packets=%v ambiguous=%d, want none", packets, ambiguous)
}
}
// A colliding prefix counts for the node only under the ingestor's strict rule
// (cmd/ingestor/path_resolver.go resolvePathWithContext): the previous hop is
// itself identified without a tiebreak, and exactly one candidate is its
// graph neighbor. The server resolver's pick (affinity, GPS, advert count)
// plays no part.
func TestNodeHopPackets_CollisionNeedsStrictNeighborAttribution(t *testing.T) {
advert := func(id int, hash, origin string, path string) *StoreTx {
tx := hopTx(id, hash, RouteFlood, PayloadADVERT, path)
tx.DecodedJSON = `{"pubKey":"` + origin + `"}`
return tx
}
txs := []*StoreTx{
hopTx(1, "after-neighbor", RouteFlood, PayloadGRP_TXT, `["77","AB"]`),
hopTx(2, "no-previous-hop", RouteFlood, PayloadGRP_TXT, `["AB","77"]`),
hopTx(3, "after-collider-neighbor", RouteFlood, PayloadGRP_TXT, `["88","AB"]`),
hopTx(4, "after-common-neighbor", RouteFlood, PayloadGRP_TXT, `["55","AB"]`),
hopTx(5, "previous-hop-ambiguous", RouteFlood, PayloadGRP_TXT, `["99","AB"]`),
hopTx(6, "previous-hop-repeats", RouteFlood, PayloadGRP_TXT, `["77","88","77","AB"]`),
advert(7, "advert-from-neighbor", hopOther, `["AB"]`),
advert(8, "advert-from-stranger", hopBridge, `["AB"]`),
hopTx(9, "observations-disagree", RouteFlood, PayloadGRP_TXT, `["77","AB"]`, `["88","AB"]`),
hopTx(10, "one-observation-unresolved", RouteFlood, PayloadGRP_TXT, `["77","AB"]`, `["99","AB"]`),
hopTx(11, "observations-without-node", RouteFlood, PayloadGRP_TXT, `["77","88"]`, `["77"]`, `["77","AB"]`),
hopTx(12, "chain-broken", RouteFlood, PayloadGRP_TXT, `["77","99","AB"]`),
hopTx(13, "origin-of-non-advert", RouteFlood, PayloadTXT_MSG, `["AB"]`),
advert(14, "originator-excluded-later", hopTwinA, `["55","99","AB"]`),
}
txs[12].DecodedJSON = `{"pubKey":"` + hopOther + `"}`
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), hopGraph())
got := map[string]int{}
for _, p := range packets {
got[p.Hash] = p.Hops
}
want := map[string]int{"after-neighbor": 1, "advert-from-neighbor": 0, "one-observation-unresolved": 1,
"observations-without-node": 1, "originator-excluded-later": 2}
if !reflect.DeepEqual(got, want) {
t.Fatalf("attributed = %v, want %v", got, want)
}
if ambiguous != 9 {
t.Errorf("ambiguous = %d, want 9", ambiguous)
}
}
func TestNodeHopPackets_CollisionWithoutGraphIsAmbiguous(t *testing.T) {
txs := []*StoreTx{hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["77","AB"]`)}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if len(packets) != 0 || ambiguous != 1 {
t.Fatalf("packets=%v ambiguous=%d, want none and 1", packets, ambiguous)
}
}
func TestNodeHopPackets_PrefixAtSeveralPositionsIsAmbiguous(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "same-path", RouteFlood, PayloadGRP_TXT, `["AB","77","AB"]`),
hopTx(2, "across-obs", RouteFlood, PayloadGRP_TXT, `["AB"]`, `["77","AB"]`),
}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), hopGraph())
if len(packets) != 0 {
t.Fatalf("packets = %+v, want none: a node forwards a flood once, so two positions mean a collision", packets)
}
if ambiguous != 2 {
t.Errorf("ambiguous = %d, want 2", ambiguous)
}
}
func TestNodeHopPackets_ListenerDoesNotMakePrefixAmbiguous(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: hopTarget, Role: "repeater"},
{PublicKey: hopListener, Role: "repeater"},
})
pm.markNonRelay([]string{hopListener})
txs := []*StoreTx{hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["AB12","77AA"]`)}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", pm, nil)
if got := hopsOf(packets); !reflect.DeepEqual(got, []int{0}) || ambiguous != 0 {
t.Fatalf("hops=%v ambiguous=%d, want [0] and 0", got, ambiguous)
}
}
// When the node itself is a listener, the only relay candidate for its prefix
// is another node: nothing may be attributed to it.
func TestNodeHopPackets_OnlyRelayCandidateIsAnotherNode(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: hopTarget, Role: "repeater"},
{PublicKey: hopCollider, Role: "repeater"},
})
pm.markNonRelay([]string{hopTarget})
txs := []*StoreTx{hopTx(1, "h1", RouteFlood, PayloadGRP_TXT, `["AB"]`)}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", pm, nil)
if len(packets) != 0 || ambiguous != 1 {
t.Fatalf("packets=%v ambiguous=%d, want none and 1", packets, ambiguous)
}
}
// Uniqueness is per prefix: "ab" collides with hopCollider, "ab12" does not.
func TestNodeHopPackets_UniquenessIsPerPrefixLength(t *testing.T) {
txs := []*StoreTx{
hopTx(1, "two-byte", RouteFlood, PayloadGRP_TXT, `["AB12"]`),
hopTx(2, "one-byte", RouteFlood, PayloadGRP_TXT, `["AB"]`),
hopTx(3, "two-byte-again", RouteFlood, PayloadGRP_TXT, `["AB12"]`),
hopTx(4, "one-byte-again", RouteFlood, PayloadGRP_TXT, `["AB"]`),
}
packets, ambiguous := computeNodeHopPackets(hopTarget, txs, "", hopPM(), nil)
if got, want := hashesOf(packets), []string{"two-byte", "two-byte-again"}; !reflect.DeepEqual(got, want) {
t.Fatalf("attributed = %v, want %v", got, want)
}
if ambiguous != 2 {
t.Errorf("ambiguous = %d, want 2", ambiguous)
}
}
// ─── Store level: live ingest and cold load ────────────────────────────────
func hopInsertNode(t *testing.T, db *DB, pk, name string, adverts int) {
t.Helper()
mustExec(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES (?, ?, 'repeater', ?, '2026-01-01', ?)`, pk, name, time.Now().UTC().Format(time.RFC3339), adverts)
}
func hopInsertTx(t *testing.T, db *DB, id int, hash string, routeType, payloadType int, paths ...string) {
t.Helper()
ts := time.Now().Add(-1 * time.Hour)
mustExec(t, db, `INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type) VALUES (?, 'AA', ?, ?, ?, ?)`,
id, hash, ts.UTC().Format(time.RFC3339), routeType, payloadType)
for _, p := range paths {
mustExec(t, db, `INSERT INTO observations (transmission_id, observer_idx, path_json, timestamp, resolved_path) VALUES (?, NULL, ?, ?, NULL)`,
id, p, ts.Unix())
}
}
func hopLoadedStore(t *testing.T, db *DB) *PacketStore {
t.Helper()
store := NewPacketStore(db, nil)
store.graph.Store(loadNeighborEdgesFromDB(db.conn))
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
if !store.WaitIndexesReady(10 * time.Second) {
t.Fatal("indexes not ready")
}
return store
}
func hopAnalytics(t *testing.T, store *PacketStore, pk string) *NodeHopAnalyticsResponse {
t.Helper()
resp, err := store.GetNodeHopAnalytics(pk, 7)
if err != nil || resp == nil {
t.Fatalf("GetNodeHopAnalytics(%s): resp=%v err=%v", pk, resp, err)
}
sort.Slice(resp.Packets, func(i, j int) bool { return resp.Packets[i].Hash < resp.Packets[j].Hash })
return resp
}
// Review repro: live ingest resolves a colliding hop with a tiebreak (here the
// advert count). That pick must not become a hop count.
func TestNodeHopAnalytics_LiveIngestDoesNotTrustResolverPick(t *testing.T) {
db := setupTestDB(t)
hopInsertNode(t, db, hopTarget, "Target", 500)
hopInsertNode(t, db, hopCollider, "Collider", 1)
store := hopLoadedStore(t, db)
hopInsertTx(t, db, 1, "collides", RouteFlood, PayloadGRP_TXT, `["AB"]`)
if _, maxID := store.IngestNewFromDB(0, 100); maxID != 1 {
t.Fatalf("IngestNewFromDB maxID = %d, want 1", maxID)
}
for _, pk := range []string{hopTarget, hopCollider} {
resp := hopAnalytics(t, store, pk)
if len(resp.Packets) != 0 || resp.Ambiguous != 1 {
t.Errorf("%s: packets=%+v ambiguous=%d, want none and 1", pk, resp.Packets, resp.Ambiguous)
}
}
}
// Live ingest and a cold load of the same DB must give the same answer: hops
// are read from every observation's raw path and attributed from the prefix
// map and neighbor graph, not from what each load path indexed.
func TestNodeHopAnalytics_SameResultAfterRestart(t *testing.T) {
db := setupTestDB(t)
mustExec(t, db, `CREATE TABLE neighbor_edges (node_a TEXT NOT NULL, node_b TEXT NOT NULL,
count INTEGER DEFAULT 1, last_seen TEXT, PRIMARY KEY (node_a, node_b))`)
mustExec(t, db, `INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, 50, ?)`,
hopOther, hopTarget, time.Now().UTC().Format(time.RFC3339))
hopInsertNode(t, db, hopTarget, "Target", 1)
hopInsertNode(t, db, hopCollider, "Collider", 500)
hopInsertNode(t, db, hopOther, "Other", 1)
live := hopLoadedStore(t, db)
hopInsertTx(t, db, 1, "after-neighbor", RouteFlood, PayloadGRP_TXT, `["77","AB"]`, `["77","AB","99"]`)
hopInsertTx(t, db, 2, "no-previous-hop", RouteFlood, PayloadGRP_TXT, `["AB"]`)
hopInsertTx(t, db, 3, "two-byte", RouteTransportFlood, PayloadGRP_TXT, `["77AA","AB12"]`)
hopInsertTx(t, db, 4, "three-byte", RouteFlood, PayloadGRP_TXT, `["77AA88","77AA88","AB12CD"]`)
// The longest observation runs through another branch of the flood.
hopInsertTx(t, db, 5, "not-on-longest-path", RouteFlood, PayloadGRP_TXT, `["77AA","77AA","77AA"]`, `["77AA","AB12"]`)
live.IngestNewFromDB(0, 100)
fromLive := hopAnalytics(t, live, hopTarget)
restarted := hopAnalytics(t, hopLoadedStore(t, db), hopTarget)
want := []NodeHopPacket{
{Hash: "after-neighbor", Hops: 1, Tags: hopTagsUnscoped},
{Hash: "not-on-longest-path", Hops: 1, Tags: hopTagsUnscoped},
{Hash: "three-byte", Hops: 2, Tags: hopTagsUnscoped},
{Hash: "two-byte", Hops: 1, Tags: hopTagsScoped},
}
for name, resp := range map[string]*NodeHopAnalyticsResponse{"live ingest": fromLive, "cold load": restarted} {
got := make([]NodeHopPacket, len(resp.Packets))
for i, p := range resp.Packets {
got[i] = NodeHopPacket{Hash: p.Hash, Hops: p.Hops, Tags: p.Tags}
}
if !reflect.DeepEqual(got, want) || resp.Ambiguous != 1 {
t.Errorf("%s: packets=%+v ambiguous=%d, want %+v and 1", name, got, resp.Ambiguous, want)
}
}
if !reflect.DeepEqual(fromLive.Packets, restarted.Packets) || fromLive.Ambiguous != restarted.Ambiguous {
t.Errorf("live ingest %+v/%d differs from cold load %+v/%d",
fromLive.Packets, fromLive.Ambiguous, restarted.Packets, restarted.Ambiguous)
}
}
// End to end through the route: days window, response shape, 404.
func TestHandleNodeHopAnalytics(t *testing.T) {
db := setupTestDB(t)
recent := time.Now().Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := time.Now().Add(-1 * time.Hour).Unix()
old := time.Now().Add(-3 * 24 * time.Hour).Format(time.RFC3339)
oldEpoch := time.Now().Add(-3 * 24 * time.Hour).Unix()
hopInsertNode(t, db, hopTarget, "Target", 1)
hopInsertNode(t, db, hopOther, "Other", 1)
hopInsertNode(t, db, hopCollider, "Collider", 1)
mustExec(t, db, `INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type) VALUES (1, 'AA', 'hop_recent_2', ?, 1, 5)`, recent)
mustExec(t, db, `INSERT INTO observations (transmission_id, observer_idx, path_json, timestamp) VALUES (1, NULL, '["77AA","77AA","AB12"]', ?)`, recentEpoch)
mustExec(t, db, `INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type) VALUES (2, 'BB', 'hop_recent_1byte', ?, 0, 4)`, recent)
mustExec(t, db, `INSERT INTO observations (transmission_id, observer_idx, path_json, timestamp) VALUES (2, NULL, '["AB"]', ?)`, recentEpoch)
mustExec(t, db, `INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type) VALUES (3, 'CC', 'hop_scoped_advert', ?, 0, 4)`, recent)
mustExec(t, db, `INSERT INTO observations (transmission_id, observer_idx, path_json, timestamp) VALUES (3, NULL, '["AB12"]', ?)`, recentEpoch)
mustExec(t, db, `INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type) VALUES (4, 'DD', 'hop_old', ?, 1, 5)`, old)
mustExec(t, db, `INSERT INTO observations (transmission_id, observer_idx, path_json, timestamp) VALUES (4, NULL, '["AB12"]', ?)`, oldEpoch)
srv := NewServer(db, &Config{Port: 3000}, NewHub())
router := mux.NewRouter()
srv.RegisterRoutes(router)
get := func(url string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
router.ServeHTTP(w, httptest.NewRequest("GET", url, nil))
return w
}
srv.store = hopLoadedStore(t, db)
w := get("/api/nodes/" + hopTarget + "/hop_analytics?days=1")
if w.Code != http.StatusOK {
t.Fatalf("code=%d body=%s", w.Code, w.Body.String())
}
var resp NodeHopAnalyticsResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("unmarshal: %v", err)
}
byHash := map[string]NodeHopPacket{}
for _, p := range resp.Packets {
byHash[p.Hash] = p
}
if len(resp.Packets) != 2 {
t.Fatalf("packets = %+v, want the two attributable packets inside the 1-day window", resp.Packets)
}
if p := byHash["hop_recent_2"]; p.Hops != 2 || p.Timestamp != recent || !reflect.DeepEqual(p.Tags, []string{"flood", "unscoped"}) {
t.Errorf("hop_recent_2 = %+v, want hops 2, timestamp %s, tags [flood unscoped]", p, recent)
}
if p := byHash["hop_scoped_advert"]; p.Hops != 0 || !reflect.DeepEqual(p.Tags, []string{"flood", "scoped", "advert"}) {
t.Errorf("hop_scoped_advert = %+v, want hops 0 tags [flood scoped advert]", p)
}
if resp.TimeRange.Days != 1 || resp.Ambiguous != 1 {
t.Errorf("timeRange.days=%d ambiguous=%d, want 1 and 1 (colliding 1-byte prefix, no neighbor graph)", resp.TimeRange.Days, resp.Ambiguous)
}
if w := get("/api/nodes/" + hopTarget + "/hop_analytics?days=7"); w.Code != http.StatusOK || !json.Valid(w.Body.Bytes()) {
t.Fatalf("days=7: code=%d", w.Code)
} else {
var wide NodeHopAnalyticsResponse
_ = json.Unmarshal(w.Body.Bytes(), &wide)
if len(wide.Packets) != 3 {
t.Errorf("days=7 packets = %d, want 3", len(wide.Packets))
}
}
if w := get("/api/nodes/ffffffffffffffff/hop_analytics"); w.Code != http.StatusNotFound {
t.Errorf("unknown node: code=%d, want 404", w.Code)
}
}
// BenchmarkNodeHopPackets sizes one request for a busy repeater over 7 days,
// as counted in a 1,669-node mesh DB on 2026-09-13: 73,782 flood packets
// with 1,430,280 observations (19 per packet). A third of the packets pass the
// node under a unique 2-byte prefix, a sixth under a 1-byte prefix shared with
// another relay (strict neighbor walk), the rest do not pass it; in each, half
// of the observations contain the node. Observation slices are shared between
// packets to keep the fixture small; the scan only reads them.
func BenchmarkNodeHopPackets(b *testing.B) {
obsOf := func(self, other string) []*StoreObs {
through := `["` + other + `","` + self + `","` + other + `","` + other + `","` + other + `","` + other + `"]`
elsewhere := `["` + other + `","` + other + `","` + other + `","` + other + `","` + other + `","` + other + `"]`
obs := make([]*StoreObs, 19)
for j := range obs {
obs[j] = &StoreObs{PathJSON: elsewhere}
if j%2 == 0 {
obs[j].PathJSON = through
}
}
return obs
}
unique, colliding, elsewhere := obsOf("AB12", "77AA"), obsOf("AB", "77"), obsOf("88CC", "77AA")
rt, pt := RouteFlood, PayloadGRP_TXT
txs := make([]*StoreTx, 73782)
for i := range txs {
txs[i] = &StoreTx{ID: i, Hash: "h", FirstSeen: hopTimestamp, RouteType: &rt, PayloadType: &pt, Observations: elsewhere}
switch i % 6 {
case 0, 1:
txs[i].Observations = unique
case 2:
txs[i].Observations = colliding
}
}
pm, graph := hopPM(), hopGraph()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
computeNodeHopPackets(hopTarget, txs, "", pm, graph)
}
}
+4
View File
@@ -87,6 +87,10 @@ func routeDescriptions() map[string]routeMeta {
"GET /api/nodes/{pubkey}/health": {Summary: "Get node health", Tag: "nodes"},
"GET /api/nodes/{pubkey}/paths": {Summary: "Get node routing paths", Tag: "nodes"},
"GET /api/nodes/{pubkey}/analytics": {Summary: "Get node analytics", Description: "Per-node packet counts, timing, and RF stats.", Tag: "nodes"},
"GET /api/nodes/{pubkey}/hop_analytics": {Summary: "Get node hop counts", Description: "One entry per flood packet the node forwarded in the window, with the hop count its flood.max check saw (the node's zero-based index in the path) and tags (flood, scoped or unscoped, advert). DIRECT packets are excluded. Every observation in the window is read. A colliding path prefix is attributed only when the previous hop's neighbor_edges neighbors leave this node as the one candidate; the server's resolved-path pick is not used. Packets carrying this node's prefix that cannot be attributed are counted in `ambiguous`.", Tag: "nodes",
QueryParams: []paramMeta{
{Name: "days", Description: "Lookback window in days (default 7, clamped 1-365)", Type: "integer"},
}},
"GET /api/nodes/{pubkey}/neighbors": {Summary: "Get node neighbors", Description: "Returns the queried node's first-hop neighbors with affinity scores and observation metadata (count, SNR, distance, observers). Ambiguous edges carry candidate pubkeys.", Tag: "nodes", Response: schemaRef("NodeNeighborsResponse")},
"GET /api/scope-audit": {Summary: "Network-wide scope audit", Description: "For every repeater that has answered a declared-regions request: the regions it declares, which of those it has NOT been observed forwarding in the window, which scopes it forwards without declaring, and whether it forwards unscoped floods while omitting the '*' wildcard. '*' is never listed as a region — it governs unscoped floods, not a scope. Repeaters never successfully asked are absent rather than shown as declaring nothing. Rows with missing regions sort first; a short window is weak evidence, since a quiet region simply has no traffic.", Tag: "analytics",
+25 -7
View File
@@ -296,6 +296,7 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/nodes/{pubkey}/health", s.handleNodeHealth).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/paths", s.handleNodePaths).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/analytics", s.handleNodeAnalytics).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/hop_analytics", s.handleNodeHopAnalytics).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/battery", s.handleNodeBattery).Methods("GET")
r.HandleFunc("/api/nodes/clock-skew", s.handleFleetClockSkew).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/clock-skew", s.handleNodeClockSkew).Methods("GET")
@@ -2080,13 +2081,7 @@ func (s *Server) handleNodeAnalytics(w http.ResponseWriter, r *http.Request) {
writeError(w, 404, "Not found")
return
}
days := queryInt(r, "days", 7)
if days < 1 {
days = 1
}
if days > 365 {
days = 365
}
days := nodeAnalyticsDays(r)
if s.store != nil {
result, err := s.store.GetNodeAnalytics(pubkey, days)
@@ -2101,6 +2096,29 @@ func (s *Server) handleNodeAnalytics(w http.ResponseWriter, r *http.Request) {
writeError(w, 404, "Not found")
}
// nodeAnalyticsDays reads the node analytics range picker's ?days= (default 7,
// clamped to 1-365).
func nodeAnalyticsDays(r *http.Request) int {
return min(max(queryInt(r, "days", 7), 1), 365)
}
// handleNodeHopAnalytics serves the hop count at this node for each flood
// packet it forwarded (issue #1812). Separate from /analytics so the hop
// scan does not slow down the main analytics response.
func (s *Server) handleNodeHopAnalytics(w http.ResponseWriter, r *http.Request) {
pubkey := mux.Vars(r)["pubkey"]
if s.cfg.IsBlacklisted(pubkey) || s.isPubkeyHidden(pubkey) || s.store == nil {
writeError(w, 404, "Not found")
return
}
result, err := s.store.GetNodeHopAnalytics(pubkey, nodeAnalyticsDays(r))
if err != nil || result == nil {
writeError(w, 404, "Not found")
return
}
writeJSON(w, result)
}
func (s *Server) handleNodeClockSkew(w http.ResponseWriter, r *http.Request) {
pubkey := mux.Vars(r)["pubkey"]
if s.store == nil {
+42 -31
View File
@@ -6798,6 +6798,31 @@ func (s *PacketStore) InvalidateNodeCache() {
s.cacheMu.Unlock()
}
// relayCandidates returns the nodes whose pubkey starts with hop and that may
// appear as a path hop.
//
// Issue #1290: observer-known listener-only nodes are dropped from the
// candidate set. By firmware contract a node that advertises `repeat:off` in
// its MQTT /status will never relay a packet, so it cannot legitimately be a
// hop in someone else's path. Filtering shrinks ambiguous candidate sets
// without affecting any upstream caller (only no_match becomes more likely
// when the only matching prefix belonged to a listener). Empty pm.nonRelay
// preserves the pre-#1290 behavior exactly (back-compat).
func (pm *prefixMap) relayCandidates(hop string) []nodeInfo {
candidates := pm.m[strings.ToLower(hop)]
if len(pm.nonRelay) == 0 || len(candidates) == 0 {
return candidates
}
filtered := candidates[:0:0]
for i := range candidates {
if _, isListener := pm.nonRelay[strings.ToLower(candidates[i].PublicKey)]; isListener {
continue
}
filtered = append(filtered, candidates[i])
}
return filtered
}
func (pm *prefixMap) resolve(hop string) *nodeInfo {
h := strings.ToLower(hop)
candidates := pm.m[h]
@@ -6840,27 +6865,7 @@ func (pm *prefixMap) resolve(hop string) *nodeInfo {
// (e.g., the originator, observer, or adjacent hops in the path).
// graph may be nil, in which case tier-1 is skipped.
func (pm *prefixMap) resolveWithContext(hop string, contextPubkeys []string, graph *NeighborGraph) (*nodeInfo, string, float64) {
h := strings.ToLower(hop)
candidates := pm.m[h]
// Issue #1290: drop observer-known listener-only nodes from the
// candidate set. By firmware contract a node that advertises
// `repeat:off` in its MQTT /status will never relay a packet, so it
// cannot legitimately be a hop in someone else's path. Filtering
// here shrinks ambiguous candidate sets without affecting any
// upstream caller (the returned shape and confidence labels are
// preserved; only no_match becomes more likely when the only
// matching prefix belonged to a listener). Empty pm.nonRelay
// preserves the pre-#1290 behavior exactly (back-compat).
if len(pm.nonRelay) > 0 && len(candidates) > 0 {
filtered := candidates[:0:0]
for i := range candidates {
if _, isListener := pm.nonRelay[strings.ToLower(candidates[i].PublicKey)]; isListener {
continue
}
filtered = append(filtered, candidates[i])
}
candidates = filtered
}
candidates := pm.relayCandidates(hop)
if len(candidates) == 0 {
return nil, "no_match", 0
}
@@ -9638,6 +9643,21 @@ func (s *PacketStore) GetNodeHealth(pubkey string) (map[string]interface{}, erro
}, nil
}
// nodeTxsSince returns the node's transmissions first seen after fromISO,
// from the byNode index (sender, recipient and resolved relay hops).
// Raw JSON text search is intentionally avoided: a GRP_TXT packet whose message
// text contains a node's pubkey is not a packet *for* that node.
// Must be called with s.mu held.
func (s *PacketStore) nodeTxsSince(pubkey, fromISO string) []*StoreTx {
var packets []*StoreTx
for _, p := range s.byNode[pubkey] {
if p.FirstSeen > fromISO {
packets = append(packets, p)
}
}
return packets
}
// GetNodeAnalytics computes analytics for a single node using in-memory byNode index.
func (s *PacketStore) GetNodeAnalytics(pubkey string, days int) (*NodeAnalyticsResponse, error) {
node, err := s.db.GetNodeByPubkey(pubkey)
@@ -9652,16 +9672,7 @@ func (s *PacketStore) GetNodeAnalytics(pubkey string, days int) (*NodeAnalyticsR
s.mu.RLock()
defer s.mu.RUnlock()
// Collect packets from byNode index (time-filtered).
// Raw JSON text search is intentionally avoided: a GRP_TXT packet whose message
// text contains a node's pubkey is not a packet *for* that node.
indexed := s.byNode[pubkey]
var packets []*StoreTx
for _, p := range indexed {
if p.FirstSeen > fromISO {
packets = append(packets, p)
}
}
packets := s.nodeTxsSince(pubkey, fromISO)
// Activity timeline (hourly buckets)
timelineBuckets := map[string]int{}
+15
View File
@@ -576,6 +576,21 @@ type NodeAnalyticsResponse struct {
ClockSkew *NodeClockSkew `json:"clockSkew,omitempty"`
}
// NodeHopPacket is one flood packet this node forwarded, with the hop count
// the node's flood.max check saw for it (issue #1812).
type NodeHopPacket struct {
Hash string `json:"hash"`
Timestamp string `json:"timestamp"`
Hops int `json:"hops"`
Tags []string `json:"tags"`
}
type NodeHopAnalyticsResponse struct {
TimeRange TimeRangeResp `json:"timeRange"`
Packets []NodeHopPacket `json:"packets"`
Ambiguous int `json:"ambiguous"`
}
// ─── Analytics — RF ────────────────────────────────────────────────────────────
type PayloadTypeSignal struct {
+66
View File
@@ -23,6 +23,7 @@
- [GET /api/nodes/:pubkey/health](#get-apinodespubkeyhealth)
- [GET /api/nodes/:pubkey/paths](#get-apinodespubkeypaths)
- [GET /api/nodes/:pubkey/analytics](#get-apinodespubkeyanalytics)
- [GET /api/nodes/:pubkey/hop_analytics](#get-apinodespubkeyhop_analytics)
- [GET /api/nodes/:pubkey/reach](#get-apinodespubkeyreach)
- [GET /api/packets](#get-apipackets)
- [GET /api/packets/timestamps](#get-apipacketstimestamps)
@@ -675,6 +676,71 @@ Per-node analytics over a time range.
---
## GET /api/nodes/:pubkey/hop_analytics
Hop count at this node for every flood packet it forwarded, to help choose
`flood.max`, `flood.max.unscoped` and `flood.max.advert`. A repeater checks
those limits against the number of hashes already in the path, then appends
its own hash, so the hop count is the node's zero-based index in the observed
path (firmware `src/helpers/RoutingPolicy.h`, `src/Mesh.cpp` `routeRecvPacket`).
This is not the `hopDistribution` of `/analytics`, which is the path length at
the observer.
- One entry per packet hash. Values are raw so the client can filter and bin.
- Only floods (route types 0 and 1). DIRECT packets carry the remaining route,
not a hop count, and are left out. Packets the node originated are left out.
- Every observation of every flood packet in the window is read, not only the
packet's longest path, so a relay on a shorter branch of the flood counts too.
- A packet is attributed when the node's path prefix sits at exactly one index
across its observations, and either no other relay-capable node shares that
prefix, or the hop resolves to the node under the ingestor's strict rule
(every earlier hop identified without a tiebreak, and exactly one candidate
is a `neighbor_edges` neighbor of the previous hop, or of the originator for
an advert) in at least one observation and to another node in none. The
server's resolved-path pick (affinity, GPS distance, advert count) is not
used, so the result is the same before and after a restart. Everything else
with the node's prefix is counted in `ambiguous` and left out; in practice
that is most packets with a colliding 1-byte path hash.
- Size: for a busy repeater on a 1,669-node mesh over 7 days (2026-09-13)
the response held 23,068 entries, 2.3 MB of JSON, 375 KB gzipped. `hash`
and `timestamp` are 61% of the raw and 91% of the gzipped bytes; they stay
so a client can join entries to packets and bin by time (issue #1812).
### Query Parameters
| Param | Type | Default | Description |
|--------|--------|---------|--------------------------|
| `days` | number | `7` | Lookback window (1-365) |
### Response `200`
```jsonc
{
"timeRange": { "from": string (ISO), "to": string (ISO), "days": number },
"packets": [
{
"hash": string,
"timestamp": string (ISO), // first seen
"hops": number, // 0 = heard straight from the originator
"tags": [string] // "flood", then "scoped" or "unscoped", then "advert" if an ADVERT
}
],
"ambiguous": number // prefix matched, hop position not attributable to this node
}
```
Filters that match the firmware limits: `flood.max` uses all entries,
`flood.max.unscoped` the entries tagged `unscoped`, `flood.max.advert` the
entries tagged `advert`.
### Response `404`
```json
{ "error": "Not found" }
```
---
## GET /api/nodes/:pubkey/reach
Per-node RF reach report (two-way link quality). Computes **directional** link counts from raw
+1
View File
@@ -225,6 +225,7 @@
<script src="mqtt-status-panel.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="observer-detail.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="compare.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-hop-analytics.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-analytics.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-reach-map.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-reach-coverage.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
+7
View File
@@ -124,6 +124,11 @@
<div class="analytics-chart-desc">How many repeater hops packets take — 0 means direct</div>
<canvas id="hopChart" role="img" aria-label="Hop distribution chart"></canvas>
</div>
<div class="analytics-chart-card full">
<h4>Hop Count at This Node</h4>
<div class="analytics-chart-desc">For each flood packet this node forwarded: how many hops it already had, the number the flood.max, flood.max.unscoped and flood.max.advert limits are checked against. Bars count packets per hop count; the box above spans the middle half of the packets with a line at the median, and the whiskers reach the furthest packet within 1.5 box widths of the box.</div>
<div id="hopCountSection"><div style="padding:20px;text-align:center;color:var(--text-muted);font-size:12px">Loading hop counts...</div></div>
</div>
<div class="analytics-chart-card full">
<h4>Battery Voltage <span id="batteryStatusBadge" style="font-size:11px;font-weight:normal;margin-left:8px"></span></h4>
<div class="analytics-chart-desc">Battery voltage over time from observer status reports — flat line means full, downward slope means draining</div>
@@ -166,6 +171,7 @@
buildHopChart(data);
buildHeatmap(data);
loadBatteryChart(pubkey, currentDays);
NodeHopAnalytics.load(document.getElementById('hopCountSection'), pubkey, currentDays);
}
function buildActivityChart(data) {
@@ -370,6 +376,7 @@
function destroy() {
destroyCharts();
NodeHopAnalytics.destroy();
currentPubkey = null;
}
+178
View File
@@ -0,0 +1,178 @@
/* === CoreScope: node-hop-analytics.js === */
'use strict';
// Hop count at this node (issue #1812): for every flood packet the node
// forwarded, the number of hops the packet already had when the node's
// flood.max check ran. Data: GET /api/nodes/{pubkey}/hop_analytics?days=N.
(function () {
// One filter per repeater CLI limit (firmware src/helpers/RoutingPolicy.h):
// flood.max applies to every flood, flood.max.unscoped to un-scoped floods,
// flood.max.advert to adverts.
const FILTERS = [
{ id: 'flood', label: 'flood.max', tag: null, noun: 'flood packets' },
{ id: 'flood_unscoped', label: 'flood.max.unscoped', tag: 'unscoped', noun: 'un-scoped flood packets' },
{ id: 'flood_adverts', label: 'flood.max.advert', tag: 'advert', noun: 'flood adverts' },
];
let chart = null;
function filterById(id) {
return FILTERS.find(f => f.id === id) || FILTERS[0];
}
function filterHops(packets, filterId) {
const tag = filterById(filterId).tag;
const out = [];
for (const p of packets || []) {
if (!tag || (p.tags && p.tags.indexOf(tag) >= 0)) out.push(Number(p.hops));
}
return out;
}
function hopHistogram(values) {
let max = -1;
for (const v of values) if (v > max) max = v;
const counts = new Array(max + 1).fill(0);
for (const v of values) counts[v]++;
return counts;
}
// Quartiles by linear interpolation between closest ranks, Tukey whiskers at
// 1.5 IQR. Hop counts are small integers, so sorting a copy stays cheap.
function hopBoxStats(values) {
if (!values.length) return null;
const s = values.slice().sort((a, b) => a - b);
const q = p => {
const h = (s.length - 1) * p;
const lo = Math.floor(h);
return lo + 1 < s.length ? s[lo] + (h - lo) * (s[lo + 1] - s[lo]) : s[lo];
};
const q1 = q(0.25), median = q(0.5), q3 = q(0.75);
const lowFence = q1 - 1.5 * (q3 - q1), highFence = q3 + 1.5 * (q3 - q1);
let whiskerLow = q1, whiskerHigh = q3, outliers = 0;
for (const v of s) {
if (v < lowFence || v > highFence) { outliers++; continue; }
if (v < whiskerLow) whiskerLow = v;
if (v > whiskerHigh) whiskerHigh = v;
}
return { n: s.length, min: s[0], q1, median, q3, max: s[s.length - 1], whiskerLow, whiskerHigh, outliers };
}
function plural(n, word) {
return n + ' ' + word + (n === 1 ? '' : 's');
}
function renderHopSection(data, filterId) {
const filter = filterById(filterId);
const values = filterHops(data.packets, filter.id);
const stats = hopBoxStats(values);
const ambiguous = Number(data.ambiguous) || 0;
const chips = FILTERS.map(f =>
`<button type="button" data-hop-filter="${f.id}" aria-pressed="${f.id === filter.id}"${f.id === filter.id ? ' class="active"' : ''}>${f.label}</button>`
).join('');
const summary = stats
? `${plural(stats.n, 'packet')} (${filter.noun}) · median ${stats.median} · middle half ${stats.q1} to ${stats.q3} · max ${stats.max} hops`
: '';
const body = stats
? `<canvas id="hopCountChart" role="img" aria-label="Histogram and box plot of hop counts at this node for ${filter.noun}"></canvas>`
: `<div style="padding:20px;text-align:center;color:var(--text-muted);font-size:12px">No forwarded ${filter.noun} attributed to this node in this window.</div>`;
const note = ambiguous
? `<div class="analytics-chart-desc">${plural(ambiguous, 'packet')} left out: the path prefix of this node is shared with another node there, so its hop position is unknown.</div>`
: '';
return `
<div class="analytics-time-range" role="group" aria-label="Hop limit to inspect">${chips}</div>
<div style="font-size:12px;margin-bottom:6px">${summary}</div>
${body}
${note}`;
}
function cssVar(name) {
return getComputedStyle(document.documentElement).getPropertyValue(name).trim();
}
// Box plot drawn in the chart's top padding, on the histogram's own x axis:
// the category scale puts hop count h at index h, so fractional quartiles are
// interpolated between neighbouring category centres.
function boxPlotPlugin(stats) {
return {
id: 'hopBoxPlot',
afterDatasetsDraw(c) {
const x = c.scales.x;
const px = v => {
const lo = Math.floor(v);
const a = x.getPixelForValue(lo);
return v === lo ? a : a + (v - lo) * (x.getPixelForValue(lo + 1) - a);
};
const ctx = c.ctx;
const mid = c.chartArea.top - 16;
ctx.save();
ctx.strokeStyle = cssVar('--text');
ctx.fillStyle = cssVar('--accent-bg');
ctx.lineWidth = 1.5;
ctx.beginPath();
ctx.moveTo(px(stats.whiskerLow), mid); ctx.lineTo(px(stats.q1), mid);
ctx.moveTo(px(stats.q3), mid); ctx.lineTo(px(stats.whiskerHigh), mid);
ctx.moveTo(px(stats.whiskerLow), mid - 5); ctx.lineTo(px(stats.whiskerLow), mid + 5);
ctx.moveTo(px(stats.whiskerHigh), mid - 5); ctx.lineTo(px(stats.whiskerHigh), mid + 5);
ctx.stroke();
ctx.fillRect(px(stats.q1), mid - 8, px(stats.q3) - px(stats.q1), 16);
ctx.strokeRect(px(stats.q1), mid - 8, px(stats.q3) - px(stats.q1), 16);
ctx.beginPath();
ctx.moveTo(px(stats.median), mid - 8); ctx.lineTo(px(stats.median), mid + 8);
ctx.stroke();
ctx.restore();
}
};
}
function drawChart(data, filterId) {
if (chart) { chart.destroy(); chart = null; }
const canvas = document.getElementById('hopCountChart');
if (!canvas) return;
const values = filterHops(data.packets, filterId);
const counts = hopHistogram(values);
chart = new Chart(canvas, {
type: 'bar',
data: {
labels: counts.map((_, h) => String(h)),
datasets: [{ label: 'Packets', data: counts, backgroundColor: cssVar('--accent'), borderWidth: 0 }]
},
options: {
responsive: true,
layout: { padding: { top: 32 } },
plugins: { legend: { display: false } },
scales: {
x: { title: { display: true, text: 'Hops already in the path when this node forwarded' } },
y: { beginAtZero: true, title: { display: true, text: 'Packets' } }
}
},
plugins: [boxPlotPlugin(hopBoxStats(values))]
});
}
function render(container, data, filterId) {
container.innerHTML = renderHopSection(data, filterId);
container.querySelectorAll('[data-hop-filter]').forEach(btn => {
btn.addEventListener('click', () => render(container, data, btn.dataset.hopFilter));
});
drawChart(data, filterId);
}
async function load(container, pubkey, days) {
destroy();
let data;
try {
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/hop_analytics?days=' + days, { ttl: CLIENT_TTL.nodeAnalytics });
} catch (e) {
container.innerHTML = '<div style="padding:20px;text-align:center;color:var(--text-muted);font-size:12px">Hop counts unavailable: ' + escapeHtml(e.message) + '</div>';
return;
}
if (!container.isConnected) return;
render(container, data, 'flood');
}
function destroy() {
if (chart) { chart.destroy(); chart = null; }
}
window.NodeHopAnalytics = { FILTERS, filterHops, hopHistogram, hopBoxStats, renderHopSection, load, destroy };
})();
+1
View File
@@ -140,6 +140,7 @@ node test-map-clustering.js
node test-mqtt-status-panel.js
node test-my-repeaters-dashboard.js
node test-naive-banner-tone.js
node test-node-hop-analytics.js
node test-node-reach-coverage-debounce.js
node test-node-reach-coverage.js
node test-nodes-export-wiring.js
+84
View File
@@ -0,0 +1,84 @@
'use strict';
// Issue #1812: hop count at this node, node analytics page. Loads the browser
// IIFE in a vm sandbox (pattern from test-node-reach-coverage.js) and exercises
// the filter, histogram, box statistics and section render.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const code = fs.readFileSync(path.join(__dirname, 'public', 'node-hop-analytics.js'), 'utf8');
const sandbox = { window: {}, console };
vm.createContext(sandbox);
vm.runInContext(code, sandbox);
const H = sandbox.window.NodeHopAnalytics;
let passed = 0;
function test(name, fn) {
fn();
passed++;
console.log(' ok ' + name);
}
const packets = [
{ hash: 'a', hops: 0, tags: ['flood', 'unscoped'] },
{ hash: 'b', hops: 2, tags: ['flood', 'scoped'] },
{ hash: 'c', hops: 5, tags: ['flood', 'unscoped', 'advert'] },
{ hash: 'd', hops: 1, tags: ['flood', 'scoped', 'advert'] },
];
test('filters follow the firmware limits: flood.max all, .unscoped and .advert by tag', () => {
assert.deepStrictEqual(Array.from(H.filterHops(packets, 'flood')), [0, 2, 5, 1]);
assert.deepStrictEqual(Array.from(H.filterHops(packets, 'flood_unscoped')), [0, 5]);
assert.deepStrictEqual(Array.from(H.filterHops(packets, 'flood_adverts')), [5, 1]);
assert.deepStrictEqual(Array.from(H.filterHops(packets, 'bogus')), [0, 2, 5, 1], 'unknown filter falls back to flood');
});
test('histogram has one bucket per hop count from 0 to max', () => {
assert.deepStrictEqual(Array.from(H.hopHistogram([0, 1, 1, 3])), [1, 2, 0, 1]);
assert.deepStrictEqual(Array.from(H.hopHistogram([])), []);
});
test('box stats: interpolated quartiles, 1.5 IQR whiskers, outliers counted', () => {
// 8 lies between the 1.5 IQR fence (6) and a 3 IQR fence (9), so the
// fence factor decides whether it is an outlier.
const s = H.hopBoxStats([8, 0, 1, 3, 1]);
assert.strictEqual(s.n, 5);
assert.strictEqual(s.min, 0);
assert.strictEqual(s.q1, 1);
assert.strictEqual(s.median, 1);
assert.strictEqual(s.q3, 3);
assert.strictEqual(s.max, 8);
assert.strictEqual(s.whiskerLow, 0);
assert.strictEqual(s.whiskerHigh, 3, 'upper fence is q3 + 1.5*IQR = 6, so the whisker stops at 3');
assert.strictEqual(s.outliers, 1);
const even = H.hopBoxStats([4, 1, 3, 2]);
assert.strictEqual(even.q1, 1.75);
assert.strictEqual(even.median, 2.5);
assert.strictEqual(even.q3, 3.25);
assert.strictEqual(H.hopBoxStats([]), null);
});
test('section render: firmware-named chips, active filter, summary and ambiguous note', () => {
const html = H.renderHopSection({ packets, ambiguous: 2 }, 'flood_adverts');
assert.ok(html.includes('>flood.max<'), 'flood.max chip');
assert.ok(html.includes('>flood.max.unscoped<'), 'flood.max.unscoped chip');
assert.ok(html.includes('>flood.max.advert<'), 'flood.max.advert chip');
assert.ok(/data-hop-filter="flood_adverts"[^>]*aria-pressed="true"/.test(html), 'selected chip is pressed');
assert.ok(/data-hop-filter="flood"[^>]*aria-pressed="false"/.test(html), 'other chips are not pressed');
assert.ok(html.includes('2 packets'), 'packet count for the advert filter');
assert.ok(html.includes('median 3'), 'median of [5, 1]');
assert.ok(html.includes('max 5'), 'max of [5, 1]');
assert.ok(html.includes('id="hopCountChart"'), 'chart canvas');
assert.ok(html.includes('2 packets left out'), 'ambiguous packets are reported');
});
test('section render: empty filter shows a message instead of a chart', () => {
const html = H.renderHopSection({ packets: [packets[1]], ambiguous: 0 }, 'flood_unscoped');
assert.ok(!html.includes('id="hopCountChart"'), 'no canvas without data');
assert.ok(html.includes('No forwarded'), 'empty message');
assert.ok(!html.includes('left out'), 'no ambiguous note when zero');
});
console.log('node-hop-analytics: ' + passed + ' tests passed');