Files
meshcore-analyzer/cmd/server/notify_source_test.go
T
efitenandClaude Opus 5.5 a08f2f4e7c fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold
follows in a separate PR).

## Problem

A repeater silent since 2026-10-18 showed **Active — Last relayed 4m
ago** and **Last Heard 36m**, and its offline mail never fired. Every
hit after that date was the 1-byte hop `67`, shared by nine nodes:

- the live resolver's affinity tiebreak (`resolvePathForObs`) put those
transmissions in `byNode`, which fed `stats.lastHeard` and
`LastHeardMap`;
- `last_relayed` folds in the 1-byte prefix bucket on purpose (#662),
and drove the status in `roles.js` and the notify offline check.

## Fix

The store keeps `certainHeard` per node: the newest transmission that
can only be that node.

| Evidence | Counts? |
|---|---|
| Own packet (`pubKey`, `srcPubKey`) | ✅ |
| Flood relay hop whose prefix, at the packet's path hash size, matches
exactly one path-capable node | ✅ |
| 1-byte hop shared with other repeaters | ❌ |
| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌
|
| TRACE (path bytes are SNR) | ❌ |
| Destination (`destPubKey`) | ❌ |

A multibyte repeater legitimately appears as a 1-byte hop (the
originator picks the hash size, `Mesh::routeRecvPacket` appends at that
size), so only uniqueness decides. The rule is applied in every load
path through `indexResolvedPathHops` and the new
`indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live
ingest, late observations.

Liveness reads only that:

- `/api/nodes` and the node detail: `last_heard` is the newer of
`last_seen` and the certain last heard. Before, it was a plain copy of
`last_seen`.
- health / bulk-health `stats.lastHeard`.
- `LastHeardMap` → notify offline check. The separate relay input
(`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed.
- `roles.js getEffectiveHeardMs`: newest known timestamp, without
`last_relayed`. The side pane no longer says "Last relayed"; the Last
Relayed row stays (analytics) with an updated tooltip.
- New status **`uncertain`** (repeaters and rooms): nothing certain
within the silent window, but the shared prefix (`last_relayed`) still
in paths. Grey badge with explanation, an *Uncertain* filter on the
nodes list and the map, dimmed like stale. The offline mail for such a
node says only its shared prefix was still seen.
- A prefix also held by a path-capable node in `inactive_nodes` is not
unique; adverts with `signatureValid: false` are no evidence;
`/api/nodes/search` reports the same `last_heard`.

Attribution for analytics (`byNode`, packet counts, `last_relayed`,
relay counters, usefulness) is unchanged.

## Performance

Hot path: `noteCertainRelays` runs per observation on ingest under the
write lock: two hex bytes of header parsing plus one prefix-map lookup
per resolved hop.

```
BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)
```

Memory: one string per node ever certainly heard, dropped together with
the node's `byNode` list on eviction.

## Tests

- `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk,
LoadChunked — each with and without resolved_path — live ingest, late
observations) feeds `certainHeard`; each of the 8 call sites verified
red with a nil prefix map.
- `certain_heard_test.go`: hop width per route/payload type; retired
namesake prefix; failed signature; uncertain offline mail; shared vs
unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load
path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`,
detail and health agree.
- Regression for #2260: last certain packet on day X, 20 shared-prefix
hops after it → `LastHeardMap` stays at X and `node.offline` goes bad
after 72 h. Verified red with the uniqueness check mutated out.
- Notify: `TestEvaluateCertainHeard` replaces the relay-input test.
- Frontend: `last_relayed` alone no longer keeps a repeater or room
active; newest timestamp wins (an old `_liveSeen` can't hide a fresh
certain relay).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork with 457 extra commits.

## Validation

Ran the server on the freshened e2e fixture: list, detail and health
report the same last heard; e.g. a repeater whose `last_relayed` is 3
min newer than anything certain keeps its certain time. **No browser
screenshot**: Playwright isn't installed on this machine; CI's E2E
covers the pages.

## Not changed

- The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a
shared hop only when exactly one candidate is adjacent in the neighbor
graph; it did not fire in this case. Left as is.
- The `lastHeard=` list filter still filters `last_seen` in SQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:11:30 +02:00

139 lines
4.9 KiB
Go

package main
import (
"fmt"
"testing"
"time"
)
func TestDBNotifyNodes(t *testing.T) {
db := setupTestDB(t)
for _, q := range []string{
fmt.Sprintf(`INSERT INTO nodes (public_key, name, role, last_seen, battery_mv, foreign_advert) VALUES ('%s', 'Alpha', 'repeater', '2026-10-07T11:00:00Z', 3712, 0)`, evPkA),
fmt.Sprintf(`INSERT INTO nodes (public_key, name, role, last_seen, foreign_advert) VALUES ('%s', NULL, 'companion', NULL, 0)`, evPkB),
fmt.Sprintf(`INSERT INTO nodes (public_key, name, role, last_seen, foreign_advert) VALUES ('%s', 'Foxtrot', 'repeater', '2026-10-07T10:00:00Z', 1)`, evPkF),
} {
if _, err := db.conn.Exec(q); err != nil {
t.Fatal(err)
}
}
got, err := db.NotifyNodes([]string{evPkA, evPkB, evPkG}, false)
if err != nil {
t.Fatal(err)
}
a, b := got[evPkA], got[evPkB]
if len(got) != 2 || a.Pubkey != evPkA || a.Name != "Alpha" || a.Role != "repeater" || a.LastSeen != "2026-10-07T11:00:00Z" ||
a.BatteryMv == nil || *a.BatteryMv != 3712 || a.Foreign {
t.Fatalf("NotifyNodes = %+v", got)
}
if b.Name != "" || b.LastSeen != "" || b.BatteryMv != nil {
t.Fatalf("NULL columns = %+v", b)
}
got, err = db.NotifyNodes([]string{evPkA}, true)
if err != nil || len(got) != 2 || !got[evPkF].Foreign || got[evPkF].Name != "Foxtrot" {
t.Fatalf("with foreign = %+v, %v", got, err)
}
if got, err := db.NotifyNodes(nil, false); err != nil || len(got) != 0 {
t.Fatalf("no keys = %+v, %v", got, err)
}
}
func TestDBNotifyNodesChunksLongLists(t *testing.T) {
db := setupTestDB(t)
keys := make([]string, 0, 1200)
for i := 0; i < 1200; i++ {
keys = append(keys, fmt.Sprintf("%064x", i))
}
for _, i := range []int{3, 600, 1199} {
if _, err := db.conn.Exec(`INSERT INTO nodes (public_key, role) VALUES (?, 'companion')`, keys[i]); err != nil {
t.Fatal(err)
}
}
got, err := db.NotifyNodes(keys, false)
if err != nil || len(got) != 3 {
t.Fatalf("1200 keys over %d-key chunks = %d nodes, %v; want 3", notifyNodeChunk, len(got), err)
}
}
func TestPacketStoreLastHeardMap(t *testing.T) {
s := &PacketStore{certainHeard: map[string]string{
evPkA: "2026-10-07T11:30:00.000Z",
evPkB: "not a time",
}}
got, lock := s.LastHeardMap([]string{evPkA, evPkB, evPkF})
if len(got) != 1 || !got[evPkA].Equal(time.Date(2026, 10, 7, 11, 30, 0, 0, time.UTC)) {
t.Fatalf("LastHeardMap = %v", got)
}
if lock < 0 || lock > time.Second {
t.Fatalf("lock time = %v", lock)
}
}
// 5,000 watched nodes, the spec's upper figure. Quote the result in the PR:
// this runs under the store's read lock (one map lookup per node, #2260).
func BenchmarkLastHeardMap(b *testing.B) {
s := &PacketStore{certainHeard: map[string]string{}}
keys := make([]string, 0, 5000)
for i := 0; i < 5000; i++ {
pk := fmt.Sprintf("%064x", i)
keys = append(keys, pk)
s.certainHeard[pk] = evNow.Add(-time.Duration(i) * time.Minute).Format(time.RFC3339)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.LastHeardMap(keys)
}
}
func TestNotifyObserversFrom(t *testing.T) {
name, seen := "Roof", "2026-10-07T11:00:00Z"
got := notifyObserversFrom([]Observer{{ID: "OBS1", Name: &name, LastSeen: &seen}, {ID: "OBS2"}})
if len(got) != 2 || got[0] != (notifyObserver{ID: "OBS1", Name: "Roof", LastSeen: seen}) || got[1] != (notifyObserver{ID: "OBS2"}) {
t.Fatalf("notifyObserversFrom = %+v", got)
}
}
func TestServerNotifySourceWithoutStore(t *testing.T) {
src := serverNotifySource{s: &Server{cfg: &Config{}}}
if heard, lock := src.lastHeard([]string{evPkA}); len(heard) != 0 || lock != 0 {
t.Fatal("a server without a packet store reported times")
}
if src.lowBatteryMv() != 3300 || src.health().InfraSilentHours != 72 {
t.Fatal("defaults not taken from the config")
}
}
// The startup load (hot window plus background fill) must be finished, not
// only /api/healthz readiness: readiness can be reached while the newest
// packets are still loading, which would mail "offline" then "back online".
func TestServerNotifySourceReadyWaitsForTheStartupLoad(t *testing.T) {
readiness.Store(1)
defer readiness.Store(0)
store := &PacketStore{clockSkew: &ClockSkewEngine{}}
src := serverNotifySource{s: &Server{cfg: &Config{}, store: store}}
if src.ready() {
t.Fatal("ready before the startup load finished")
}
store.signalStartupLoadDone()
if !src.ready() {
t.Fatal("not ready after readiness and the startup load")
}
readiness.Store(0)
if src.ready() {
t.Fatal("ready without readiness")
}
if (serverNotifySource{s: &Server{cfg: &Config{}}}).ready() {
t.Fatal("ready without a packet store")
}
}
func TestPacketStoreNewestFirstSeen(t *testing.T) {
if got := (&PacketStore{}).NewestFirstSeen(); !got.IsZero() {
t.Fatalf("empty store = %v", got)
}
s := &PacketStore{packets: []*StoreTx{{FirstSeen: "2026-10-07T10:00:00Z"}, {FirstSeen: "2026-10-07T11:30:00.000Z"}}}
if got := s.NewestFirstSeen(); !got.Equal(time.Date(2026, 10, 7, 11, 30, 0, 0, time.UTC)) {
t.Fatalf("NewestFirstSeen = %v", got)
}
}