mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 19:17:42 +00:00
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
123 lines
4.4 KiB
Go
123 lines
4.4 KiB
Go
package main
|
|
|
|
import (
|
|
"database/sql"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// Review (#2261): every load path must feed certainHeard. A nil prefix map
|
|
// at any of the call sites in Load, loadChunk, scanAndMergeChunk, live
|
|
// ingest or late observations would leave the unit tests green; these go
|
|
// through the real paths and assert LastHeardMap.
|
|
|
|
const lpRelay = "4f2a" + "5555555555555555555555555555555555555555555555555555555555555"
|
|
|
|
// lpFixture is one flood transmission whose single 2-byte hop is lpRelay,
|
|
// a registered repeater, observed a minute ago. withResolved keeps the
|
|
// observation's resolved_path (the ingestor resolved the hop); otherwise it
|
|
// is NULL and the server re-resolves from path_json. hopInFirstObs false
|
|
// leaves the first observation without a path (for the late-observation
|
|
// case).
|
|
func lpFixture(t *testing.T, withResolved, hopInFirstObs bool) (*DB, *sql.DB, string) {
|
|
t.Helper()
|
|
path := createTestDBWithResolvedPath(t, 1, []string{lpRelay})
|
|
w, err := sql.Open("sqlite3", path+"?_journal_mode=WAL")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
t.Cleanup(func() { w.Close() })
|
|
at := time.Now().UTC().Add(-time.Minute).Format(time.RFC3339)
|
|
raw := "1541" + lpRelay[:4] + "00" // GRP_TXT, flood, path byte: 2-byte hashes x1
|
|
pathJSON, resolved := `["`+strings.ToUpper(lpRelay[:4])+`"]`, interface{}(`["`+lpRelay+`"]`)
|
|
if !withResolved {
|
|
resolved = nil
|
|
}
|
|
if !hopInFirstObs {
|
|
pathJSON, resolved = `[]`, nil
|
|
}
|
|
for _, q := range []struct {
|
|
sql string
|
|
args []interface{}
|
|
}{
|
|
{`ALTER TABLE transmissions ADD COLUMN from_pubkey TEXT`, nil},
|
|
{`UPDATE transmissions SET raw_hex=?, route_type=1, payload_type=5, first_seen=?`, []interface{}{raw, at}},
|
|
{`UPDATE observations SET raw_hex=?, timestamp=?, path_json=?, resolved_path=?`, []interface{}{raw, at, pathJSON, resolved}},
|
|
{`INSERT INTO nodes (public_key, name, role) VALUES (?, 'Relay', 'repeater')`, []interface{}{lpRelay}},
|
|
} {
|
|
if _, err := w.Exec(q.sql, q.args...); err != nil {
|
|
t.Fatalf("%s: %v", q.sql, err)
|
|
}
|
|
}
|
|
db, err := OpenDB(path)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
t.Cleanup(func() { db.conn.Close() })
|
|
return db, w, at
|
|
}
|
|
|
|
func TestCertainHeardFedByEveryLoadPath(t *testing.T) {
|
|
type mode struct {
|
|
name string
|
|
withResolved, hopInFirstObs bool
|
|
run func(t *testing.T, s *PacketStore, db *DB, w *sql.DB, at string)
|
|
}
|
|
load := func(t *testing.T, s *PacketStore, _ *DB, _ *sql.DB, _ string) {
|
|
if err := s.Load(); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
chunk := func(t *testing.T, s *PacketStore, _ *DB, _ *sql.DB, _ string) {
|
|
if err := s.loadChunk(time.Now().Add(-time.Hour), time.Now()); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
chunked := func(t *testing.T, s *PacketStore, _ *DB, _ *sql.DB, _ string) {
|
|
if err := s.LoadChunked(1); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
live := func(t *testing.T, s *PacketStore, _ *DB, _ *sql.DB, _ string) {
|
|
s.IngestNewFromDB(0, 100)
|
|
}
|
|
late := func(t *testing.T, s *PacketStore, db *DB, w *sql.DB, at string) {
|
|
if err := s.Load(); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if heard, _ := s.LastHeardMap([]string{lpRelay}); len(heard) != 0 {
|
|
t.Fatal("setup: the first observation carries no hop")
|
|
}
|
|
maxObs := db.GetMaxObservationID()
|
|
if _, err := w.Exec(`INSERT INTO observations (id, transmission_id, observer_id, observer_name, direction, snr, rssi, score, path_json, timestamp, raw_hex, resolved_path)
|
|
VALUES (2, 1, 'obs2', 'Obs2', 'RX', -9, -79, 5, ?, ?, '', NULL)`, `["`+strings.ToUpper(lpRelay[:4])+`"]`, at); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
s.IngestNewObservations(maxObs, 100)
|
|
}
|
|
for _, m := range []mode{
|
|
{"Load/resolved_path", true, true, load},
|
|
{"Load/path_json", false, true, load},
|
|
{"loadChunk/resolved_path", true, true, chunk},
|
|
{"loadChunk/path_json", false, true, chunk},
|
|
{"LoadChunked/resolved_path", true, true, chunked},
|
|
{"LoadChunked/path_json", false, true, chunked},
|
|
{"IngestNewFromDB", false, true, live},
|
|
{"IngestNewObservations", false, false, late},
|
|
} {
|
|
t.Run(m.name, func(t *testing.T) {
|
|
db, w, at := lpFixture(t, m.withResolved, m.hopInFirstObs)
|
|
s := NewPacketStore(db, &PacketStoreConfig{})
|
|
s.useResolvedPathIndex = true
|
|
s.initResolvedPathIndex()
|
|
m.run(t, s, db, w, at)
|
|
heard, _ := s.LastHeardMap([]string{lpRelay})
|
|
want, _ := time.Parse(time.RFC3339, at)
|
|
if !heard[lpRelay].Equal(want) {
|
|
t.Fatalf("LastHeardMap[relay] = %v; want %v: this path does not feed certainHeard", heard[lpRelay], want)
|
|
}
|
|
})
|
|
}
|
|
}
|