mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-14 21:47:41 +00:00
Fixes #1854. Refs #1598, #1611, #1845.
## The bug
`cmd/server/db.go:54` opens SQLite `mode=ro` (#1283/#1289).
`touchRelayLastSeen` → `TouchNodeLastSeen` issues `UPDATE nodes SET
last_seen` on that handle. It has failed on every call since, with the
error discarded at the call site:
```go
if err := s.db.TouchNodeLastSeen(pk, ts); err == nil {
s.lastSeenTouched[pk] = now
}
```
`nodes.last_seen` has therefore tracked ADVERT arrivals only. Verified
on live.saarmesh.de (1388 nodes): 1362 have `last_seen` within one
minute of their own most recent ADVERT. Reproduced directly with the
server's DSN in #1854.
Secondary effect: `lastSeenTouched` is populated only in the success
branch, so the debounce never engaged — the server retried the failing
UPDATE for every resolved pubkey in every decode window.
## The fix
The writer moves to `cmd/ingestor`, which owns `nodes` per #1283/#1287
and since #1547 already resolves hop prefixes to full pubkeys for
`observations.resolved_path`. The touch hooks into that existing
resolution point, so there is no new IPC surface and no second resolver.
Only unambiguously resolved hops qualify — a 1-byte prefix collision
cannot keep a silent node alive.
I considered the `internal/mbcapqueue` snapshot handoff used for
#903/#1324 and did not need it: that pattern exists because the
capability computation lives in the server's analytics cycle. Path
resolution already happens in the ingestor, so a file handoff would add
a hop for nothing.
`Store.TouchRelayNodes`:
- monotonic guard in SQL (`last_seen IS NULL OR last_seen < ?`) —
out-of-order ingest never rewinds
- 5-minute debounce keyed on `rxTime`, matching the interval the server
intended
- UPDATE only — unknown pubkeys never create rows
- unparsable `rxTime` is a no-op rather than writing garbage into the
node directory
- `Stats.RelayTouches` for `/api/perf` visibility
- debounce records the *attempt*, not the row match, so an unknown
pubkey is not retried per observation
## Server-side removal
`touchRelayLastSeen`, `DB.TouchNodeLastSeen`, the `lastSeenTouched` map
and the now-unused `allResolvedPKs` decode-window map are deleted.
`readonly_invariant_test.go` gains `UPDATE\s+nodes\s+SET\s+last_seen`.
`cmd/server/touch_last_seen_test.go` and two tests in
`resolved_index_test.go` go with it. Worth stating why they were green
for months: they build their `PacketStore` on `setupTestDB`, which opens
read-write. The production constraint is the one thing they did not
reproduce, which is why the added invariant regex — not a replacement
unit test — is the right guard here.
## Tests
Five tests in `cmd/ingestor/relay_touch_test.go`, committed red first
(573bbde3) with a stubbed `TouchRelayNodes` so the suite compiles and
reds on assertions:
```
--- FAIL: TestTouchRelayNodes_AdvancesLastSeen
last_seen = "2026-07-01T00:00:00Z", want "2026-07-10T12:00:00Z"
RelayTouches = 0, want 1
--- FAIL: TestTouchRelayNodes_Debounces
RelayTouches = 0, want 1 (second touch should be debounced)
```
Coverage: `AdvancesLastSeen` (core regression), `NeverGoesBackwards`
(monotonic), `Debounces` (write amplification on the hot path),
`IgnoresEmptyAndUnknown` (unresolved hops must not create rows),
`MalformedTimestamp`.
`cmd/ingestor`: full suite green, 100.7s.
`cmd/server`: green for the invariant and the affected packages, but the
suite is order-dependent on master today. Unmodified `upstream/master`
produced 8 failures on this machine (`TestHandleNodePaths_*`,
`TestHandleAnalytics*`, `TestComputeAnalyticsDistanceLockHoldDuration`);
this branch produced 5, and the set shifts between runs. All pass in
isolation. Untouched by this change — flagging rather than papering
over, and happy to open a separate issue if that is not already known.
## Impact on the open threads
This is the backend half of #1598. The frontend work there keys on relay
recency; that signal was never being written, so the two changes are
complementary rather than alternatives. It also removes the eviction
problem I raised in #1845 without touching `MoveStaleNodes`: once
`last_seen` reflects relay activity, the existing `last_seen < cutoff`
predicate stops evicting nodes that are carrying traffic.
Not addressed here: the duplicate-row behaviour between `nodes` and
`inactive_nodes` (609 keys in both on my deployment), which is an
independent defect and wants its own change.
## Verification offer
I run a 1100-repeater MeshCore deployment and can run this against
production traffic and report `RelayTouches` plus the resulting
`last_seen` distribution before/after, if that is useful for review.
---------
Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
239 lines
8.4 KiB
Go
239 lines
8.4 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
var _ = time.Second // suppress unused import
|
|
|
|
// Helper to create a minimal PacketStore with GRP_TXT packets for channel analytics testing.
|
|
func newChannelTestStore(packets []*StoreTx) *PacketStore {
|
|
ps := &PacketStore{
|
|
packets: packets,
|
|
byHash: make(map[string]*StoreTx),
|
|
byTxID: make(map[int]*StoreTx),
|
|
byObsID: make(map[int]*StoreObs),
|
|
byObserver: make(map[string][]*StoreObs),
|
|
byNode: make(map[string][]*StoreTx),
|
|
byPathHop: make(map[string][]*StoreTx),
|
|
nodeHashes: make(map[string]map[string]bool),
|
|
byPayloadType: make(map[int][]*StoreTx),
|
|
rfCache: make(map[string]*cachedResult),
|
|
topoCache: make(map[string]*cachedResult),
|
|
hashCache: make(map[string]*cachedResult),
|
|
collisionCache: make(map[string]*cachedResult),
|
|
chanCache: make(map[string]*cachedResult),
|
|
distCache: make(map[string]*cachedResult),
|
|
subpathCache: make(map[string]*cachedResult),
|
|
spIndex: make(map[string]int),
|
|
spTxIndex: make(map[string][]*StoreTx),
|
|
advertPubkeys: make(map[string]int),
|
|
clockSkew: NewClockSkewEngine(),
|
|
}
|
|
ps.byPayloadType[5] = packets
|
|
return ps
|
|
}
|
|
|
|
func makeGrpTx(channelHash int, channel, text, sender string) *StoreTx {
|
|
return makeGrpTxWithStatus(channelHash, channel, text, sender, "")
|
|
}
|
|
|
|
// makeGrpTxWithStatus is like makeGrpTx but lets the caller set the ingestor's
|
|
// decryptionStatus ("decrypted" / "no_key" / "decryption_failed" / ""), which
|
|
// computeAnalyticsChannels consults to decide whether a channel name is
|
|
// trustworthy (see #1729).
|
|
func makeGrpTxWithStatus(channelHash int, channel, text, sender, decryptionStatus string) *StoreTx {
|
|
decoded := map[string]interface{}{
|
|
"type": "CHAN",
|
|
"channelHash": float64(channelHash),
|
|
"channel": channel,
|
|
"text": text,
|
|
"sender": sender,
|
|
}
|
|
if decryptionStatus != "" {
|
|
decoded["decryptionStatus"] = decryptionStatus
|
|
}
|
|
b, _ := json.Marshal(decoded)
|
|
pt := 5
|
|
return &StoreTx{
|
|
ID: 1,
|
|
DecodedJSON: string(b),
|
|
FirstSeen: "2026-05-01T12:00:00Z",
|
|
PayloadType: &pt,
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_MergesEncryptedAndDecrypted verifies that packets
|
|
// with the same hash byte but different decryption status merge into ONE bucket.
|
|
func TestComputeAnalyticsChannels_MergesEncryptedAndDecrypted(t *testing.T) {
|
|
// Hash 129 is the real hash for #wardriving: SHA256(SHA256("#wardriving")[:16])[0] = 129
|
|
// Some packets are decrypted (have channel name), some are not (encrypted)
|
|
packets := []*StoreTx{
|
|
makeGrpTx(129, "#wardriving", "hello", "alice"),
|
|
makeGrpTx(129, "#wardriving", "world", "bob"),
|
|
makeGrpTx(129, "", "", ""), // encrypted — no channel name
|
|
makeGrpTx(129, "", "", ""), // encrypted
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "#wardriving" {
|
|
t.Errorf("expected name '#wardriving', got %q", ch["name"])
|
|
}
|
|
if ch["messages"] != 4 {
|
|
t.Errorf("expected 4 messages, got %v", ch["messages"])
|
|
}
|
|
if ch["encrypted"] != false {
|
|
t.Errorf("expected encrypted=false (some packets decrypted), got %v", ch["encrypted"])
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_RejectsRainbowTableMismatch verifies that a packet
|
|
// with channelHash=72 but channel="#wardriving" (mismatch) does NOT create a
|
|
// "#wardriving" bucket — it falls into "ch72" instead.
|
|
func TestComputeAnalyticsChannels_RejectsRainbowTableMismatch(t *testing.T) {
|
|
// Hash 72 is NOT the correct hash for #wardriving (which is 129).
|
|
// This simulates a rainbow-table collision/mismatch.
|
|
packets := []*StoreTx{
|
|
makeGrpTx(72, "#wardriving", "ghost", "eve"), // mismatch: hash 72 != wardriving's real hash
|
|
makeGrpTx(129, "#wardriving", "real", "alice"), // correct match
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 2 {
|
|
t.Fatalf("expected 2 channel buckets, got %d: %+v", len(channels), channels)
|
|
}
|
|
|
|
// Find the buckets
|
|
var ch72, ch129 map[string]interface{}
|
|
for _, ch := range channels {
|
|
if ch["hash"] == "72" {
|
|
ch72 = ch
|
|
} else if ch["hash"] == "129" {
|
|
ch129 = ch
|
|
}
|
|
}
|
|
|
|
if ch72 == nil {
|
|
t.Fatal("expected a bucket for hash 72")
|
|
}
|
|
if ch129 == nil {
|
|
t.Fatal("expected a bucket for hash 129")
|
|
}
|
|
|
|
// ch72 should NOT be named "#wardriving" — it should be the placeholder
|
|
if ch72["name"] == "#wardriving" {
|
|
t.Errorf("hash 72 bucket should NOT be named '#wardriving' (rainbow-table mismatch rejected)")
|
|
}
|
|
if ch72["name"] != "ch72" {
|
|
t.Errorf("expected hash 72 bucket named 'ch72', got %q", ch72["name"])
|
|
}
|
|
|
|
// ch129 should be named "#wardriving"
|
|
if ch129["name"] != "#wardriving" {
|
|
t.Errorf("expected hash 129 bucket named '#wardriving', got %q", ch129["name"])
|
|
}
|
|
}
|
|
|
|
// TestChannelNameMatchesHash verifies the hash validation function.
|
|
func TestChannelNameMatchesHash(t *testing.T) {
|
|
// #wardriving hashes to 129
|
|
if !channelNameMatchesHash("#wardriving", "129") {
|
|
t.Error("expected #wardriving to match hash 129")
|
|
}
|
|
if channelNameMatchesHash("#wardriving", "72") {
|
|
t.Error("expected #wardriving to NOT match hash 72")
|
|
}
|
|
// Without leading # should also work
|
|
if !channelNameMatchesHash("wardriving", "129") {
|
|
t.Error("expected wardriving (without #) to match hash 129")
|
|
}
|
|
}
|
|
|
|
// TestIsPlaceholderName verifies placeholder detection.
|
|
func TestIsPlaceholderName(t *testing.T) {
|
|
if !isPlaceholderName("ch129") {
|
|
t.Error("ch129 should be placeholder")
|
|
}
|
|
if !isPlaceholderName("ch0") {
|
|
t.Error("ch0 should be placeholder")
|
|
}
|
|
if isPlaceholderName("#wardriving") {
|
|
t.Error("#wardriving should NOT be placeholder")
|
|
}
|
|
if isPlaceholderName("Public") {
|
|
t.Error("Public should NOT be placeholder")
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_PublicChannelPreserved is the regression test for
|
|
// #1729: the firmware-default "Public" channel (channel-hash byte 0x11 = 17,
|
|
// key-derived via SHA256(key)[0], NOT the hashtag scheme's 186) was being
|
|
// discarded by the #978 rainbow-table validation and rendered as
|
|
// "Encrypted (0x11)". The ingestor decrypts Public packets with the builtin
|
|
// well-known key and marks them decryptionStatus:"decrypted"; the server must
|
|
// trust that name instead of re-applying the hashtag hash check.
|
|
func TestComputeAnalyticsChannels_PublicChannelPreserved(t *testing.T) {
|
|
packets := []*StoreTx{
|
|
makeGrpTxWithStatus(17, "Public", "hello net", "alice", "decrypted"),
|
|
makeGrpTxWithStatus(17, "Public", "morning", "bob", "decrypted"),
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "Public" {
|
|
t.Errorf("expected name 'Public' preserved, got %q (must not be downgraded to ch17)", ch["name"])
|
|
}
|
|
if ch["encrypted"] != false {
|
|
t.Errorf("expected encrypted=false for decrypted Public channel, got %v", ch["encrypted"])
|
|
}
|
|
if ch["hash"] != "17" {
|
|
t.Errorf("expected hash '17', got %v", ch["hash"])
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_UndecryptedNameStillValidated ensures the #1729
|
|
// fix does not weaken #978: when the ingestor did NOT mark the packet
|
|
// "decrypted" (no_key / decryption_failed / absent), a channel name that fails
|
|
// the hashtag hash check is still discarded to "chNN", even if text/sender
|
|
// happen to be present (e.g. a stale/foreign rainbow-table hit).
|
|
func TestComputeAnalyticsChannels_UndecryptedNameStillValidated(t *testing.T) {
|
|
// Hash 17 is NOT the hashtag hash of #Public (that is 186). Without a
|
|
// "decrypted" status, the name must be rejected.
|
|
packets := []*StoreTx{
|
|
makeGrpTxWithStatus(17, "#Public", "leaked", "eve", ""),
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "ch17" {
|
|
t.Errorf("expected undecrypted mismatch downgraded to 'ch17', got %q", ch["name"])
|
|
}
|
|
if ch["encrypted"] != true {
|
|
t.Errorf("expected encrypted=true for rejected rainbow-table name, got %v", ch["encrypted"])
|
|
}
|
|
}
|