mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-12 15:37:39 +00:00
## Summary
The subpath index keeps every contiguous run of 2-8 hops from every
stored transmission's path, in two maps:
- `spIndex`: key → count
- `spTxIndex`: key → transmissions containing that subpath, one entry
per occurrence
Both were always updated together (the only production callers were
`addTxToSubpathIndexFull` / `removeTxFromSubpathIndexFull`, which touch
both), so every count equalled `len(spTxIndex[key])`. The existing
`TestSubpathTxIndexPopulated` asserted exactly that. The index therefore
held its 1M+ keys twice.
This PR removes `spIndex`. The two analytics consumers read `len(txs)`,
and the add/remove helpers take only the transaction map. No API
response changes.
No existing upstream issue covers this; it came out of profiling a
production instance where garbage collection was 87% of CPU (30 s CPU
profile: `gcDrain` 39.9 s of 45.9 s sampled) with an 805 MB live heap
against a 1 GiB `GOMEMLIMIT`. That heap profile put 186 MB in this
index; on that instance's 7-day window there are 1.2M distinct subpath
keys (79% seen once) and 3.07M entries.
## Performance
### Go benchmark
`BenchmarkBuildSubpathIndex` builds the index for a synthetic,
production-shaped window: 41k transmissions with path lengths drawn from
the production hop-count histogram, and hops from random walks over a
400-node graph with 3-byte prefixes. That gives 1.48M keys, 85% seen
once (production: 1.2M, 79%). No real data is used. 10 runs each:
```
cpu: AMD Ryzen 5 3600XT 6-Core Processor
│ master │ this PR │
│ sec/op │ sec/op vs base │
BuildSubpathIndex-12 2.157 ± 1% 1.610 ± 5% -25.33% (p=0.000 n=10)
│ retained-B │ retained-B vs base │
BuildSubpathIndex-12 238.5Mi ± 0% 187.7Mi ± 0% -21.31% (p=0.000 n=10)
│ B/op │ B/op vs base │
BuildSubpathIndex-12 572.0Mi ± 0% 465.6Mi ± 0% -18.61% (p=0.000 n=10)
```
(`retained-B` is the live heap the index keeps after a forced GC: 250.1
MB → 196.8 MB.)
### Staging heap
Live heap after a forced GC (`/debug/pprof/heap?gc=1`), on a staging
host running a copy of a production database and its live MQTT feed,
attributed to the subpath index builder:
| | master | this PR |
|---|---|---|
| subpath index (`buildSubpathIndex`, cum) | 172.4 MB | 116.1 MB |
| packets in memory at the time | 27,521 | 24,086 |
| per packet | 6.27 KB | 4.82 KB (**-23%**) |
The two snapshots were taken minutes apart with different numbers of
packets loaded, hence the per-packet figure; it agrees with the
benchmark's -21%.
The same A/B replay used for the other PRs (30 minutes of real
production API traffic) showed no latency change (p50 10.3 ms vs 10.7
ms, p95 59.5 ms vs 52.9 ms, 0 errors). GC count and CPU were lower in
this run (44 → 28 collections, 13.0% → 9.6% CPU), but runs on this
shared host vary by that much on their own, so no claim is made from
them.
## Tests
- Red commit `ded359ff`: the fixture, `TestSubpathFixtureShape` (keeps
the fixture production-like: 60-95% singleton keys),
`BenchmarkBuildSubpathIndex`, and `TestSubpathIndexRetainedHeap`, which
allows at most 225 MB for the 41k fixture. It fails on `master` with
250.1 MB.
- Green commit `e399c561`: the change. The heap test passes at about 197
MB.
- `make test`: 16 of 16 modules pass; `go test -race` over the subpath,
eviction, path-hop and resolved-index tests passes.
Test changes in the green commit are mechanical, as required for a
structural change:
- struct literals stop setting `spIndex`; where a literal did not set
`spTxIndex` it now does, because appending to a nil map panics where the
old code skipped the transaction map when passed `nil`;
- count assertions read `len(spTxIndex[key])`;
- the old "spIndex and spTxIndex agree" check becomes "no key is kept
with an empty list", which is what keeps `len()` a correct count.
## Review notes
- Removal still takes out one occurrence per subpath occurrence and
deletes a key whose list becomes empty, so `len()` stays equal to the
old count through eviction.
- The store's memory estimate (`perSubpathEntryBytes`) is unchanged
here; accounting accuracy is a separate item.
- No new `map[string]interface{}`.
## SonarLint
SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.
- **Fixed** in `8af9582f`: S3776 (cognitive complexity 16 > 15) in the
test fixture. Path generation is split into graph and hop-count helpers;
the random draws stay in the same order, so the fixture is unchanged.
## Checked against the repo rules
- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
238 lines
8.4 KiB
Go
238 lines
8.4 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
var _ = time.Second // suppress unused import
|
|
|
|
// Helper to create a minimal PacketStore with GRP_TXT packets for channel analytics testing.
|
|
func newChannelTestStore(packets []*StoreTx) *PacketStore {
|
|
ps := &PacketStore{
|
|
packets: packets,
|
|
byHash: make(map[string]*StoreTx),
|
|
byTxID: make(map[int]*StoreTx),
|
|
byObsID: make(map[int]*StoreObs),
|
|
byObserver: make(map[string][]*StoreObs),
|
|
byNode: make(map[string][]*StoreTx),
|
|
byPathHop: make(map[string][]*StoreTx),
|
|
nodeHashes: make(map[string]map[string]bool),
|
|
byPayloadType: make(map[int][]*StoreTx),
|
|
rfCache: make(map[string]*cachedResult),
|
|
topoCache: make(map[string]*cachedResult),
|
|
hashCache: make(map[string]*cachedResult),
|
|
collisionCache: make(map[string]*cachedResult),
|
|
chanCache: make(map[string]*cachedResult),
|
|
distCache: make(map[string]*cachedResult),
|
|
subpathCache: make(map[string]*cachedResult),
|
|
spTxIndex: make(map[string][]*StoreTx),
|
|
advertPubkeys: make(map[string]int),
|
|
clockSkew: NewClockSkewEngine(),
|
|
}
|
|
ps.byPayloadType[5] = packets
|
|
return ps
|
|
}
|
|
|
|
func makeGrpTx(channelHash int, channel, text, sender string) *StoreTx {
|
|
return makeGrpTxWithStatus(channelHash, channel, text, sender, "")
|
|
}
|
|
|
|
// makeGrpTxWithStatus is like makeGrpTx but lets the caller set the ingestor's
|
|
// decryptionStatus ("decrypted" / "no_key" / "decryption_failed" / ""), which
|
|
// computeAnalyticsChannels consults to decide whether a channel name is
|
|
// trustworthy (see #1729).
|
|
func makeGrpTxWithStatus(channelHash int, channel, text, sender, decryptionStatus string) *StoreTx {
|
|
decoded := map[string]interface{}{
|
|
"type": "CHAN",
|
|
"channelHash": float64(channelHash),
|
|
"channel": channel,
|
|
"text": text,
|
|
"sender": sender,
|
|
}
|
|
if decryptionStatus != "" {
|
|
decoded["decryptionStatus"] = decryptionStatus
|
|
}
|
|
b, _ := json.Marshal(decoded)
|
|
pt := 5
|
|
return &StoreTx{
|
|
ID: 1,
|
|
DecodedJSON: string(b),
|
|
FirstSeen: "2026-05-01T12:00:00Z",
|
|
PayloadType: &pt,
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_MergesEncryptedAndDecrypted verifies that packets
|
|
// with the same hash byte but different decryption status merge into ONE bucket.
|
|
func TestComputeAnalyticsChannels_MergesEncryptedAndDecrypted(t *testing.T) {
|
|
// Hash 129 is the real hash for #wardriving: SHA256(SHA256("#wardriving")[:16])[0] = 129
|
|
// Some packets are decrypted (have channel name), some are not (encrypted)
|
|
packets := []*StoreTx{
|
|
makeGrpTx(129, "#wardriving", "hello", "alice"),
|
|
makeGrpTx(129, "#wardriving", "world", "bob"),
|
|
makeGrpTx(129, "", "", ""), // encrypted — no channel name
|
|
makeGrpTx(129, "", "", ""), // encrypted
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "#wardriving" {
|
|
t.Errorf("expected name '#wardriving', got %q", ch["name"])
|
|
}
|
|
if ch["messages"] != 4 {
|
|
t.Errorf("expected 4 messages, got %v", ch["messages"])
|
|
}
|
|
if ch["encrypted"] != false {
|
|
t.Errorf("expected encrypted=false (some packets decrypted), got %v", ch["encrypted"])
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_RejectsRainbowTableMismatch verifies that a packet
|
|
// with channelHash=72 but channel="#wardriving" (mismatch) does NOT create a
|
|
// "#wardriving" bucket — it falls into "ch72" instead.
|
|
func TestComputeAnalyticsChannels_RejectsRainbowTableMismatch(t *testing.T) {
|
|
// Hash 72 is NOT the correct hash for #wardriving (which is 129).
|
|
// This simulates a rainbow-table collision/mismatch.
|
|
packets := []*StoreTx{
|
|
makeGrpTx(72, "#wardriving", "ghost", "eve"), // mismatch: hash 72 != wardriving's real hash
|
|
makeGrpTx(129, "#wardriving", "real", "alice"), // correct match
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 2 {
|
|
t.Fatalf("expected 2 channel buckets, got %d: %+v", len(channels), channels)
|
|
}
|
|
|
|
// Find the buckets
|
|
var ch72, ch129 map[string]interface{}
|
|
for _, ch := range channels {
|
|
if ch["hash"] == "72" {
|
|
ch72 = ch
|
|
} else if ch["hash"] == "129" {
|
|
ch129 = ch
|
|
}
|
|
}
|
|
|
|
if ch72 == nil {
|
|
t.Fatal("expected a bucket for hash 72")
|
|
}
|
|
if ch129 == nil {
|
|
t.Fatal("expected a bucket for hash 129")
|
|
}
|
|
|
|
// ch72 should NOT be named "#wardriving" — it should be the placeholder
|
|
if ch72["name"] == "#wardriving" {
|
|
t.Errorf("hash 72 bucket should NOT be named '#wardriving' (rainbow-table mismatch rejected)")
|
|
}
|
|
if ch72["name"] != "ch72" {
|
|
t.Errorf("expected hash 72 bucket named 'ch72', got %q", ch72["name"])
|
|
}
|
|
|
|
// ch129 should be named "#wardriving"
|
|
if ch129["name"] != "#wardriving" {
|
|
t.Errorf("expected hash 129 bucket named '#wardriving', got %q", ch129["name"])
|
|
}
|
|
}
|
|
|
|
// TestChannelNameMatchesHash verifies the hash validation function.
|
|
func TestChannelNameMatchesHash(t *testing.T) {
|
|
// #wardriving hashes to 129
|
|
if !channelNameMatchesHash("#wardriving", "129") {
|
|
t.Error("expected #wardriving to match hash 129")
|
|
}
|
|
if channelNameMatchesHash("#wardriving", "72") {
|
|
t.Error("expected #wardriving to NOT match hash 72")
|
|
}
|
|
// Without leading # should also work
|
|
if !channelNameMatchesHash("wardriving", "129") {
|
|
t.Error("expected wardriving (without #) to match hash 129")
|
|
}
|
|
}
|
|
|
|
// TestIsPlaceholderName verifies placeholder detection.
|
|
func TestIsPlaceholderName(t *testing.T) {
|
|
if !isPlaceholderName("ch129") {
|
|
t.Error("ch129 should be placeholder")
|
|
}
|
|
if !isPlaceholderName("ch0") {
|
|
t.Error("ch0 should be placeholder")
|
|
}
|
|
if isPlaceholderName("#wardriving") {
|
|
t.Error("#wardriving should NOT be placeholder")
|
|
}
|
|
if isPlaceholderName("Public") {
|
|
t.Error("Public should NOT be placeholder")
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_PublicChannelPreserved is the regression test for
|
|
// #1729: the firmware-default "Public" channel (channel-hash byte 0x11 = 17,
|
|
// key-derived via SHA256(key)[0], NOT the hashtag scheme's 186) was being
|
|
// discarded by the #978 rainbow-table validation and rendered as
|
|
// "Encrypted (0x11)". The ingestor decrypts Public packets with the builtin
|
|
// well-known key and marks them decryptionStatus:"decrypted"; the server must
|
|
// trust that name instead of re-applying the hashtag hash check.
|
|
func TestComputeAnalyticsChannels_PublicChannelPreserved(t *testing.T) {
|
|
packets := []*StoreTx{
|
|
makeGrpTxWithStatus(17, "Public", "hello net", "alice", "decrypted"),
|
|
makeGrpTxWithStatus(17, "Public", "morning", "bob", "decrypted"),
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "Public" {
|
|
t.Errorf("expected name 'Public' preserved, got %q (must not be downgraded to ch17)", ch["name"])
|
|
}
|
|
if ch["encrypted"] != false {
|
|
t.Errorf("expected encrypted=false for decrypted Public channel, got %v", ch["encrypted"])
|
|
}
|
|
if ch["hash"] != "17" {
|
|
t.Errorf("expected hash '17', got %v", ch["hash"])
|
|
}
|
|
}
|
|
|
|
// TestComputeAnalyticsChannels_UndecryptedNameStillValidated ensures the #1729
|
|
// fix does not weaken #978: when the ingestor did NOT mark the packet
|
|
// "decrypted" (no_key / decryption_failed / absent), a channel name that fails
|
|
// the hashtag hash check is still discarded to "chNN", even if text/sender
|
|
// happen to be present (e.g. a stale/foreign rainbow-table hit).
|
|
func TestComputeAnalyticsChannels_UndecryptedNameStillValidated(t *testing.T) {
|
|
// Hash 17 is NOT the hashtag hash of #Public (that is 186). Without a
|
|
// "decrypted" status, the name must be rejected.
|
|
packets := []*StoreTx{
|
|
makeGrpTxWithStatus(17, "#Public", "leaked", "eve", ""),
|
|
}
|
|
|
|
store := newChannelTestStore(packets)
|
|
result := store.computeAnalyticsChannels("", "", TimeWindow{})
|
|
|
|
channels := result["channels"].([]map[string]interface{})
|
|
if len(channels) != 1 {
|
|
t.Fatalf("expected 1 channel bucket, got %d: %+v", len(channels), channels)
|
|
}
|
|
ch := channels[0]
|
|
if ch["name"] != "ch17" {
|
|
t.Errorf("expected undecrypted mismatch downgraded to 'ch17', got %q", ch["name"])
|
|
}
|
|
if ch["encrypted"] != true {
|
|
t.Errorf("expected encrypted=true for rejected rainbow-table name, got %v", ch["encrypted"])
|
|
}
|
|
}
|