mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 11:57:57 +00:00
Refs #1870. ## Problem `Load`, `loadChunk` and `scanAndMergeChunk` decode each observation's `resolved_path` only to collect its non-empty pubkeys. On live data that's a JSON array of pubkeys and nulls, about 420 characters, present on most observations. Decoding into `[]*string` allocated a string and a pointer for every hop, plus two slices. ## Change - `resolvedPubkeysFromJSON` reads the usual array of escape-free strings and nulls in place. Anything else falls back to decoding, so it returns exactly `extractResolvedPubkeys(unmarshalResolvedPath(s))`. - Relay pubkeys repeat across nearly every observation, so each one goes through `internStr` (from #2183) and is kept once. A pubkey seen before is found by substring, without copying, and the indexes it feeds (`byNode` and friends) share that copy. Interned keys live for the process, evicted nodes' included; the table is bounded at 64k entries. - The three bulk-load call sites use it. - **Eviction too.** `resolvedPubkeysForEvictionBatch` runs on every eviction batch, about once a minute in steady state, and spent about 130 MB per five minutes on a busy week. It now reads each pubkey in place and de-duplicates each packet's short list by scanning it instead of building a map per packet. - Other callers of `unmarshalResolvedPath` are unchanged. ## Measured | | before | after | |---|---|---| | allocations for 3 known pubkeys | 13 | 1 | | eviction, allocations per evicted packet (3 observations × 4 hops) | 51 | 15 | | this path's allocations over startup (copy of a busy week) | 236 MB | 61 MB | | server CPU over the run | 175.6 s | 170.6 s | The startup rows need that dataset and are from a single run. The benchmarks are rerunnable (`go test -run X -bench ResolvedPubkeys -benchmem ./cmd/server`, medians of 3): | | master | this PR | |---|---|---| | `BenchmarkResolvedPubkeysFromJSON` (5 relays + 3 nulls, pool of 2,000) | n/a | 619 ns, 128 B, 1 alloc | | `BenchmarkResolvedPubkeysForEvictionBatch` (1,000 packets × 3 observations) | 14.3 ms, 3.96 MB, 65,495 allocs | 6.7 ms, 1.36 MB, 18,501 allocs | ## Tests - `TestResolvedPubkeysFromJSONAllocations`: red commit first, with the bound tightened in green. - `TestResolvedPubkeysForEvictionBatch`: keys, order, and an allocation bound. It has its own red commit. - `TestResolvedPubkeysFromJSONMatchesDecoding` compares against decode-then-extract on 26 edge cases and 20,000 random inputs. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
38 lines
937 B
Go
38 lines
937 B
Go
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// One resolved_path per observation: 5 relays and 3 unresolved hops, drawn
|
|
// from a pool of 2,000 relays, as on a busy instance.
|
|
func BenchmarkResolvedPubkeysFromJSON(b *testing.B) {
|
|
const pool = 2000
|
|
paths := make([]string, 256)
|
|
for i := range paths {
|
|
hops := make([]string, 0, 8)
|
|
for h := 0; h < 5; h++ {
|
|
hops = append(hops, fmt.Sprintf(`"%064x"`, (i*7+h*131)%pool))
|
|
}
|
|
hops = append(hops, "null", "null", "null")
|
|
paths[i] = "[" + strings.Join(hops, ",") + "]"
|
|
}
|
|
b.ReportAllocs()
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
_ = resolvedPubkeysFromJSON(paths[i%len(paths)])
|
|
}
|
|
}
|
|
|
|
// An eviction batch of 1,000 packets with 3 observations each.
|
|
func BenchmarkResolvedPubkeysForEvictionBatch(b *testing.B) {
|
|
s, ids := evictionResolvedStore(b, 1000)
|
|
b.ReportAllocs()
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
_ = s.resolvedPubkeysForEvictionBatch(ids)
|
|
}
|
|
}
|