mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 11:57:57 +00:00
417c8dd00a243f29822f34e5c9aaadbd2e522213
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex BandClaude Opus 5.5
|
8d39a9d465 |
perf(store): count subpaths per request instead of indexing every run (#2182)
## Problem `spTxIndex` kept every 2–8 hop run of every stored path as a map key, with one transmission pointer per occurrence. On a busy regional network's 7-day window (59k transmissions) that's about 0.9–1.07 M keys and 130–160 MB of live heap, roughly a quarter of the store. It was also rebuilt at the end of every startup, which is when memory peaks. Two smaller issues came with it: - Eviction never decremented `spTotalPaths`, so the percentages were divided by paths that were no longer stored. - Route Patterns took 1.2–3.6 s uncached, because the fast path still split, resolved and re-joined about a million raw keys per request. ## Change - **Count on request.** Subpaths are counted when the analytics are asked for, from the stored paths. Results stay cached for `rfCacheTTL`, as before. - **Pointer-free tallies.** `tallyNamedSubpaths` counts runs in one pointer-free `map[uint64]subpathTally`, keyed by a `maphash` of the resolved name built in a reused buffer. The GC doesn't scan the map, and counting allocates nothing but the map. - **Names only for the top entries.** Each tally records the first transmission and offset where its run appeared, so names and raw hops are built only for each group's top entries. `topSubpathTallies` picks those with a `container/heap` min-heap of `limit` entries instead of sorting every name, and the existing `rankSubpaths` formats the response, so no new response maps are added. - **Hash collisions.** If two names shared a hash their counts would merge; the odds are about 1 in 10⁷ for a few million names. - **Subpath detail.** `GetSubpathDetail` finds the run with `txsWithSubpath`, a scan of the stored paths. - **Counts unchanged:** runs of 2–8 hops, counted once per occurrence, case-folded. The tests pin this. - **Totals.** `spTotalPaths` stays, and eviction now decrements it. - **Memory estimate.** The O(path²) subpath term is dropped from the estimate. ## Perf justification (AGENTS.md rule 0) - **Complexity.** A cold Route Patterns request is O(Σ runs), about 1.5 M for this window: one pass over the stored paths, with one map lookup per run against a reused buffer. Ranking is O(tallies · log limit). Results are cached for `rfCacheTTL`, so this runs at most once per TTL per group set. - **Subpath detail** is O(stored hops), one pass with no allocation per packet. It's an on-click endpoint, not a hot path. - **Ingest and eviction** no longer touch a subpath index at all. Removal used to be a linear search of every key's transmission list. **Measured on a copy of that window**, upstream master vs this branch, same queries: | | master | this PR | |---|---|---| | peak RSS (`GOMEMLIMIT=750MiB`) | 948 MB | 767 MB | | in-use heap after load (750MiB) | 592 MB | 456 MB | | `/api/analytics/subpaths` cold (750MiB) | 3.56 s | 0.64 s | | `/api/analytics/subpaths-bulk` cold (750MiB) | 3.33 s | 0.70 s | | `/api/analytics/subpaths` cold (1GiB) | 1.18 s | 0.61 s | | `/api/analytics/subpath-detail` | 0.6 ms | 8 ms | | cached responses | 0.6 ms | 0.5 ms | At 750 MiB, master's request time is mostly GC pressure from the index. On the production-shaped fixture that `subpath_index_memory_test.go` builds (41k transmissions): | | master | this PR | |---|---|---| | retained by the store for subpaths | 197 MB | 0 MB | | allocated per bulk request | 336 MB | 111 MB | This has also run on a staging instance with live MQTT ingest for about nine hours, with Route Patterns answering in about 1 s. ## Tests - `subpaths_on_demand_test.go`: - per-occurrence, case-folded counts; - the 8-hop cap; - bulk matches single; - detail counts, including repeats, case and the length limits; - eviction drops evicted paths from the totals. - `TestSubpathIndexRetainedHeap`: the bound is now 5 MB (it was 225 MB). - `TestSubpathRequestAllocations`: 150 MB per request. - Each change has its red commit first, failing on assertions. - Tests that reached into `spTxIndex` now assert through `txsWithSubpath` and the public counts. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. **Overlap with #2181:** both edit the same memory-estimate comment block in `store.go`. Whichever lands second needs a trivial rebase there; there's no code dependency. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
dc7ebcd6f5 |
perf(store): check observation duplicates without per-transmission maps (#2181)
## Problem Every `StoreTx` carried two maps for observation de-duplication: - `obsKeys`, keyed by `observerID + "|" + pathJSON` (a concatenated string allocated for every observation); - `observerSet`, for `UniqueObserverCount`. On a busy regional network's 7-day window (64k transmissions, 400k observations), these maps and their concatenated keys held about 60 MB of heap. Most transmissions are heard a handful of times; the busiest in that week was heard 40 times. The maps cost a few hundred bytes per transmission to de-duplicate lists that are usually under ten long. ## Change - `hasObservation` and `hasObserver` scan `tx.Observations`, comparing the observer ID and path strings the observations already hold. These are the same two values the old map key was built from, so the answers are identical. - Once a transmission passes 64 observations (`obsIndexThreshold`), it builds the maps. They're keyed by a struct that shares the observations' strings, so no key bytes are allocated, and busy transmissions keep O(1) checks (the case #543 / #355 addressed). - The five places that appended an observation now call `addObservation`, which keeps `UniqueObserverCount` and `ObservationCount` in step. Those three lines were previously copied at each site. - The memory estimate no longer charges for the per-transmission maps or the concatenated keys. ## Perf justification (AGENTS.md rule 0) - **Complexity.** - Below the threshold, a duplicate check is O(k) for a transmission with k < 64 observations, so building it costs at most 63·64/2 = 2016 string-header comparisons. - At and above 64 observations, checks are O(1) map lookups, as before. - Nothing is O(n²) in the number of transmissions or observations overall: the scan is bounded per transmission. - **Scale.** Real data peaks at 40 observations per transmission. The scan compares string headers (length first, then bytes), and 64-character observer IDs differ early. `BenchmarkObsDedup` (a miss, the common case on ingest): | observations on tx | check used | time | |---|---|---| | 4 | scan | 15 ns | | 16 | scan | 70 ns | | 40 | scan | 151 ns | | 63 | scan | 233 ns | | 64+ | map (struct key) | 15 ns | For comparison, the old map lookup was about 15 ns too, plus building and allocating the `observerID|path` key string on every check. At 40 observations that's about 3 µs extra to build a whole transmission, against roughly 2 KB of maps no longer allocated for it. **Measured on a copy of that 7-day window** (`GOMEMLIMIT=750MiB`, same queries): | | before | after | |---|---|---| | in-use heap after load | 612 MB | 568 MB | | HeapAlloc after load | 660 MB | 564 MB | | peak RSS | 1077 MB | 1009 MB | | startup to ready | 31 s | 21 s | It has also run on a staging instance (live MQTT ingest) for about nine hours alongside other store changes, with no change in observation or observer counts. ## Tests - `obs_dedup_test.go`: - duplicates (same observer and path) are rejected, while a new path or a new observer is kept; - a transmission under the threshold carries no maps; - across the threshold the maps take over with the same answers; - empty observer IDs aren't counted as observers; - `BenchmarkObsDedup`. - Red commit first (stubs, failing on assertions), then green. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex B
|
e6323ec587 |
fix(store): account path, decode-cache and dedup-key bytes so maxMemoryMB eviction triggers (#2035)
trackedBytes undercounted the packet store by about 2.5x, so packetStore.maxMemoryMB never triggered: a tx was charged at creation, before pickBestObservation set its path, so the byPathHop and spTxIndex costs were never added, and eviction then re-estimated with the path known and subtracted more than had been added, drifting the total downwards. The ParsedDecoded cache, the obsKeys dedup key and several per-observation strings were not estimated at all. StoreTx.accountedBytes now records what was charged, rechargeTx returns the delta after every pickBestObservation, and eviction subtracts accountedBytes instead of re-estimating. Measured by the author on a production database copy: trackedMB 151 against 402 MB of heap in use before, 351 against 396 MB after, with GC cycles dropping from ~0.71/s to ~0.011/s over 12 h on their instance. Reviewed by auditing the accounting rather than the arithmetic: all six production pickBestObservation sites recharge, all three subtraction sites read accountedBytes, every recharge site holds s.mu (Load from :857, the two ingest paths at :2778 and :3142 with deferred unlocks), observations are charged only after acceptance, and no charged tx is discarded during the chunk merge. That lock audit is the independent check, because CI's race job covers the ingestor only. Operator impact, both from the estimate growing rather than any limit moving: where maxMemoryMB is set, eviction now caps the real store size, and the cold load clamp drops about a third of the boot walk (124420 to 84374 packets at 650 MB). Where it is unset, which is the default, the change is inert. docs/go-migration.md claimed the Go server ignored the setting, which was never true, and is corrected here. Merged by the interim maintainer without a second human reviewer: CI (run 35258839322) plus the review above are the independent checks. |
||
|
Kpa-clawbotandyou
|
9e90548637 |
perf(#800): remove per-StoreTx ResolvedPath, replace with membership index + on-demand decode (#806)
## Summary Remove `ResolvedPath []*string` field from `StoreTx` and `StoreObs` structs, replacing it with a compact membership index + on-demand SQL decode. This eliminates the dominant heap cost identified in profiling (#791, #799). **Spec:** #800 (consolidated from two rounds of expert + implementer review on #799) Closes #800 Closes #791 ## Design ### Removed - `StoreTx.ResolvedPath []*string` - `StoreObs.ResolvedPath []*string` - `TransmissionResp.ResolvedPath`, `ObservationResp.ResolvedPath` struct fields ### Added | Structure | Purpose | Est. cost at 1M obs | |---|---|---:| | `resolvedPubkeyIndex map[uint64][]int` | FNV-1a(pubkey) → []txID forward index | 50–120 MB | | `resolvedPubkeyReverse map[int][]uint64` | txID → []hashes for clean removal | ~40 MB | | `apiResolvedPathLRU` (10K entries) | FIFO cache for on-demand API decode | ~2 MB | ### Decode-window discipline `resolved_path` JSON decoded once per packet. Consumers fed in order, temp slice dropped — never stored on struct: 1. `addToByNode` — relay node indexing 2. `touchRelayLastSeen` — relay liveness DB updates 3. `byPathHop` resolved-key entries 4. `resolvedPubkeyIndex` + reverse insert 5. WebSocket broadcast map (raw JSON bytes) 6. Persist batch (raw JSON bytes for SQL UPDATE) ### Collision safety When the forward index returns candidates, a batched SQL query confirms exact pubkey presence using `LIKE '%"pubkey"%'` on the `resolved_path` column. ### Feature flag `useResolvedPathIndex` (default `true`). Off-path is conservative: all candidates kept, index not consulted. For one-release rollback safety. ## Files changed | File | Changes | |---|---| | `resolved_index.go` | **New** — index structures, LRU cache, on-demand SQL helpers, collision safety | | `store.go` | Remove RP fields, decode-window discipline in Load/Ingest, on-demand txToMap/obsToMap/enrichObs, eviction cleanup via SQL, memory accounting update | | `types.go` | Remove RP fields from TransmissionResp/ObservationResp | | `routes.go` | Replace `nodeInResolvedPath` with `nodeInResolvedPathViaIndex`, remove RP from mapSlice helpers | | `neighbor_persist.go` | Refactor backfill: reverse-map removal → forward+reverse insert → LRU invalidation | ## Tests added (27 new) **Unit:** - `TestStoreTx_ResolvedPathFieldAbsent` — reflection guard - `TestResolvedPubkeyIndex_BuildFromLoad` — forward+reverse consistency - `TestResolvedPubkeyIndex_HashCollision` — SQL collision safety - `TestResolvedPubkeyIndex_IngestUpdate` — maps reflect new ingests - `TestResolvedPubkeyIndex_RemoveOnEvict` — clean removal via reverse map - `TestResolvedPubkeyIndex_PerObsCoverage` — non-best obs pubkeys indexed - `TestAddToByNode_WithoutResolvedPathField` - `TestTouchRelayLastSeen_WithoutResolvedPathField` - `TestWebSocketBroadcast_IncludesResolvedPath` - `TestBackfill_InvalidatesLRU` - `TestEviction_ByNodeCleanup_OnDemandSQL` - `TestExtractResolvedPubkeys`, `TestMergeResolvedPubkeys` - `TestResolvedPubkeyHash_Deterministic` - `TestLRU_EvictionOnFull` **Endpoint:** - `TestPathsThroughNode_NilResolvedPathFallback` - `TestPacketsAPI_OnDemandResolvedPath` - `TestPacketsAPI_OnDemandResolvedPath_LRUHit` - `TestPacketsAPI_OnDemandResolvedPath_Empty` **Feature flag:** - `TestFeatureFlag_OffPath_PreservesOldBehavior` - `TestFeatureFlag_Toggle_NoStateLeak` **Concurrency:** - `TestReverseMap_NoLeakOnPartialFailure` - `TestDecodeWindow_LockHoldTimeBounded` - `TestLivePolling_LRUUnderConcurrentIngest` **Regression:** - `TestRepeaterLiveness_StillAccurate` **Benchmarks:** - `BenchmarkLoad_BeforeAfter` - `BenchmarkResolvedPubkeyIndex_Memory` - `BenchmarkPathsThroughNode_Latency` - `BenchmarkLivePolling_UnderIngest` ## Benchmark results ``` BenchmarkResolvedPubkeyIndex_Memory/pubkeys=50K 429ms 103MB 777K allocs BenchmarkResolvedPubkeyIndex_Memory/pubkeys=500K 4205ms 896MB 7.67M allocs BenchmarkLoad_BeforeAfter 65ms 20MB 202K allocs BenchmarkPathsThroughNode_Latency 3.9µs 0B 0 allocs BenchmarkLivePolling_UnderIngest 5.4µs 545B 7 allocs ``` Key: per-obs `[]*string` overhead completely eliminated. At 1M obs with 3 hops average, this saves ~72 bytes/obs × 1M = ~68 MB just from the slice headers + pointers, plus the JSON-decoded string data (~900 MB at scale per profiling). ## Design choices - **FNV-1a instead of xxhash**: stdlib availability, no external dependency. Performance is equivalent for this use case (pubkey strings are short). - **FIFO LRU instead of true LRU**: simpler implementation, adequate for the access pattern (mostly sequential obs IDs from live polling). - **Grouped packets view omits resolved_path**: cold path, not worth SQL round-trip per page render. - **Backfill pending check uses reverse-map presence** instead of per-obs field: if a tx has any indexed pubkeys, its observations are considered resolved. Closes #807 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
401fd070f8 |
fix: improve trackedBytes accuracy for memory estimation (#751)
## Problem Fixes #743 — High memory usage / OOM with relatively small dataset. `trackedBytes` severely undercounted actual per-packet memory because it only tracked base struct sizes and string field lengths, missing major allocations: | Structure | Untracked Cost | Scale Impact | |-----------|---------------|--------------| | `spTxIndex` (O(path²) subpath entries) | 40 bytes × path combos | 50-150MB | | `ResolvedPath` on observations | 24 bytes × elements | ~25MB | | Per-tx maps (`obsKeys`, `observerSet`) | 200 bytes/tx flat | ~11MB | | `byPathHop` index entries | 50 bytes/hop | 20-40MB | This caused eviction to trigger too late (or not at all), leading to OOM. ## Fix Expanded `estimateStoreTxBytes` and `estimateStoreObsBytes` to account for: - **Per-tx maps**: +200 bytes flat for `obsKeys` + `observerSet` map headers - **Path hop index**: +50 bytes per hop in `byPathHop` - **Subpath index**: +40 bytes × `hops*(hops-1)/2` combinations for `spTxIndex` - **Resolved paths**: +24 bytes per `ResolvedPath` element on observations Updated the existing `TestEstimateStoreTxBytes` to match new formula. All existing eviction tests continue to pass — the eviction logic itself is unchanged. Also exposed `avgBytesPerPacket` in the perf API (`/api/perf`) so operators can monitor per-packet memory costs. ## Performance Benchmark confirms negligible overhead (called on every insert): ``` BenchmarkEstimateStoreTxBytes 159M ops 7.5 ns/op 0 B/op 0 allocs BenchmarkEstimateStoreObsBytes 1B ops 1.0 ns/op 0 B/op 0 allocs ``` ## Tests - 6 new tests in `tracked_bytes_test.go`: - Reasonable value ranges for different packet sizes - 10-hop packets estimate significantly more than 2-hop (subpath cost) - Observations with `ResolvedPath` estimate more than without - 15 observations estimate >10x a single observation - `trackedBytes` matches sum of individual estimates after batch insert - Eviction triggers correctly with improved estimates - 2 benchmarks confirming sub-10ns estimate cost - Updated existing `TestEstimateStoreTxBytes` for new formula - Full test suite passes --------- Co-authored-by: you <you@example.com> |