3 Commits

Include renames

Author	SHA1	Message	Date
Alex BandClaude Opus 5.5	be6aeffe45	perf(store): drop the duplicate subpath count map (#2163) ... ## Summary The subpath index keeps every contiguous run of 2-8 hops from every stored transmission's path, in two maps: - `spIndex`: key → count - `spTxIndex`: key → transmissions containing that subpath, one entry per occurrence Both were always updated together (the only production callers were `addTxToSubpathIndexFull` / `removeTxFromSubpathIndexFull`, which touch both), so every count equalled `len(spTxIndex[key])`. The existing `TestSubpathTxIndexPopulated` asserted exactly that. The index therefore held its 1M+ keys twice. This PR removes `spIndex`. The two analytics consumers read `len(txs)`, and the add/remove helpers take only the transaction map. No API response changes. No existing upstream issue covers this; it came out of profiling a production instance where garbage collection was 87% of CPU (30 s CPU profile: `gcDrain` 39.9 s of 45.9 s sampled) with an 805 MB live heap against a 1 GiB `GOMEMLIMIT`. That heap profile put 186 MB in this index; on that instance's 7-day window there are 1.2M distinct subpath keys (79% seen once) and 3.07M entries. ## Performance ### Go benchmark `BenchmarkBuildSubpathIndex` builds the index for a synthetic, production-shaped window: 41k transmissions with path lengths drawn from the production hop-count histogram, and hops from random walks over a 400-node graph with 3-byte prefixes. That gives 1.48M keys, 85% seen once (production: 1.2M, 79%). No real data is used. 10 runs each: ``` cpu: AMD Ryzen 5 3600XT 6-Core Processor │ master │ this PR │ │ sec/op │ sec/op vs base │ BuildSubpathIndex-12 2.157 ± 1% 1.610 ± 5% -25.33% (p=0.000 n=10) │ retained-B │ retained-B vs base │ BuildSubpathIndex-12 238.5Mi ± 0% 187.7Mi ± 0% -21.31% (p=0.000 n=10) │ B/op │ B/op vs base │ BuildSubpathIndex-12 572.0Mi ± 0% 465.6Mi ± 0% -18.61% (p=0.000 n=10) ``` (`retained-B` is the live heap the index keeps after a forced GC: 250.1 MB → 196.8 MB.) ### Staging heap Live heap after a forced GC (`/debug/pprof/heap?gc=1`), on a staging host running a copy of a production database and its live MQTT feed, attributed to the subpath index builder: | | master | this PR | |---|---|---| | subpath index (`buildSubpathIndex`, cum) | 172.4 MB | 116.1 MB | | packets in memory at the time | 27,521 | 24,086 | | per packet | 6.27 KB | 4.82 KB (**-23%**) | The two snapshots were taken minutes apart with different numbers of packets loaded, hence the per-packet figure; it agrees with the benchmark's -21%. The same A/B replay used for the other PRs (30 minutes of real production API traffic) showed no latency change (p50 10.3 ms vs 10.7 ms, p95 59.5 ms vs 52.9 ms, 0 errors). GC count and CPU were lower in this run (44 → 28 collections, 13.0% → 9.6% CPU), but runs on this shared host vary by that much on their own, so no claim is made from them. ## Tests - Red commit `ded359ff`: the fixture, `TestSubpathFixtureShape` (keeps the fixture production-like: 60-95% singleton keys), `BenchmarkBuildSubpathIndex`, and `TestSubpathIndexRetainedHeap`, which allows at most 225 MB for the 41k fixture. It fails on `master` with 250.1 MB. - Green commit `e399c561`: the change. The heap test passes at about 197 MB. - `make test`: 16 of 16 modules pass; `go test -race` over the subpath, eviction, path-hop and resolved-index tests passes. Test changes in the green commit are mechanical, as required for a structural change: - struct literals stop setting `spIndex`; where a literal did not set `spTxIndex` it now does, because appending to a nil map panics where the old code skipped the transaction map when passed `nil`; - count assertions read `len(spTxIndex[key])`; - the old "spIndex and spTxIndex agree" check becomes "no key is kept with an empty list", which is what keeps `len()` a correct count. ## Review notes - Removal still takes out one occurrence per subpath occurrence and deletes a key whose list becomes empty, so `len()` stays equal to the old count through eviction. - The store's memory estimate (`perSubpathEntryBytes`) is unchanged here; accounting accuracy is a separate item. - No new `map[string]interface{}`. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `8af9582f`: S3776 (cognitive complexity 16 > 15) in the test fixture. Path generation is split into graph and hop-count helpers; the random draws stay in the same order, so the fixture is unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 21:21:13 +02:00
Kpa-clawbotandopenclaw-bot	ad41b9bb7b	fix(tests): subpaths_window tests wait for index readiness after #1595 chunked load (#1621) ... ## Why master is red After PRs #1592 (route-window subpath regression test) and #1595 (background/chunked index build with 503 readiness gate) were merged together, two tests in `cmd/server/subpaths_window_test.go` started failing on master: ``` --- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[]) --- FAIL: TestSubpathsHandlerHonorsTimeWindow subpaths_window_test.go:116: GET /api/analytics/subpaths?...: status=503 body={"error":"index loading","retryAfter":5} ``` Both branches passed in isolation; the conflict only manifested post-merge. Reason: - **#1592** added tests that call `store.Load()` then immediately query `GetAnalyticsSubpathsWithWindow` / hit `/api/analytics/subpaths`. - **#1595** moved the subpath + path-hop index builds off the critical path of `Load()` into background goroutines, and hard-gated the analytics handlers behind `SubpathIndexReady()` (returning 503 + `Retry-After: 5` until the build completes). So after `Load()` returns, `s.spIndex` is still empty for a short window and the handler returns 503. The store-level test sees `totalPaths=0`; the handler test sees the 503. ## Fix (test-only) Add `store.WaitIndexesReady(5 * time.Second)` between `Load()` and the assertions in both tests. This matches the established pattern already used by `routes_test.go` and `repeater_enrich_recomputer_1008_test.go`. The 503 readiness gate from #1595 is intentional production behavior and is **not** touched. No production code is modified. ## Repro Before: ``` $ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=1 --- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s) subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[]) --- FAIL: TestSubpathsHandlerHonorsTimeWindow (0.02s) subpaths_window_test.go:116: GET /api/analytics/subpaths?minLen=2&maxLen=8: status=503 body={"error":"index loading","retryAfter":5} FAIL ``` After: ``` $ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=3 --- PASS: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s) --- PASS: TestSubpathsHandlerHonorsTimeWindow (0.02s) ... (x3) ... PASS ok github.com/corescope/server 0.097s $ go test ./cmd/server/ -count=1 -timeout 300s ok github.com/corescope/server 46.292s ``` ## Files changed - `cmd/server/subpaths_window_test.go` (+11 lines, test-only) ## Notes - TDD exemption: this is a test-fix PR for a merge-conflict-induced failure. The "failing test" already exists on master; this PR makes it pass correctly by waiting on the readiness gate the test was previously unaware of. - Unblocks staging deploys. Co-authored-by: openclaw-bot <bot@openclaw>	2026-06-06 21:59:23 -07:00
Kpa-clawbotandKpa-clawbot	d6384c3c59	fix(#1217): honor time-window filter on Route Patterns analytics (#1592) ... ## What The Route Patterns chart on `/#/analytics` ignored the Time window picker — every selection returned identical data. This PR threads `?window=` through to the backing endpoints and the store-level computation. ## Root cause `cmd/server/routes.go:2065` (`handleAnalyticsSubpaths`) and `cmd/server/routes.go:2090` (`handleAnalyticsSubpathsBulk`) never called `ParseTimeWindow(r)`. The store-level entry points (`GetAnalyticsSubpaths`, `GetAnalyticsSubpathsBulk`) had no window-aware variant. The frontend (`public/analytics.js`) didn't append `&window=` to the `/analytics/subpaths-bulk` request. ## Fix ### Backend (`cmd/server/store.go`) Added `GetAnalyticsSubpathsWithWindow` + `GetAnalyticsSubpathsBulkWithWindow`. Zero `TimeWindow` → byte-equivalent to the existing fast path (no perf regression on the default view). Non-zero window → iterate `s.packets`, filter on `tx.FirstSeen` via `TimeWindow.Includes`, reuse `rankSubpaths`. Cached by `(region|area|window)`. ```diff -data := s.store.GetAnalyticsSubpaths(region, minLen, maxLen, limit) +window := ParseTimeWindow(r) +data := s.store.GetAnalyticsSubpathsWithWindow(region, minLen, maxLen, limit, window) ``` ```diff -results := s.store.GetAnalyticsSubpathsBulk(region, groups) +results := s.store.GetAnalyticsSubpathsBulkWithWindow(region, groups, ParseTimeWindow(r)) ``` ### Frontend (`public/analytics.js`) `renderSubpaths` now appends `&window=<value>` to the `/analytics/subpaths-bulk` request, matching how RF / topology / channels tabs already wire the picker. ## Before / after ``` GET /api/analytics/subpaths?window=24h → totalPaths=2 (all data — ignored window) GET /api/analytics/subpaths?window=24h → totalPaths=1 (24h-bounded — honored) ``` ## Tests `cmd/server/subpaths_window_test.go`: - `TestSubpathsHonorsTimeWindow_StoreLevel` — seeds a 1h-old tx with path `[aa,bb]` + a 30d-old tx with path `[cc,dd]`; asserts the unbounded call sees both and the 24h-windowed call sees only the recent one. - `TestSubpathsHandlerHonorsTimeWindow` — same scenario via the HTTP handlers for `/api/analytics/subpaths` and `/api/analytics/subpaths-bulk`. TDD: red commit `eefc27d3` (test fails on assertion with stub that ignores window), green commit `4c4c45d0` (implementation makes it pass). Full `go test ./...` in `cmd/server` green locally (~47s). ## Performance Default view (no window selected) is unchanged — `window.IsZero()` short-circuits to the existing precomputed-index hot path. Windowed view is O(N_tx · path²), same complexity as the existing region-filtered slow path. Results cached per `(region|area|window)`. Closes #1217 --------- Co-authored-by: Kpa-clawbot <bot@corescope>	2026-06-06 20:43:49 -07:00