mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-26 16:07:55 +00:00
13bdee57d4
## What Three of the four P0s from #1481's scale-test findings. Each cuts a distinct hot path; together they target /api/observers, /api/analytics/neighbor-graph, and /api/observers/{id}/analytics — the top three live offenders. ### P0-1: 5-min atomic-pointer cache for default neighbor-graph response - Live p95 10.8s on the most-trafficked organic endpoint. - Background recomputer (5-min cadence per operator directive) builds the default-filter (`minCount=5 minScore=0.1`, no region, no role) `NeighborGraphResponse` and stores it via `atomic.Pointer`. - `handleNeighborGraph` short-circuits on the default shape; non-default filters take the extracted `computeNeighborGraphResponse` path (identical semantics to the previous inline build). ### P0-2: cache parsed `StoreObs.Timestamp` + drop RLock window - `handleObserverAnalytics` re-parsed the RFC3339 timestamp three times per observation, for 60k+ observations per active observer, under `s.store.mu.RLock` — blocking writers for the full scan. - `StoreObs.ParsedTime()` parses once via `sync.Once` (mirrors `StoreTx.ParsedDecoded`). - Handler snapshots the `byObserver[id]` pointer slice, releases the RLock immediately, then iterates locally. ### P0-3: 30s cache for `/api/observers` + sargable `IN` + covering index - Three SQL queries on every request → ~1.7s p50 at 50-concurrent. - Atomic-pointer 30s cache for the default (no-filter) query. - `GetNodeLocationsByKeys` drops `LOWER(public_key) IN (...)` (non-sargable); callers pre-lowercase in Go and the plain `IN` matches the existing `public_key` index. - New ingestor migration `obs_observer_ts_idx_v1` adds composite index `idx_observations_observer_idx_timestamp(observer_idx, timestamp)` so `GetObserverPacketCounts` can resolve its GROUP-BY + range filter from the index without scanning the 1.9M-row observations table. ### P0-4: deferred `perfMiddleware`'s global mutex was claimed to serialize every API request. A direct test (`50 concurrent requests through the middleware, handler sleeps 20ms each`) shows total elapsed ≈ 25ms, not 1s — the lock is held only for the post-handler bookkeeping (a few µs). Real impact is below measurement noise. Skipping to avoid invasive churn on PerfStats consumers without a demonstrable win. ## Test plan Red → green per P0: - `observers_cache_test.go` — handler reads `s.observersCache` before SQL, TTL boundary, atomic.Pointer (no mutex contention). - `storeobs_parsedtime_test.go` — parses three timestamp shapes, caches result, no race under concurrent readers. - `neighbor_graph_cache_test.go` — handler serves from atomic pointer when set, bypasses cache when `?region=` (or any non-default filter) is passed. Full server + ingestor suites pass: `go test -count=1 ./...`. ## Perf proof Before/after p50/p95/p99 (50 requests × 50 concurrent) against prod (before) and staging once CI deploys (after) will be posted as a PR comment per the operator's "no merge without proof of improvement" gate. Closes #1481 ## TDD exemption — P0-1 and P0-2 (net-new surfaces, AGENTS.md) Per CoreScope `AGENTS.md` § "Exemptions": **net-new code surfaces with no prior tests to break** may land tests in the same PR without a strict test-first → impl commit split. - **P0-1 (neighbor-graph atomic-pointer cache)** — `neighborGraphCache`, `recomputeNeighborGraphCache`, `loadNeighborGraphCacheBytes`, `startNeighborGraphRecomputer` and the default-shape short-circuit in `handleNeighborGraph` were brand-new code with no pre-existing assertions covering them. There was no green test to first turn red. - **P0-2 (cached `StoreObs.Timestamp` + RLock window drop)** — `StoreObs.ParsedTime()` and the snapshot+release pattern in `handleObserverAnalytics` were new surfaces; the prior code did the parse inline per call with no behavioural test to break. P0-3 was authored properly red-then-green (commit `6e63ec6a` red, then `83ae129b` green) and does NOT use this exemption. ## Default-filter detection vs frontend reality (#1483 follow-up) The Neighbor Graph analytics tab in `public/analytics.js` fetches `/analytics/neighbor-graph?min_count=1&min_score=0` because the client-side sliders need the full edge set to filter from. That shape did NOT match the `(5, 0.1)` cached default, so the UI tab still paid the cold compute cost despite #1481 P0-1. The #1483 follow-up commit caches BOTH shapes in the same recomputer pass: - `(minCount=5, minScore=0.1, no region, no role)` — `live.js` affinity-scoring consumer. - `(minCount=1, minScore=0, no region, no role)` — analytics tab. Both are served from `atomic.Pointer` with an `X-Cache-Age-Seconds` header. The per-shape cost in the background goroutine is roughly linear in edge count; total recompute time stays well under the 5-minute cadence on prod-scale graphs. --------- Co-authored-by: openclaw-bot <bot@openclaw.dev> Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
64 lines
1.7 KiB
Go
64 lines
1.7 KiB
Go
package main
|
|
|
|
import (
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// TestStoreObsParsedTimeCaches asserts ParsedTime parses once and
|
|
// returns the same value on subsequent calls — required so #1481 P0-2
|
|
// (handleObserverAnalytics RLock-held parse loop) doesn't reparse 60k
|
|
// times per request.
|
|
func TestStoreObsParsedTimeCaches(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
ts string
|
|
want time.Time
|
|
ok bool
|
|
}{
|
|
{"rfc3339nano", "2026-05-29T07:00:00.123456789Z",
|
|
time.Date(2026, 5, 29, 7, 0, 0, 123456789, time.UTC), true},
|
|
{"rfc3339", "2026-05-29T07:00:00Z",
|
|
time.Date(2026, 5, 29, 7, 0, 0, 0, time.UTC), true},
|
|
{"sql-shape", "2026-05-29 07:00:00",
|
|
time.Date(2026, 5, 29, 7, 0, 0, 0, time.UTC), true},
|
|
{"empty", "", time.Time{}, false},
|
|
{"garbage", "not-a-time", time.Time{}, false},
|
|
}
|
|
for _, c := range cases {
|
|
t.Run(c.name, func(t *testing.T) {
|
|
o := &StoreObs{Timestamp: c.ts}
|
|
got, ok := o.ParsedTime()
|
|
if ok != c.ok {
|
|
t.Fatalf("ok=%v want %v", ok, c.ok)
|
|
}
|
|
if ok && !got.Equal(c.want) {
|
|
t.Fatalf("got %v want %v", got, c.want)
|
|
}
|
|
// Second call must return identical result (cached).
|
|
got2, ok2 := o.ParsedTime()
|
|
if got2 != got || ok2 != ok {
|
|
t.Fatalf("cache inconsistent: %v/%v vs %v/%v", got, ok, got2, ok2)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestStoreObsParsedTimeConcurrent asserts no race under concurrent
|
|
// readers — sync.Once guards the parse.
|
|
func TestStoreObsParsedTimeConcurrent(t *testing.T) {
|
|
o := &StoreObs{Timestamp: "2026-05-29T07:00:00Z"}
|
|
var wg sync.WaitGroup
|
|
for i := 0; i < 100; i++ {
|
|
wg.Add(1)
|
|
go func() {
|
|
defer wg.Done()
|
|
if _, ok := o.ParsedTime(); !ok {
|
|
t.Error("expected ok")
|
|
}
|
|
}()
|
|
}
|
|
wg.Wait()
|
|
}
|