## What
Three of the four P0s from #1481's scale-test findings. Each cuts a
distinct
hot path; together they target /api/observers,
/api/analytics/neighbor-graph,
and /api/observers/{id}/analytics — the top three live offenders.
### P0-1: 5-min atomic-pointer cache for default neighbor-graph response
- Live p95 10.8s on the most-trafficked organic endpoint.
- Background recomputer (5-min cadence per operator directive) builds
the
default-filter (`minCount=5 minScore=0.1`, no region, no role)
`NeighborGraphResponse` and stores it via `atomic.Pointer`.
- `handleNeighborGraph` short-circuits on the default shape; non-default
filters take the extracted `computeNeighborGraphResponse` path
(identical
semantics to the previous inline build).
### P0-2: cache parsed `StoreObs.Timestamp` + drop RLock window
- `handleObserverAnalytics` re-parsed the RFC3339 timestamp three times
per observation, for 60k+ observations per active observer, under
`s.store.mu.RLock` — blocking writers for the full scan.
- `StoreObs.ParsedTime()` parses once via `sync.Once` (mirrors
`StoreTx.ParsedDecoded`).
- Handler snapshots the `byObserver[id]` pointer slice, releases the
RLock immediately, then iterates locally.
### P0-3: 30s cache for `/api/observers` + sargable `IN` + covering
index
- Three SQL queries on every request → ~1.7s p50 at 50-concurrent.
- Atomic-pointer 30s cache for the default (no-filter) query.
- `GetNodeLocationsByKeys` drops `LOWER(public_key) IN (...)`
(non-sargable);
callers pre-lowercase in Go and the plain `IN` matches the existing
`public_key` index.
- New ingestor migration `obs_observer_ts_idx_v1` adds composite index
`idx_observations_observer_idx_timestamp(observer_idx, timestamp)` so
`GetObserverPacketCounts` can resolve its GROUP-BY + range filter from
the index without scanning the 1.9M-row observations table.
### P0-4: deferred
`perfMiddleware`'s global mutex was claimed to serialize every API
request.
A direct test (`50 concurrent requests through the middleware, handler
sleeps 20ms each`) shows total elapsed ≈ 25ms, not 1s — the lock is held
only for the post-handler bookkeeping (a few µs). Real impact is below
measurement noise. Skipping to avoid invasive churn on PerfStats
consumers
without a demonstrable win.
## Test plan
Red → green per P0:
- `observers_cache_test.go` — handler reads `s.observersCache` before
SQL,
TTL boundary, atomic.Pointer (no mutex contention).
- `storeobs_parsedtime_test.go` — parses three timestamp shapes, caches
result, no race under concurrent readers.
- `neighbor_graph_cache_test.go` — handler serves from atomic pointer
when set, bypasses cache when `?region=` (or any non-default filter)
is passed.
Full server + ingestor suites pass: `go test -count=1 ./...`.
## Perf proof
Before/after p50/p95/p99 (50 requests × 50 concurrent) against prod
(before)
and staging once CI deploys (after) will be posted as a PR comment per
the
operator's "no merge without proof of improvement" gate.
Closes#1481
## TDD exemption — P0-1 and P0-2 (net-new surfaces, AGENTS.md)
Per CoreScope `AGENTS.md` § "Exemptions": **net-new code surfaces with
no
prior tests to break** may land tests in the same PR without a strict
test-first → impl commit split.
- **P0-1 (neighbor-graph atomic-pointer cache)** — `neighborGraphCache`,
`recomputeNeighborGraphCache`, `loadNeighborGraphCacheBytes`,
`startNeighborGraphRecomputer` and the default-shape short-circuit in
`handleNeighborGraph` were brand-new code with no pre-existing
assertions covering them. There was no green test to first turn red.
- **P0-2 (cached `StoreObs.Timestamp` + RLock window drop)** —
`StoreObs.ParsedTime()` and the snapshot+release pattern in
`handleObserverAnalytics` were new surfaces; the prior code did the
parse inline per call with no behavioural test to break.
P0-3 was authored properly red-then-green (commit `6e63ec6a` red, then
`83ae129b` green) and does NOT use this exemption.
## Default-filter detection vs frontend reality (#1483 follow-up)
The Neighbor Graph analytics tab in `public/analytics.js` fetches
`/analytics/neighbor-graph?min_count=1&min_score=0` because the
client-side sliders need the full edge set to filter from. That shape
did NOT match the `(5, 0.1)` cached default, so the UI tab still paid
the cold compute cost despite #1481 P0-1.
The #1483 follow-up commit caches BOTH shapes in the same recomputer
pass:
- `(minCount=5, minScore=0.1, no region, no role)` — `live.js`
affinity-scoring consumer.
- `(minCount=1, minScore=0, no region, no role)` — analytics tab.
Both are served from `atomic.Pointer` with an `X-Cache-Age-Seconds`
header. The per-shape cost in the background goroutine is roughly
linear in edge count; total recompute time stays well under the
5-minute cadence on prod-scale graphs.
---------
Co-authored-by: openclaw-bot <bot@openclaw.dev>
Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
