Files
meshcore-analyzer/cmd/server/repeater_enrich_recomputer.go
T
df61660a5e perf(load): background subpath+pathHop index builds with ready gates (#1008) (#1604)
## Summary

Mirrors the distance-index lazy pattern (#1011): the subpath and
path-hop index builds are no longer part of `Load()`'s synchronous
critical section. They now run in **two parallel background goroutines**
kicked off after `s.loaded = true`, so HTTP comes up immediately even at
Cascadia scale (5M observations, previously ~60s blocked on these two
builds inside `Load()` under `s.mu`).

Fixes #1008.

## Approach

Two new `atomic.Bool` fields on `PacketStore` (`subpathReady`,
`pathHopReady`) plus a one-shot broadcast channel (`indexReadyChan`) for
waiters. `Load()` removes the synchronous `s.buildSubpathIndex()` /
`s.buildPathHopIndex()` calls and instead kicks
`s.startBackgroundIndexBuilds()` right before returning. That function
spawns **two independent goroutines** (review m7), one per index. Each
goroutine:

1. acquires `s.mu.Lock()` (blocks until `Load()`'s deferred Unlock
fires),
2. runs its builder, releases the lock, stores its `ready = true`,
3. closes the broadcast channel if both flags are now true,
4. logs `[startup] index build complete: subpath (Xs)` (or pathHop).

Analytics handlers whose entire response IS the index aggregate —
`/api/analytics/subpaths`, `/api/analytics/subpaths-bulk`,
`/api/analytics/subpath-detail`, `/api/nodes/{pubkey}/paths` — gate
reads behind the corresponding atomic and respond with `503 Service
Unavailable`, `Retry-After: 5`, body `{"error":"index
loading","retryAfter":5}` until the build completes — matching the
triage spec.

### Handler scope (review M2)

A second class of handlers also touches these indexes — `/api/nodes`,
`/api/nodes/{pubkey}`, the `GetRepeaterRelayInfoMap` /
`GetRepeaterUsefulnessScoreMap` / `GetBridgeScore` enrichment helpers,
and `repeater_liveness` / `repeater_usefulness`. These are
**intentionally NOT 503-gated**: they expose the index via optional
enrichment fields that callers already treat as "may be empty", and
503-ing the SPA bootstrap to wait for an index that only affects
relay-activity badges would be a worse UX than a 30–60s window of "—"
values. The rationale is documented in the package doc-comment at the
top of `index_ready_1008.go`.

The recomputer's synchronous prewarm path
(`StartRepeaterEnrichmentRecomputer`) gates on `WaitIndexesReady(60s)`
(review M1) so it never snapshots an empty `byPathHop` into
`s.repeaterRelayCache`; on timeout it skips the prewarm and lets the
5-minute ticker pick up the populated index.

## Concurrency safety

Each build goroutine acquires `s.mu.Lock()` before calling the existing
`buildSubpathIndex()` / `buildPathHopIndex()` helpers, which replace
`s.spIndex` / `s.spTxIndex` / `s.byPathHop` with freshly-allocated maps.
Visibility of the populated maps to handlers that observe
`Ready()==true` is established by Go 1.19+ sync/atomic acquire-release
semantics: the atomic store of `true` happens-after `s.mu.Unlock()`, and
the handler's atomic load synchronizes-with that store. The handler's
subsequent `s.mu.RLock` serializes against concurrent ingest writers,
not against the builder.

The existing `main.go` boot sequence does not start ingest goroutines
until after `store.Load()` returns and graph init completes, so the
brief window between `Load()` returning and the two goroutines acquiring
`s.mu` does not race with concurrent ingest writes.

## TDD: red → green

- **Red** commit `63e79e11`: `cmd/server/index_ready_1008_test.go` adds
four assertions; `cmd/server/index_ready_1008.go` adds compile-only
stubs returning `true` so the tests fail on assertions, not build
errors.
- **Green** commit `fb1d22b0`: implements the real atomic gates, the
background goroutine, and the four handler 503 branches; also updates
four existing tests that read indexes directly post-`Load()` to call
`store.WaitIndexesReady(5s)` first.
- **Race-fix commit `b77d56eb`** (review m8 — test-infra exemption):
adds `WaitIndexesReady` calls in test helpers/setup paths so the race
detector no longer flags the read-after-Load() pattern in existing
tests. Per AGENTS.md, race-detector flakes are observable evidence (test
crashes under `-race`) and qualify for the test-infra exemption from the
TDD red-commit requirement; no behavior change in production code.
- **Polish round 2 — M1 red `408c7462` / green `85e82c8a`**:
`TestIssue1008_M1_PrewarmWaitsForIndexes` asserts the recomputer prewarm
SKIPs when indexes are not ready. Red commit adds the assertion + a stub
`repeaterEnrichmentPrewarmWait` var; green commit wires
`WaitIndexesReady` into the prewarm path and adds the handler-scope docs
for M2.
- **Polish round 2 — minor cleanups `fd089bd0`** (m3..m7): chunk-loader
wires `markIndexesReadySync`, memory-model comment rewritten to cite
acquire-release, sentinel deleted, polling replaced with a broadcast
channel, two parallel goroutines for the builds.
`TestIssue1008_m7_BothFlagsSetAfterParallelStart` covers the parallel
path.

## Reproduction

```
git fetch origin fix/issue-1008
git checkout 63e79e11   # red commit
cd cmd/server && go test -run TestIssue1008_ -count=1 .   # FAILs

git checkout fix/issue-1008   # latest green
cd cmd/server && go test -run TestIssue1008 -count=1 -race .   # all pass
cd cmd/server && go test -count=1 -race -short ./...           # full suite ok
```

## Files changed

| file | role |
|---|---|
| `cmd/server/store.go` | atomic.Bool fields + indexReadyChan broadcast
field; remove sync build calls in Load(); kick goroutines; wire
markIndexesReadySync from chunk loader |
| `cmd/server/index_ready_1008.go` | ready flags, two-goroutine
background builds, 503 helper, channel-based WaitIndexesReady,
handler-scope docs |
| `cmd/server/index_ready_1008_test.go` | red-commit contract tests +
parallel-start assertion |
| `cmd/server/repeater_enrich_recomputer.go` | gate prewarm on
WaitIndexesReady (M1) |
| `cmd/server/repeater_enrich_recomputer_1008_test.go` | M1 red+green
assertions |
| `cmd/server/routes.go` | 503 gate on 4 analytics handlers |
| `cmd/server/routes_test.go` | setup helpers wait for ready; collision
test waits |
| `cmd/server/coverage_test.go` | three tests wait for ready before
reading indexes |

## Out of scope

- Distance index (already deferred in #1011) — untouched.
- The `pickBestObservation` + `indexByNode` per-tx loop in `Load()` —
kept synchronous per triage Findings (ordering-sensitive,
contiguous-memory, fast).

---------

Co-authored-by: bot <bot@noreply.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
2026-06-06 20:46:42 -07:00

127 lines
4.7 KiB
Go

package main
import (
"sync"
"time"
)
// repeaterEnrichmentRecomputerInterval is the default tick interval
// for the steady-state recompute of the repeater enrichment bulk
// caches. The on-request TTL fallback in repeater_enrich_bulk.go is
// kept as a safety net — the recomputer just makes sure the cache is
// populated before any request arrives.
//
// 5min mirrors the analytics_recomputer default from #1240 and is
// plenty fresh for an at-a-glance status column.
const repeaterEnrichmentRecomputerDefaultInterval = 5 * time.Minute
// repeaterEnrichmentPrewarmWait is the upper bound on how long the
// synchronous prewarm in StartRepeaterEnrichmentRecomputer will wait
// for the background subpath+pathHop index builds to flip ready before
// skipping the prewarm. Override in tests via the package-level var.
//
// Background (issue #1008 review M1): the prewarm computes against
// s.byPathHop. If the background index builds haven't finished, the
// snapshot is built against an empty map and locked into
// s.repeaterRelayCache for `interval` (default 5min) — every
// /api/nodes during that window would report relay_count_24h=0. We
// wait up to this deadline and, on timeout, skip the prewarm entirely
// so the next ticker fire (which will see ready=true) does the work.
var repeaterEnrichmentPrewarmWait = 60 * time.Second
// StartRepeaterEnrichmentRecomputer is the steady-state background
// recompute loop for the repeater enrichment bulk caches consumed by
// handleNodes (GetRepeaterRelayInfoMap + GetRepeaterUsefulnessScoreMap).
//
// Why this exists (issue #1262): PR #1260 added a 15s-TTL bulk cache,
// but the rebuild itself runs on the request-serving goroutine on the
// first request after startup or after the TTL expires. On staging
// (75k tx, 600 nodes) that cold rebuild took 15.7s and was triggered
// by every cold SPA load via live.js's /api/nodes?limit=2000 call.
//
// On Start this does an initial synchronous compute (so the next
// request hits cache) and then ticks every `interval` to keep the
// snapshot fresh — same pattern as analytics_recomputer.go (#1240).
//
// Returns a stop closure that signals the goroutine and waits for it
// to exit (with a 5s defensive timeout).
//
// Safe to call multiple times: subsequent calls are no-ops and return
// a no-op stop closure (the original goroutine retains ownership).
func (s *PacketStore) StartRepeaterEnrichmentRecomputer(windowHours float64, interval time.Duration) func() {
if interval <= 0 {
interval = repeaterEnrichmentRecomputerDefaultInterval
}
s.repeaterEnrichRecompMu.Lock()
if s.repeaterEnrichRecompStarted {
s.repeaterEnrichRecompMu.Unlock()
return func() {}
}
s.repeaterEnrichRecompStarted = true
stop := make(chan struct{})
done := make(chan struct{})
s.repeaterEnrichRecompStop = stop
s.repeaterEnrichRecompDone = done
s.repeaterEnrichRecompMu.Unlock()
// Initial synchronous prewarm — the entire point of this recomputer
// is to make sure the very first /api/nodes?limit=2000 from
// live.js's SPA bootstrap (issue #1262) hits a populated cache
// instead of paying the on-thread rebuild cost.
//
// Issue #1008 review M1: skip the prewarm if the background
// subpath+pathHop index builds haven't finished — otherwise we'd
// snapshot against an empty s.byPathHop and serve relay_count_24h=0
// for the entire `interval` window. The next ticker fire will pick
// up the populated index.
if s.WaitIndexesReady(repeaterEnrichmentPrewarmWait) {
recomputeRepeaterEnrichmentSafe(s, windowHours)
}
var stopOnce sync.Once
go func() {
defer close(done)
t := time.NewTicker(interval)
defer t.Stop()
for {
select {
case <-t.C:
recomputeRepeaterEnrichmentSafe(s, windowHours)
case <-stop:
return
}
}
}()
return func() {
stopOnce.Do(func() {
close(stop)
})
select {
case <-done:
case <-time.After(5 * time.Second):
}
}
}
// recomputeRepeaterEnrichmentSafe runs both bulk-cache compute paths
// behind a panic recover — a panic in compute must not kill the
// background goroutine (the previous snapshot remains valid).
func recomputeRepeaterEnrichmentSafe(s *PacketStore, windowHours float64) {
defer func() { _ = recover() }()
// Write directly to the cache fields under mutex rather than going
// through the public Get* helpers — those return the existing
// non-nil cache immediately, so calling them here would be a no-op.
relay := s.computeRepeaterRelayInfoMap(windowHours)
useful := s.computeRepeaterUsefulnessScoreMap()
now := time.Now()
s.repeaterEnrichMu.Lock()
s.repeaterRelayCache = relay
s.repeaterRelayCacheWin = windowHours
s.repeaterRelayAt = now
s.repeaterUsefulCache = useful
s.repeaterUsefulAt = now
s.repeaterEnrichMu.Unlock()
}