Files
meshcore-analyzer/cmd/server/index_ready_1008_test.go
T
n30nex cf67a5e5ec fix: remove evicted resolved path hops and preserve relay snapshots (#1966)
Eviction removes raw wire hops but leaves resolved full-key entries in
`byPathHop`, retaining expired transmissions and stale relay
counts/scopes. Filter every hop bucket once per eviction batch using the
existing evicted-ID set, remove duplicate references and empty buckets,
and clear discarded pointer slots.

Bulk relay aggregation now owns its bucket snapshots before releasing
the read lock, so eviction and raw-path updates cannot mutate an
in-flight reader. Three existing handler test fixtures also wait for
index readiness or explicitly simulate not-ready state, preserving their
original 200/503 assertions.

Fixes #1908.

Validation:

- Regression commits fail before their corresponding fixes: resolved
keys/counts/scopes remain after eviction, and saved relay snapshots
change during eviction.
- Targeted eviction, relay, scope, cache and concurrent-reader checks
pass under `-race`; coverage includes time/cap eviction, missing
resolved-path prefetch, disabled membership indexing, duplicate
references, retained backing arrays and surviving entries.
- Local browser smoke: nodes, node details/path attribution and
analytics render using the fixture-backed Go server.
- The last full Windows server race run, before the final snapshot-copy
correction, had one remaining DB-only timing failure
(`TestGetChannelMessagesPerfLargeChannel`: 2.198s against a 1.5s
budget). The final correction was checked with focused race tests. The
unchanged ingestor suite also cannot create one symlink without Windows
privileges. These thresholds/assertions were preserved; full Linux
Go/E2E results still require upstream CI approval.

Performance tradeoff: cleanup is O(total indexed pointers) per nonempty
eviction batch, under the existing write lock. The minute-based ticker
pays for one sweep instead of repeated scans of shared raw buckets. No
per-transmission string index or dependency is added. Synthetic
benchmark medians (three single-iteration runs, shared Windows host):

| Transmissions | Evicted | Before | After |
|---:|---:|---:|---:|
| 30,000 | 1 | 1.07 ms | 14.19 ms |
| 30,000 | 3,000 | 56.27 ms | 61.68 ms |
| 30,000 | 7,500 | 83.58 ms | 65.54 ms |
| 100,000 | 1 | 0.30 ms | 56.95 ms |
| 100,000 | 10,000 | 949.93 ms | 190.71 ms |
| 100,000 | 25,000 | 1,834.48 ms | 320.38 ms |

Fixture: eight raw plus eight resolved hops per transmission, two
observations, 2,048 relays; 480,000/1,600,000 hop entries. Timing
includes acquiring the store lock and omits unrelated secondary indexes.
Small batches now pay for the complete sweep; shared-host timing is
noisy.

Owning the bulk reader's arrays also has a measured cost on cold/bulk
recomputation, rather than cached hits. Snapshot medians from three
samples of ten iterations:

| Transmissions / relay nodes | Before time / bytes per operation |
After time / bytes per operation |
|---|---:|---:|
| 30,000 / 50 | 0.0068 ms / 5,416 B | 23.65 ms / 4,101,435 B |
| 30,000 / 2,000 | 0.1374 ms / 196,768 B | 26.77 ms / 4,274,336 B |
| 100,000 / 2,000 | 0.1376 ms / 196,768 B | 27.89 ms / 13,959,337 B |

These are total snapshot costs, comparing the unsafe header-only
snapshot with owned pointer arrays. Cleanup guarantees here apply to
`byPathHop`; other indexes and existing periodic bulk-cache freshness
are outside this change.

Following #1922, this runtime fix is separate from the release-routing
and frontend-runner PRs. Current Go and E2E job results should be
assessed separately from workflow-approval or staging-runner state.
2026-09-06 21:00:44 +02:00

149 lines
5.3 KiB
Go

// Issue #1008: subpath + pathHop index builds must move off the
// synchronous Load() critical path into a background goroutine.
//
// Contract:
// 1. Immediately after Load() returns, SubpathIndexReady() and
// PathHopIndexReady() report false (the goroutine has not finished).
// 2. Analytics handlers that depend on those indices respond 503 with
// Retry-After: 5 until the corresponding ready flag flips true.
// 3. After the background build completes (waitable via a helper),
// both flags flip true and handlers respond 200.
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
)
// TestIssue1008_SubpathIndexReadyFalseImmediatelyAfterLoad asserts the
// subpath ready flag is false the instant Load() returns. Red commit: the
// stub returns true → assertion fires. Green commit: the flag is owned by
// the background goroutine, which has not yet run, so the assertion holds.
func TestIssue1008_SubpathIndexReadyFalseImmediatelyAfterLoad(t *testing.T) {
db := setupRichTestDB(t)
defer db.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("Load() error: %v", err)
}
if store.SubpathIndexReady() {
t.Fatal("expected SubpathIndexReady()==false immediately after Load(); want background-deferred build (#1008)")
}
}
// TestIssue1008_PathHopIndexReadyFalseImmediatelyAfterLoad: same contract
// for the path-hop index.
func TestIssue1008_PathHopIndexReadyFalseImmediatelyAfterLoad(t *testing.T) {
db := setupRichTestDB(t)
defer db.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("Load() error: %v", err)
}
if store.PathHopIndexReady() {
t.Fatal("expected PathHopIndexReady()==false immediately after Load(); want background-deferred build (#1008)")
}
}
// TestIssue1008_HandlerReturns503WhileSubpathIndexLoading asserts the
// analytics/subpaths handler returns 503 + Retry-After: 5 + a JSON body
// matching the triage spec while the subpath index is still building.
func TestIssue1008_HandlerReturns503WhileSubpathIndexLoading(t *testing.T) {
db := setupRichTestDB(t)
defer db.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("Load() error: %v", err)
}
// Finish the background build before simulating not-ready, so it cannot
// flip the flag back to true before the handler checks it.
if !store.WaitIndexesReady(5 * time.Second) {
t.Fatal("background builds never finished")
}
store.subpathReady.Store(false)
cfg := &Config{}
cfg.applyListLimitsDefaults()
srv := &Server{store: store, cfg: cfg}
req := httptest.NewRequest("GET", "/api/analytics/subpaths?minLen=2&maxLen=4&limit=10", nil)
rec := httptest.NewRecorder()
srv.handleAnalyticsSubpaths(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status = %d, want 503 (subpath index loading, #1008)", rec.Code)
}
if got := rec.Header().Get("Retry-After"); got != "5" {
t.Errorf("Retry-After header = %q, want %q", got, "5")
}
var body map[string]interface{}
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("body not valid JSON: %v (body=%s)", err, rec.Body.String())
}
if body["error"] != "index loading" {
t.Errorf(`body["error"] = %v, want "index loading"`, body["error"])
}
}
// TestIssue1008_HandlerRecoversAfterIndexReady asserts that, once the
// background build completes, the handler returns 200.
func TestIssue1008_HandlerRecoversAfterIndexReady(t *testing.T) {
db := setupRichTestDB(t)
defer db.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("Load() error: %v", err)
}
// Wait up to 5s for both background builds to finish on this small
// fixture (rich test DB has ~3 packets; build is sub-millisecond).
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if store.SubpathIndexReady() && store.PathHopIndexReady() {
break
}
time.Sleep(10 * time.Millisecond)
}
if !store.SubpathIndexReady() {
t.Fatal("SubpathIndexReady() never flipped true within 5s")
}
if !store.PathHopIndexReady() {
t.Fatal("PathHopIndexReady() never flipped true within 5s")
}
cfg := &Config{}
cfg.applyListLimitsDefaults()
srv := &Server{store: store, cfg: cfg}
req := httptest.NewRequest("GET", "/api/analytics/subpaths?minLen=2&maxLen=4&limit=10", nil)
rec := httptest.NewRecorder()
srv.handleAnalyticsSubpaths(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status after ready = %d, want 200 (body=%s)", rec.Code, rec.Body.String())
}
}
// TestIssue1008_m7_BothFlagsSetAfterParallelStart verifies that the
// parallel two-goroutine version of startBackgroundIndexBuilds (review
// m7) sets BOTH ready flags after a bounded wait, regardless of which
// goroutine wins the race to s.mu.Lock(). Sanity check that breaking
// the two builds apart didn't drop the pathHop flag flip.
func TestIssue1008_m7_BothFlagsSetAfterParallelStart(t *testing.T) {
db := setupRichTestDB(t)
defer db.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("Load: %v", err)
}
if !store.WaitIndexesReady(5 * time.Second) {
t.Fatal("indexes never ready after parallel start (#1008 m7)")
}
if !store.SubpathIndexReady() {
t.Error("subpath flag not set after WaitIndexesReady returned true")
}
if !store.PathHopIndexReady() {
t.Error("pathHop flag not set after WaitIndexesReady returned true")
}
}