2 Commits

Include renames

Author	SHA1	Message	Date
4 people	fa15ab0a30	fix(#1809): gate background loader on LoadChunked completion (#1811) ... Partial fix for #1809. Red commit: c9c782b5 (CI runs on PR open; standalone red-branch CI not configured — local repro proves the gate, see below). ## Problem Issue #1809: at startup the background fill loader logged `background load FAILED` within seconds and `backgroundLoadFailed=true` was set, leaving the coverage gate tripped even though LoadChunked itself completed normally. ## Root cause `main.go:225-245` spawned `go store.loadBackgroundChunks()` as soon as `FirstChunkReady` fired (chunk #1 = 10000 tx). But `s.oldestLoaded` is only assigned at the end of `LoadChunked` (`chunked_load.go:329-333`), ~tens of seconds later. The bg loader read `oldestLoaded==""` at `store.go:1462-1466`, broke out immediately, walked zero chunks, and the coverage gate at `store.go:1543-1554` flipped `backgroundLoadFailed=true`. ## Fix (initial) Introduce `PacketStore.RunStartupLoad(chunkSize)` (`chunked_load.go`). It runs `LoadChunked` first; only on success and only when `hotStartupHours > 0` does it call `loadBackgroundChunks`. `main.go` invokes `RunStartupLoad` in the same goroutine pattern as before, so `FirstChunkReady` still unblocks the HTTP listener bind at chunk #1 — only the bg loader is gated. ## Round-1 followups (this push) Reviewer-driven hardening on top of the initial fix: ### Production behavior (commit db5592f6) - **Steady-state semantics tightened.** `RunStartupLoad` now picks a terminal state on every branch: - LoadChunked error → `backgroundLoadFailed=true` with captured error (was: `done=false, failed=false` indefinite). - `hotStartupHours == 0` → `backgroundLoadDone=true` immediately, `progress=100` (was: `done=false` forever → healthz stuck on `backgroundLoadComplete=false`). - Successful hot-window path → terminal state is whatever `loadBackgroundChunks` sets (#1690 semantics, unchanged). - **Runtime invariant assertion (A7).** `loadBackgroundChunks` panics when `oldestLoaded==""` and packets exist — a future refactor that re-introduces the parallel-spawn race fails loudly instead of silently shipping the same coverage regression. - **`RunStartupLoad` cleanup.** Inlined the superfluous goroutine + channel that wrapped `LoadChunked` (direct call is equivalent). - **Logging.** Added an INFO line between `LoadChunked` completion and bg-loader start (the #1809 post-mortem needed exactly this signal). Fixed the lying `"background load will start"` log that fired even on the `hotStartupHours==0` branch. - **Immutability documented.** `hotStartupHours` is now explicitly documented as immutable post-construction, so the lock-free reads in `LoadChunked` / `RunStartupLoad` / `loadBackgroundChunks` are sound. ### Test coverage (commits db5592f6 + e9e12acf) - **Tautology fix (B1, commit e9e12acf).** The original `Test1809_StartupLoad_BgLoaderSeesOldestLoaded` fixture seeded all 100 rows inside the 1h hot window, so `LoadChunked` alone produced coverage=1.0 — the test passed even if `loadBackgroundChunks` was a no-op. Rewrote the fixture to spread 100 rows over 14 days with `hotStartupHours=24`, so only ~7 rows are hot and the remaining ~93 MUST be loaded by the bg loader for the assertions to hold. Original red-commit assertions kept intact; added `len(packets) > hot-only cap` and `oldestLoaded < hot-cutoff - 12h` assertions on top. - **New tests (commit db5592f6, `runstartup_load_test.go`)** codify the new contracts: - `TestRunStartupLoad_HotStartupHoursZero_SetsDoneImmediately` - `TestRunStartupLoad_LoadChunkedError_SetsFailedTerminal` - `TestRunStartupLoad_EmptyDB_SetsDoneTerminal` - `TestRunStartupLoad_BgLoaderRunsAfterLoadChunkedSets_OldestLoaded` - `TestLoadBackgroundChunks_PanicsOnOldestLoadedEmpty_Invariant` ### Docs (commit 70fa16f7) - Package-level doc in `chunked_load.go` now documents `RunStartupLoad` as the orchestrator entry point alongside `LoadChunked` / `loadStatusMiddleware` / `OnChunkLoaded`. ### Preflight (commit eec1b48c) - Test-fixture DDL annotated with `// PREFLIGHT: async=true reason="unit-test fixture"` so the async-migration gate distinguishes ephemeral test schema from prod migration paths. ## What's still NOT fixed (left intentionally open) This PR addresses the startup race specifically. Issue #1809 will be closed by the operator after observing healthy startup logs (`background load complete: ... coverage=100.0%`) in prod for a full restart cycle. Do not auto-close — leave open until the operator verifies. ## Test Local repro (red branch state, before green commit): `go test ./cmd/server/ -run Test1809_StartupLoad_BgLoaderSeesOldestLoaded` → FAIL on assertion `backgroundLoadFailed=true ... oldest="2026-06-30T18:56:09Z"`. After green commit + round-1 followups: PASS. Full `cmd/server/...` suite: ok in ~70s. ## Risk Low — startup path only. New behavior gates surfaced by the new tests; coverage-gate semantics unchanged. Runtime panic is a new failure mode but only fires on a state (`oldestLoaded=="" && len(packets)>0`) that is unreachable on the current code path — it exists solely as a refactor tripwire. --------- Co-authored-by: mc-bot <bot@corescope.local> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: meshcore-bot <bot@meshcore>	2026-06-30 17:19:56 -07:00
3 people	048143f54f	fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691) ... ## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com>	2026-06-12 12:47:53 -07:00