mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-19 04:36:21 +00:00
openclaw-bot
915b10119f
Replaces the single correlated UPDATE used by tx_last_seen_backfill_v1 (introduced in #1690) with a chunked loop that yields the single SQLite writer between batches. Symptom (pre-fix, operator scale ~71K tx / 1.5M obs / 2GB DB): - backgroundLoadComplete=true fires. - The async migration starts the single full-table UPDATE under SetMaxOpenConns(1), holds the writer for 10-15 minutes. - Every /api/healthz, /api/packets, /api/stats request queues behind sqlite_busy_timeout. UI appears frozen long after warm-up clears. Fix (this commit): - cmd/ingestor/tx_last_seen_backfill.go (new): chunkedTxLastSeenBackfill snapshots MAX(id), counts eligible rows (last_seen=0 AND has observations AND id<=maxID), then loops bounded UPDATEs (batchSize=5000) with time.NewTimer-based sleeps (no Timer leak via time.After) between batches (yieldDelay=100ms). EXISTS gate skips orphan transmissions so the loop terminates. maxID snapshot keeps concurrent INSERTs out of scope (those are handled inline by stmtBumpTxLastSeen on the writer fast path). Ctx cancellation between batches returns context.Canceled with partial counts; partial commits are visible (migration does NOT flip to done). All errors propagate (snapshot, count, UPDATE, RowsAffected) — the migration cannot silently mark itself done. Progress callback fires per non-empty batch + once terminal with final stable counts; never on a stale n=0 batch. - cmd/ingestor/db.go: wire the helper into the tx_last_seen_backfill_v1 async migration, explicit batchSize=5000, yieldDelay=100ms. Math reality-check: ~71K tx / 5000 ≈ 15 batches × (~50ms exec + 100ms yield) ≈ ~2.5s wall time with readers slotted in at most every 150ms. PR #1725's description claimed ~300 batches × 150ms ≈ 45s — that confused observations (1.5M) with transmissions (71K); real number is ~20x smaller. Indexes idx_tx_last_seen (transmissions(last_seen)) and idx_observations_transmission_id already exist (see internal/dbschema and cmd/ingestor/db.go base schema) — no additional index work required at this commit. Tests: cmd/ingestor/tx_last_seen_backfill_test.go (added in prior commit) pin all the contract points reviewers flagged on PR #1725. Cancel-mid-loop test timing widened from 30ms to 250ms to give the real chunked impl room to commit a batch before the cancel fires; assertion semantics unchanged (partial commits + context.Canceled + no full completion).