mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-03 22:31:43 +00:00
7e7be5efea
Replace the single full-table correlated UPDATE that pinned the SQLite
writer 10-15 min on prod-sized DBs (1.5M obs) with a bounded LIMIT-N
loop (5000 rows / 100ms sleep) that releases the writer between batches.
Reader p95 on /api/stats /api/healthz /api/packets recovers from
catastrophic (213s / 51s / 60s) to <500ms during the backfill window.
Changes:
- cmd/ingestor/tx_last_seen_backfill.go: chunked backfill helper with
configurable batch size + yield delay + per-batch progress callback;
bounds the WHERE clause by max(id) snapshot so concurrent INSERTs
don't keep the loop alive past shutdown.
- cmd/ingestor/db.go: register the v1 migration with the chunked
helper + a progress callback that streams snapshots to the new
_async_migrations columns.
- cmd/ingestor/async_migration.go: additive rows_processed /
rows_total / last_update_at columns on _async_migrations + a
Store.recordAsyncMigrationProgress writer.
- cmd/server/async_migrations.go (NEW): read-only DB reader exposing
AsyncMigrationInfo (status, rate, etaSeconds) to /api/perf and
/api/healthz.
- cmd/server/routes.go: include asyncMigrations in /api/perf.
- cmd/server/healthz.go: surface async_migrations + the
async_migrations_running flag so the warm-up banner stays up
while a migration is in flight.
- cmd/server/types.go: PerfResponse.AsyncMigrations field.
- public/warmup-banner.js: keep the banner up while
async_migrations_running=true and render a per-migration progress
line.
TDD: tx_last_seen_backfill_test.go::TestIssue1724_TxLastSeenBackfillIsChunked
asserts the loop emits ≥2 progress events and each per-batch delta is
bounded by batchSize. RED commit (716730f7) ran the original
single-shot UPDATE and failed both assertions; this commit makes them
pass.