2 Commits

Include renames

Author	SHA1	Message	Date
3 people	40f664c587	chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941) ... Rebase of #1881 by @SaarMesh-Bot onto current master. Their three commits are preserved, two of them cherry-picked with authorship intact; the sweep itself had to be regenerated. Opened as a new PR rather than force-pushing their branch. Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2 landed as #1937. ## Why regenerated rather than merged The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on current master is cheaper and less error-prone than resolving 72 conflicts that are all whitespace. The drift it fixes also grew in the meantime: 66 files now, against 72 then, but spread differently. ## The three commits 1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files. 2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet` copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range variable copied a `Config` embedding `sync.Once`. Cherry-picked unchanged. 3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one change, noted in the commit message: the ignore file pointed at `04bc80ee`, the sweep commit on their branch, which does not exist on this base and would make `git blame --ignore-revs-file` error. Repointed at `d3a02599`, the sweep here. ## Verification The claim "formatting only" is checked twice rather than asserted: - Every changed file is byte-identical to `gofmt(previous content)`. 0 of 66 deviate. - With line comments and all whitespace stripped, 0 of 66 files differ, so no code outside comments changed. 14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt` re-indents indented comment blocks to tabs and inserts a blank comment line before them; the behavior matrix above `resolveHopWithContext` in `cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own output, not an edit, but it is worth naming because it makes the diff look larger than "whitespace" suggests. The gate was run locally exactly as the workflow runs it: `gofmt` clean, and `go vet` clean in all 14 modules, including `cmd/ingestor` which is what commit 2 fixes. Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s), `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically on bare master with "A required privilege is not held by the client" (Windows symlink privilege on my host, not code). ## Sequencing This should go last in the queue. The sweep touches 66 files, so merging it before the remaining open Go PRs gives each of them a conflict about nothing but formatting. After it lands the gate is active, and any PR with drift fails CI until it runs `gofmt -w`. Excluded from the sweep: the misnamed `Dockerfile.go`, which is a Dockerfile that gofmt cannot parse (the workflow excludes it too), and `docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a spurious modification against `docs/deployment.md` and is unrelated. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 18:52:03 +02:00
3 people	e438451dc9	feat(preflight): hard-fail gate on sync schema migrations + async runner (#1541) ... Closes the recurring "sync migration on large table" regression class (#791-style, #1483-style). ## Problem Pattern that keeps repeating: 1. A perf/feature PR adds `CREATE INDEX` / `ALTER TABLE` / `UPDATE ... WHERE` in a migration file (typically `cmd/ingestor/db.go`). 2. Local dev DB has ~100 rows. Migration returns in milliseconds. CI is green. 3. Reviewers approve on plan correctness; nobody knows what the prod table size is. 4. First prod boot at scale (Cascadia: ~2600 nodes, 80K+ obs; previous prod: 1.9M+ obs) pins the ingestor at `[migration] Adding index...` for minutes. 5. Healthcheck times out → container restart → loop. Operator pages. Hotfix. Most recent case: `obs_observer_ts_idx_v1` in v3.8.3 — release notes already document an "expect a longer first boot" warning because we knew it would hit prod hard. ## What this PR adds **Async helper (`cmd/ingestor/async_migration.go`):** - `Store.RunAsyncMigration(ctx, name, fn)` — registers the migration as `pending_async` in a new `_async_migrations` bookkeeping table, returns to caller immediately, schedules `fn` in a goroutine on the shared backfill `WaitGroup`, transitions to `done` (or `failed` with error captured) on completion. - `Store.AsyncMigrationStatus(name)` and `Store.WaitForAsyncMigrations()` for tests/shutdown. - Idempotent: `done` rows short-circuit; `pending_async`/`failed` rows are retried on next boot. **Retroactive #1483 conversion (`cmd/ingestor/db.go`):** - `obs_observer_ts_idx_v1` (the composite `(observer_idx, timestamp)` index build on `observations`) is now scheduled via `RunAsyncMigration` from `OpenStore()` so the ingestor accepts packets immediately while the index builds in the background. - Legacy `_migrations` gate is preserved by the async fn → DBs that already completed the sync build stay no-op. **Annotation convention (`MIGRATIONS.md`):** Every new `CREATE INDEX` / `ALTER TABLE` / data-rewrite in a migration file must do ONE of: 1. Run via `Store.RunAsyncMigration(...)` (preferred for backfills). 2. Carry a `// PREFLIGHT: async=true reason="..."` comment directly above the migration block. 3. Include a `PREFLIGHT-MIGRATION-SCALE: <30s N=<scale>` line in the PR body. **TDD pair:** - Red commit `2c6744cc` — `TestRunAsyncMigration_PendingThenDone` against a stub helper. Build passes, assertion fails (`async migration fn did not start within 2s`). - Green commit `38354f32` — real helper + retroactive fix + docs. Test green. **Fixtures (`cmd/ingestor/testdata/preflight-migrations/`):** - `bad_sync_migration.go` — known-bad sample with no annotation. - `good_annotated_migration.go` — known-good sample with annotation. The preflight gate script can be unit-tested against these. ## Gate location (NOT in this PR) The actual `check-async-migrations.sh` lives in the OpenClaw skills directory at `~/.openclaw/skills/pr-preflight/scripts/` (separate from the repo) and is wired into `run-all.sh`. It greps the diff for new/modified migration blocks and hard-fails (exit 1) on any sync schema mutation lacking one of the three opt-outs above. The fixtures in this PR give maintainers a reproducible target. ## Why annotation-discipline, not size detection You cannot determine table size from a diff. The gate enforces that every author who adds a schema migration must consciously decide which bucket it falls into and write that down. That is the cheapest possible intervention that breaks the cycle. ## Testing - `go test ./...` in `cmd/ingestor` — all tests pass including the new `TestRunAsyncMigration_PendingThenDone`. - Manual: red commit fails on assertion (not build), green commit passes — verifiable by `git checkout 2c6744cc -- cmd/ingestor/async_migration.go && go test -run TestRunAsync ./cmd/ingestor` from the green commit. ## Preflight overrides None — clean run after the convention is applied. --------- Co-authored-by: clawbot <bot@openclaw.local> Co-authored-by: clawbot <bot@openclaw>	2026-06-03 21:03:59 +00:00