Fixes#1740.
## What
Replace the full `idx_tx_last_seen` with a partial index `WHERE
last_seen=0`. Two ordered, sequential migrations in
`internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn`:
- **(a)** `CREATE INDEX IF NOT EXISTS idx_tx_last_seen_zero ON
transmissions(id) WHERE last_seen=0`
- **(b)** `DROP INDEX IF EXISTS idx_tx_last_seen` — gated after (a)
succeeds (sequential `Exec`; (b) never runs if (a) errors)
## Why
The only consumer is `chunkedTxLastSeenBackfill`'s `WHERE last_seen=0`
scan + `MAX(id)` lookup. The full index covers ALL rows including the
long tail where `last_seen != 0` after backfill converges (71K+ rows in
prod per carmack's #1740 note). The partial index degenerates to ~the
count of un-backfilled rows (0 in steady state, bounded by ingest rate
during ops) and stops competing for page cache.
## Migration cost
Both migrations are sync and annotated `PREFLIGHT: async=false`:
- (a) `CREATE INDEX` on partial subset `WHERE last_seen=0` is bounded by
un-backfilled rows — a small superset of the inflight ingest window, not
a full table scan.
- (b) `DROP INDEX` is a metadata-only schema rewrite in SQLite — no row
scan at any size.
`dbschema/` has no access to `Store.RunAsyncMigration` (that helper
lives in `cmd/ingestor/` and is the wrong layer for the schema
source-of-truth per #1321), so sync is the only path here regardless.
## TDD
- **RED** `a529f0f4`: `EXPLAIN QUERY PLAN` test asserting the backfill
`MAX(id) WHERE last_seen=0` query uses `idx_tx_last_seen_zero` + a
second test asserting the legacy `idx_tx_last_seen` is dropped
post-Apply. Both failed on assertion (planner picked the full index;
partial index didn't exist).
- **GREEN** `208fde8a`: add migrations (a) and (b). Both tests pass.
## Files touched
- `internal/dbschema/dbschema.go` — swap the index
- `internal/dbschema/dbschema_test.go` — `EXPLAIN QUERY PLAN` pin + DROP
assertion
- `cmd/ingestor/db.go` — comment refresh only (idx_tx_last_seen →
idx_tx_last_seen_zero)
## Acceptance
- ✅ New partial index created
- ✅ Old full index dropped via gated migration (sequential, order
preserved)
- ✅ Query plan test asserts partial-index usage
- ✅ Existing migration tests still green (`internal/dbschema`,
`cmd/ingestor` TestIssue1690 + applySchema)
---------
Co-authored-by: clawbot <bot@openclaw>
Red commit `2a8102b9` (failing test) → green commit `bb957c9f`. CI:
https://github.com/Kpa-clawbot/CoreScope/actions/workflows/ci.yml?query=branch%3Afix%2Fissue-1321Fixes#1321.
## Why
On staging `/api/scope-stats` 500'd with `scope_name column not present`
despite the ingestor adding the column ~0.5s after server startup.
`cmd/server/db.go detectSchema()` runs in `OpenDB` and caches
`hasScopeName`/`hasDefaultScope`/`hasObsRawHex` booleans. With
supervisord launching server + ingestor simultaneously, the server's
PRAGMA can fire BEFORE the ingestor's `ALTER TABLE` completes — and the
boolean stays false until the server restarts. Same race class as #1283;
#1289 moved server-side ensures to `dbschema` but the optional columns
the ingestor still owned were left out.
## Fix — option (c) from the issue
Made `internal/dbschema/dbschema.go` the single source of truth for the
optional columns the server detects.
**Migrations moved from `cmd/ingestor/db.go applySchema` into
`dbschema.Apply`:**
- `transmissions.scope_name` + `idx_tx_scope_name` partial index
- `nodes.default_scope`
- `inactive_nodes.default_scope`
- `observations.raw_hex`
**`AssertReady` now asserts** every one of those columns. The server
cannot start with stale-false booleans because `AssertReady` will fatal
first if the columns are missing. The ingestor's old gated blocks are
replaced with pointer comments so anyone hunting for them lands in
`dbschema.go`. The `_migrations` marker rows are preserved (`INSERT OR
IGNORE`) to keep legacy DBs idempotent.
**Documented invariant** in the package doc: any new optional column the
server PRAGMA-detects belongs in `internal/dbschema/dbschema.go`, NOT in
`cmd/ingestor/db.go applySchema`.
## Tests
Added `internal/dbschema/dbschema_test.go` (RED in `2a8102b9`):
- `TestApplyAddsOptionalColumns_CanonicalSource` — post-`Apply`, all
four columns must exist.
- `TestAssertReady_RequiresOptionalColumns` — `AssertReady` must refuse
a DB missing them AND pass after full `Apply`.
`cmd/ingestor` and `cmd/server` full suites green.
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
