mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 03:17:04 +00:00
master
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
900bffb782 |
refactor(schema): give observers an explicit idx primary key (#2407)
## Summary `observations.observer_idx` points at the **implicit rowid** of `observers`, whose declared key is `id TEXT`. Implicit rowids are not stable: - a dump-and-reload (`sqlite3 .dump`, `iterdump`) renumbers them, which silently re-links observations to the wrong observer; - PostgreSQL has no rowid at all. This change makes the key explicit. Split out of #2396 at review's request, so SQLite installs get it independently of the Postgres work. ## What changes - **One-time rebuild, in the ingestor.** On first start, `observers` is rebuilt as `idx INTEGER PRIMARY KEY AUTOINCREMENT`, copying every rowid into `idx` in one transaction. It fails closed on unknown columns or NULL ids. - **Rollback-safe.** In SQLite an `INTEGER PRIMARY KEY` *is* the rowid, so `rowid == idx` forever. Older binaries that still join on `observers.rowid` keep working, so both upgrade and rollback are safe. - **No inherited orphans.** `AUTOINCREMENT` is seeded past `MAX(observations.observer_idx)`, so a new observer can never inherit a deleted observer's leftover rows. - Fresh installs create the new shape directly, and `AssertReady` requires `observers.idx` as the primary key (a plain column named `idx` does not pass). - **Observer writes update first.** With `AUTOINCREMENT`, SQLite advances `sqlite_sequence` for every INSERT it attempts, even one `ON CONFLICT` turns into an update. The per-packet observer upsert and the clock-skew upsert now run an `UPDATE` and fall back to the unchanged `INSERT ... ON CONFLICT` only for a new observer. So the sequence moves only when an observer is added, and the next new observer gets `max(idx) + 1`. - `packets_v` now has a single definition (`dbschema.EnsurePacketsView`). - Every server and ingestor query, the fixture capture script and the test schemas use `idx`. Guard tests fail if `observers.rowid` comes back. ## Operator impact Every SQLite install gets this on its next ingestor start, with no config change. The rebuild is one transaction over `observers`, which holds one row per observer (typically hundreds), so it's quick. **When the rebuild refuses.** It refuses rather than lose data in two cases, and then the ingestor stops on every start until the cause is fixed: - `observers` has a column CoreScope doesn't know (one added locally). The rebuild would drop it. Save its data, then drop or rename it: `ALTER TABLE observers DROP COLUMN <name>`. - A row has a NULL `id` (the legacy `TEXT PRIMARY KEY` allows one). Find it with `SELECT rowid, * FROM observers WHERE id IS NULL`, then delete or fix it. The error message names the column or the row count and these steps. Starting the previous version also works: nothing was changed, and `packets_v` stays readable for external SQLite readers in the meantime. **A refusal stops the server too, not only ingest.** The server checks the schema at startup (`AssertReady`) and exits when `observers.idx` is not the primary key, so the whole site is down until the cause above is fixed or the previous version is started again. Check for a locally added column or a NULL `id` before upgrading if you are unsure. **Ingest cost.** Measured with `BenchmarkUpsertObserverAtExisting` (`cmd/ingestor`): 20000 `UpsertObserverAt` calls over 500 existing observers, on a store opened by `OpenStore` with its own pragmas (WAL, `synchronous=FULL`), Ryzen 3700X in Docker. A plain upsert on the new table writes 5.23 WAL frames per call; update-first writes 4.23, one frame less. The pre-PR table was not measured. Time per call is 676 to 726 µs before and 665 to 668 µs after; it is bound by fsync, so the difference is within noise. ## Testing - `internal/dbschema/observers_idx_test.go`: the rebuild keeps every observation linked to the same observer, refuses an unknown column or a NULL id, never reuses an `idx` (including one only orphaned observations still reference), is idempotent, and handles the full legacy table. - `TestObserversIdxPreservesLinksOnE2EFixture`: on a copy of `test-fixtures/e2e-fixture.db`, 499/499 observations keep the same observer. - Refusals and rollback: a refused rebuild leaves the table, its rows, `foreign_keys` and a readable `packets_v` as they were; a failure after the old table is dropped rolls all of it back, leaves no `observers_new`, and the next start completes the rebuild (`TestObserversIdxRollsBackMidRebuild`, `TestPacketsViewFollowsObserversShape`). - Guards: `TestNoObserversRowidInSource` (ingestor) and `TestNoObserversRowidInServerSources` fail if a query joins on `observers.rowid` again. `TestFreshObserversMatchesRebuiltShape` pins that a fresh install and a rebuilt one end up identical. - `TestUpsertObserverAtLeavesSequenceAlone` and `TestRecordNaiveSkewLeavesSequenceAlone`: 1000 calls on an existing observer leave the sequence alone (on the previous head both moved it 1 -> 1001), and a new observer then gets `max(idx) + 1`. `TestUpdateObserverMatchesUpsert` and `TestUpdateNaiveSkewMatchesUpsert` check that the update path writes the same row as the upsert (COALESCE of empty fields, `last_seen` never going back, the `can_relay` tri-state, the 24 h skew counter). - `TestAssertReadyRequiresObserversIdxPrimaryKey`: the rebuilt shape passes; a plain `idx` column or none fails. `TestOpenStoreRefusesUnconvertibleObservers`: through `OpenStore`, an unknown column or a NULL id fails the open and leaves the table, its rows and `packets_v` as they were. - `go vet` and `go test` pass on all modules (`scripts/allmod.sh`), with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Suggested order, none of it urgent: 1. **#2408** — never lose rows during startup / a failed poll. 2. **#2407** — explicit `observers.idx` primary key. 3. **#2409** — content-hash migration moved into the ingestor. 4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not affected (zig already links them with 8 MiB thread stacks); it fixes native Alpine builds. After these land, #2396 (Postgres) will be rebased onto master. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1ed5e4ab3f |
fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)). Green commit: `d1bda496`. Observer comparison counted an observer's TX-only packets as receptions, producing false overlap. It now requests `excludeTx=true`: the selected observer and non-TX direction must match the same observation in memory and SQLite v2/v3. Normal packet lists and stored TX records remain unchanged; missing directions retain legacy compatibility. Refs #667. This completes a comparison slice, not the issue's remaining observer counters or #671's fleet/SNR feature. Representative display fields and existing time-filter semantics are unchanged. API/OpenAPI docs describe the filter and multi-node bypass. Validation: - RED: Go tests compiled with 32 intended failures/16 control passes; unit and browser regressions reproduced false overlap. CI reached the new unit assertion. - GREEN: all 48 packet cases pass, plus grouped-cache isolation and OpenAPI tests. The added cache regression caught a malformed-radio cache-key collision before correction. - Full local server suite: 174s; ingestor: 185s; `npm run test:unit`: pass. Final focused packet/grouped/OpenAPI checks: pass. - E2E assertion added to `tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks pass, including correct 1/2 reception totals and 50% overlap. Performance: still two capped bulk requests, indexed candidate lookups retained, one direction check within the existing observation scan/SQL predicate, no new persistent structures. No configuration/customizer changes. Final-head CI passed; independent reviews found no must-fix items. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |