mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-23 19:41:26 +00:00
1bfbbd6bb2f3fdb10cfee461dbf16bce7d34da1f
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
0b35c7eef3 |
feat(server): persist multi-byte capability across restart + O(1) per-key lookup (#903) (#1324)
## Summary Follows the reconciliation recommendation in #916 — extracts only the NET-NEW persistence layer from that PR (which is now superseded by #1002 for the overlay UI) into a focused 6-file change against current master. **What this adds:** - `multibyte_sup_v1` migration: `multibyte_sup INTEGER NOT NULL DEFAULT 0` + `multibyte_evidence TEXT` on `nodes`/`inactive_nodes` so capability survives restart - `hasMultibyteSupCols` schema detection gates the persist/load paths - `loadMultibyteCapFromDB()`: pre-populates `mbCapSnapshot`/`mbCapIndex` at startup — cold starts serve last-known capability without waiting for the first ~15s analytics cycle - `maybePersistMultibyteCapability()` + `persistMultibyteCapability()`: after each analytics cycle; TryLock-gated (concurrent cycles coalesce); skips `sup==0` entries (data-destruction guard) - `GetMultibyteCapFor(pk)`: O(1) map lookup; both `handleNodes` and node-detail call sites updated from the O(N)-alloc `GetMultiByteCapMap()` **What this explicitly does NOT change:** - API field names (`multi_byte_status`, `multi_byte_evidence`, `multi_byte_max_hash_size`) - `EnrichNodeWithMultiByte` — unchanged - `GetMultiByteCapMap` — still present for any external callers - `public/map.js`, `public/live.css`, `Dockerfile`, `docs/` — zero frontend churn ## Test plan - [x] `TestMultibyteCapPersistRoundTrip` — confirmed values survive persist → fresh-store load - [x] `TestMultibyteCapPersistSkipsUnknown` — data-destruction guard: `sup==0` entry does not overwrite DB-confirmed value - [x] `TestMultibyteCapMaybePersistCoalesces` — TryLock coalesces 10 concurrent callers without deadlock - [x] `TestMultibyteCapGetMultibyteCapForO1` — O(1) index returns correct entry / false for unknown pubkey - [x] `TestMultibyteCapLoadFromDB` — only `sup>0` rows loaded; `sup==0` row excluded - [x] `TestSchemaMultibyteSupColumns` — migration adds columns to both tables; idempotent on second `OpenStore` - [x] All existing `TestMultiByteCapability_*` tests pass unchanged - [x] Full ingestor test suite: `ok` in 27s - [x] `go build ./cmd/server/ && go build ./cmd/ingestor/` clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
|
c7ab5f3eb9 |
fix(#1366): channels view shows latest message time — backend emits LatestSeen, not FirstSeen (#1368)
Red commit:
|
||
|
|
d9ba9937a6 |
fix(dbschema): canonical source for optional column migrations — fixes startup race (closes #1321) (#1322)
Red commit `2a8102b9` (failing test) → green commit `bb957c9f`. CI: https://github.com/Kpa-clawbot/CoreScope/actions/workflows/ci.yml?query=branch%3Afix%2Fissue-1321 Fixes #1321. ## Why On staging `/api/scope-stats` 500'd with `scope_name column not present` despite the ingestor adding the column ~0.5s after server startup. `cmd/server/db.go detectSchema()` runs in `OpenDB` and caches `hasScopeName`/`hasDefaultScope`/`hasObsRawHex` booleans. With supervisord launching server + ingestor simultaneously, the server's PRAGMA can fire BEFORE the ingestor's `ALTER TABLE` completes — and the boolean stays false until the server restarts. Same race class as #1283; #1289 moved server-side ensures to `dbschema` but the optional columns the ingestor still owned were left out. ## Fix — option (c) from the issue Made `internal/dbschema/dbschema.go` the single source of truth for the optional columns the server detects. **Migrations moved from `cmd/ingestor/db.go applySchema` into `dbschema.Apply`:** - `transmissions.scope_name` + `idx_tx_scope_name` partial index - `nodes.default_scope` - `inactive_nodes.default_scope` - `observations.raw_hex` **`AssertReady` now asserts** every one of those columns. The server cannot start with stale-false booleans because `AssertReady` will fatal first if the columns are missing. The ingestor's old gated blocks are replaced with pointer comments so anyone hunting for them lands in `dbschema.go`. The `_migrations` marker rows are preserved (`INSERT OR IGNORE`) to keep legacy DBs idempotent. **Documented invariant** in the package doc: any new optional column the server PRAGMA-detects belongs in `internal/dbschema/dbschema.go`, NOT in `cmd/ingestor/db.go applySchema`. ## Tests Added `internal/dbschema/dbschema_test.go` (RED in `2a8102b9`): - `TestApplyAddsOptionalColumns_CanonicalSource` — post-`Apply`, all four columns must exist. - `TestAssertReady_RequiresOptionalColumns` — `AssertReady` must refuse a DB missing them AND pass after full `Apply`. `cmd/ingestor` and `cmd/server` full suites green. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
9383201c07 |
refactor(db): finish #1283 — Option 4: ingestor owns neighbor-graph + schema migrations; server is read-only (fixes #1287) (#1289)
Red commit: https://github.com/Kpa-clawbot/CoreScope/commit/eae179b99b5fd34924547632aa8f8025c405aa53 (CI: pending — opens with this PR) Finishes #1283. RED test `TestServerSourceHasNoCachedRWCalls` goes from failing (13 writer call-sites) to GREEN (zero). Per #1287 Option 4 (https://github.com/Kpa-clawbot/CoreScope/issues/1287#issuecomment-4485099992): ingestor owns the neighbor graph build + persist; server reads the snapshot. **Category A — Schema migrations** → new `internal/dbschema` package. `dbschema.Apply(rw)` runs in `cmd/ingestor` startup (in `OpenStore`). `dbschema.AssertReady(ro)` runs in `cmd/server/main.go` and FATAL-LOG-EXITS if any expected column/index/table is missing — the operator must restart the ingestor first. Covers indexes, `neighbor_edges`, `observations.resolved_path`, `observers.{inactive,last_packet_at,iata}`, `(inactive_)nodes.foreign_advert`, `transmissions.from_pubkey`. **Category B — Backfill** → ingestor. `BackfillFromPubkey` and observer-blacklist soft-delete moved to `cmd/ingestor/maintenance.go`. Server keeps an inert `fromPubkeyBackfillSnapshot` stub for `/api/healthz` API compatibility. **Category C — Neighbor-graph persistence (Option 4)** → ingestor writes, server reads. - Ingestor (`cmd/ingestor/neighbor_builder.go`): every 60s scans `observations + transmissions`, extracts edges (originator↔first-hop for ADVERTs; observer↔last-hop for all), resolves hop prefixes via a node-table prefix index, upserts into `neighbor_edges`. - Server (`cmd/server/neighbor_recomputer.go`): every 60s re-reads `neighbor_edges` and atomic-swaps the resulting `NeighborGraph` into `s.graph`. Initial load is synchronous on startup. All server-side incremental edge writers (the two `asyncPersistResolvedPathsAndEdges` paths in `cmd/server/store.go`) are gone. - Neighbor-edge daily prune (`PruneNeighborEdges`) moved to ingestor. **Why Option 4**: clean read/write separation, no startup CPU spike (server loads existing snapshot instead of rebuilding from history), no IPC/delta-protocol churn. Staleness budget ~60s — same model as the analytics recomputers in #1240 / #1248 / #672 axis 2. **Recomputer interval default for neighbor graph**: 60s (`NeighborGraphRecomputerDefaultInterval`, `NeighborEdgesBuilderInterval`). **Invariants added**: - `TestServerSourceHasNoCachedRWCalls` (RED commit eae179b9): grep enforces zero `cachedRW(`, `mode=rw`, or `sql.Open(_journal_mode=WAL…)` in non-test `cmd/server/` sources. - `TestServerStartupRequiresMigratedSchema`: server refuses to start against an unmigrated DB. - `TestNeighborGraphRecomputerLoadsSnapshot`: post-write snapshot is picked up on the next refresh. - `TestNeighborEdgesBuilderUpsertsFromObservations`: end-to-end pipeline writes the expected edge. `grep cachedRW cmd/server/*.go | grep -v _test.go` → 0 matches. Fixes #1287. --------- Co-authored-by: MeshCore Bot <bot@meshcore.local> Co-authored-by: Kpa-clawbot <Kpa-clawbot@users.noreply.github.com> Co-authored-by: corescope-bot <bot@corescope.local> |