mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-15 07:09:58 +00:00
4e91c5cf0bb2e35e12a074580f4d993c32cb6f00
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 n30nex
|
b9fb3d244c |
feat(analytics): split adverts using recorded route evidence (#2088)
Red commits: `dc7dcbe`, `76e3217` ([assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37059977385)). Fixes #2041; supplies #2085's backend contract. Records at most two route bits per retained advert. Node flood counts now use the same evidence, including mixed and transport-flood adverts. Analytics failures log/count errors while core observation, path and liveness writes continue. The unrelated recent-advert limit clamp is removed. Direct adverts (empty path) describes the observed remaining path across valid hash widths. Firmware can forward an advert down to an empty path; this cannot establish original send mode or RF distance. The `zero_hop` API spelling remains compatible. Successful post-upgrade evidence writes survive observation replacement and restart. Older overwritten frames and failed writes can leave gaps. The UI explains automatic background backfill. ## Measured validation Synthetic Windows workloads; timings are not production guarantees: - 300K retained adverts, one-hour window: median 64.35→39.84 ms; 13.98 MB→145 KB allocated. Seven-day allocations unchanged; five paired samples. - 1M transmissions/11M observations: backfill 5m43s. Simple 10 Hz writer: all 3,423 observations persisted; p99 28 ms, maximum 323 ms. - Simulated pre-backfill cursor: retained evidence ready 4m21s; full catch-up 6m41s, 122 cache invalidations. Normal completed-migration startup bulk-loads current evidence. Portable opt-in scale tests and window benchmarks are included. All 183 frontend suites, full server tests, targeted races, lint and real-API desktop/mobile checks passed. Full [Linux CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37063210229) passed at `36def9c`; three independent reviews are clear. Local Windows testing hit the unchanged symlink-privilege test. No configuration changes. Preflight overrides: external script/OpenClaw profile unavailable; repository checks and Chromium used. |
||
|
Alex B
|
e6323ec587 |
fix(store): account path, decode-cache and dedup-key bytes so maxMemoryMB eviction triggers (#2035)
trackedBytes undercounted the packet store by about 2.5x, so packetStore.maxMemoryMB never triggered: a tx was charged at creation, before pickBestObservation set its path, so the byPathHop and spTxIndex costs were never added, and eviction then re-estimated with the path known and subtracted more than had been added, drifting the total downwards. The ParsedDecoded cache, the obsKeys dedup key and several per-observation strings were not estimated at all. StoreTx.accountedBytes now records what was charged, rechargeTx returns the delta after every pickBestObservation, and eviction subtracts accountedBytes instead of re-estimating. Measured by the author on a production database copy: trackedMB 151 against 402 MB of heap in use before, 351 against 396 MB after, with GC cycles dropping from ~0.71/s to ~0.011/s over 12 h on their instance. Reviewed by auditing the accounting rather than the arithmetic: all six production pickBestObservation sites recharge, all three subtraction sites read accountedBytes, every recharge site holds s.mu (Load from :857, the two ingest paths at :2778 and :3142 with deferred unlocks), observations are charged only after acceptance, and no charged tx is discarded during the chunk merge. That lock audit is the independent check, because CI's race job covers the ingestor only. Operator impact, both from the estimate growing rather than any limit moving: where maxMemoryMB is set, eviction now caps the real store size, and the cold load clamp drops about a third of the boot walk (124420 to 84374 packets at 650 MB). Where it is unset, which is the default, the change is inert. docs/go-migration.md claimed the Go server ignored the setting, which was never true, and is corrected here. Merged by the interim maintainer without a second human reviewer: CI (run 35258839322) plus the review above are the independent checks. |
||
|
efitenandClaude Opus 5
|
2d4019f719 |
fix(analytics): recompute once the store has fully loaded (#2025)
Refs #2023, #1659, #1724 ### Problem `main.go:258` waits only for the first load chunk, then `main.go:402` starts the analytics recomputers. `Start()` computes immediately on that chunk (`analytics_recomputer.go:86` on master) and the next compute waits a full interval (`:93`, 5 min default). The chunk loader walks by ascending id, so that chunk holds the oldest transmissions. - RF, topology, channels: the #1659 gate checked `LoadComplete()` after the compute (`analytics_warmup_1659.go:122`). `LoadComplete` flips at the end of the hot window (`chunked_load.go:489`), before the background fill (`store.go:1455`), so the gate could open on a snapshot without the background fill, and the 60 s force timeout (`:73`, `:162`) opened it on the first-chunk snapshot. In an end-to-end test on master, `/api/analytics/rf` returned 200 with 8 of 100 packets before the background fill ran. - Distance, hash-collisions, hash-sizes, roles, observers-clock-skew, nodes-clock-skew: no gate, partial snapshot served from the start. - Distance additionally served a snapshot from the previous index for up to one interval after each lazy index build. On a staging instance, a default analytics request returned 5,911 packets with hours-old last buckets until the next recompute (about 74k). ### Change - `StartupLoadDone()` (`chunked_load.go:108`): closed when `RunStartupLoad` returns, on every path (`chunked_load.go:202`). Closing it drops the hash-size info cache (15 s TTL) and the clock-skew engine throttle (30 s, `clock_skew.go:225`), both read by the post-load computes. - `recomputeWhenLoaded` (`analytics_recomputer.go:172`): on that signal, recompute each recomputer once, sequentially, via `RecomputeNow` (`:154`), which runs on the recomputer's own loop and restarts its ticker (`:106`). Order (`:255`): rf, topology, channels, distance, hash-collisions, hash-sizes, observers-clock-skew, nodes-clock-skew, roles (roles reads the nodes-clock-skew snapshot). Logs one line with per-recomputer durations. - Warm-up gate: now the same signal (`:343-354`), sampled before the compute starts (`:129`), so a pass that began on partial data never opens it. 503 + `Retry-After: 5` and the force timeout are unchanged; a forced-open snapshot is replaced by the post-load recompute. - Ungated endpoints: no new 503s (their API has none); snapshot replaced right after the load. - Distance: the lazy index build refreshes the distance recomputer before reporting built (`store.go:4476`). - Recompute intervals and config unchanged. ### Performance One extra compute per recomputer per process start, run sequentially so they do not all hold the store read lock at once. Ticker phases afterwards are offset by the cumulative post-load compute durations instead of all starting within the first-chunk compute window (relevant to #1724; the effect on lock waves is not measured). ### Tests `analytics_recompute_after_load_test.go`: signal open during background fill, closed after success and failure; cache drops; immediate and ordered post-load recompute; gate not opened by a pass started before the load; forced-open snapshot replaced on load; ticker restart; distance refresh before 202 ends; end to end with recomputers started before the background fill (RF 503 until load, then `totalTransmissions` equals the full store; six ungated endpoints 200 during load; all nine recomputed after load). 9 of these failed on master with stubs; 8 single-line mutations each caught. `go test ./...` in `cmd/server` passes. ### Staging validation Deployed together with the review follow-ups of #2015-#2023 (build `c646310f`), container restart: ``` 16:35:20 [store] first chunk ready (chunkSize=10000) 16:35:25 [store] LoadChunked complete ... starting background fill loader 16:36:58 [store] background load complete: 121120/121282 packets in memory (coverage=99.9%) 16:37:03 [analytics-recompute] startup load done: recomputed 10 snapshots in 5.155s (rf=955ms topology=1.684s channels=43ms distance=49ms hash-collisions=30ms hash-sizes=338ms observers-clock-skew=369ms nodes-clock-skew=692ms roles=2ms retransmissions=994ms) ``` Right after that line, `/api/analytics/rf` reported `totalTransmissions` 121,121 against 120,700 packets in memory, and the retransmissions default shape from #2023 covered the full 7 days. Before this change both waited for the next 5 minute tick. ### Merge order with #2023 #2023 adds a tenth recomputer. Whichever of the two merges second has to add `recompRetransmissions` to `analyticsRecomputersLocked`, wire it to `loadedGate` instead of `LoadComplete`, and change 9 to 10 in `TestAnalyticsRecomputers_PostLoadOrder`; the retransmissions gate test then calls `signalStartupLoadDone()` instead of setting `loadComplete`. That resolution is what ran on staging above. ### Not verified - Repeater-enrich recomputer and the region/window TTL caches (hash-collisions region results have a 1 h TTL) may also keep partial results after the load; not changed here. - Recompute order is tested structurally, not with roles/clock-skew data. - Whether this reduces the #1724 stalls; not measured. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
376c3e9f4a |
fix(packets): surface the transport region scope — detail pane row and a sortable Scope column (#1894)
## Summary `transmissions.scope_name` (#899) reached the database but never reached the UI. Two problems, one dead feature and one missing surface. ## 1. The detail pane's Scope row was dead `public/packets.js:3279` has rendered a **Scope** row since #899, gated on `pkt.scope_name != null`. It never fires in practice. `/api/packets` and `/api/packets/{id}` are served from the in-memory `PacketStore`. The store reads `scope_name` out of SQLite fine (`store.go:888`, `chunked_load.go:551` → `StoreTx.ScopeName`), but `txToMap()` did not put it in the JSON. Only packets old enough to have been evicted from the store — and thus served by the SQLite fallback in `db.go`, which does emit it — could ever show a scope. Verified against a live instance before the fix: ``` GET /api/packets/552e9687f1525537 → packet keys: ['_parsedPath','decoded_json','direction','first_seen','hash','id', 'observation_count','observations','observer_iata','observer_id', 'observer_name','path_json','payload_type','raw_hex','route_type','rssi','snr','timestamp'] ``` No `scope_name`. ### The NULL / "" distinction `StoreTx.ScopeName` was typed `string`, which collapses the two states the frontend distinguishes: | DB value | Meaning | UI | |---|---|---| | `NULL` | not transport-scoped | row hidden | | `""` | transport-scoped, region matched no configured key | muted "unknown scope" | | `"#be"` | matched region | the region name | `route_type` is **not** a usable proxy for that distinction: the ingestor writes NULL for a transport route whose `transport_code_1` is `0000` (`cmd/ingestor/db.go:1576` — `IsTransportScoped = route_type IN (0,3) AND Code1 ≠ "0000"`). So the field is now `*string`, with `nullStrPtr` preserving what `nullStrVal` collapsed. The two internal consumers (`TransportedScopes` #1751, `relayEntry.scope`) only care about non-empty named scopes and are unchanged in behaviour. ## 2. New: a Scope column on the packets table The scope was only reachable one packet at a time by opening the detail pane. It now has its own sortable column between Type and Observer, visible by default. The default view is **Group by Hash**, served by mappers that did not carry `scope_name` at all — so the column would have been empty in exactly the view most people look at. Both grouped paths now select and emit it: `groupedTxsToPage` in the store, and the dedicated grouped query in the DB fallback (v3 and legacy shapes). Rendering lives in `scopeCellHtml` (`public/app.js`, next to `transportBadge`) and is used on all three row-render sites — group header, expanded children, flat rows — so the column and the detail pane cannot drift apart. **Sorting** pins the empties last in both directions, as the nodes table already does for `default_scope`. Only ~8% of packets carry a scope, so an ascending sort would otherwise bury every scoped row under a wall of dashes. **Filtering**: `packet-filter.js` gains a `scope` field, so the cell is click-to-filter like Type and Observer, and `scope == "#be"` works in the filter bar. **Column prefs**: a `packets-known-cols` companion key. The `packets-visible-cols` array alone cannot distinguish "this column did not exist when you saved" from "you unchecked it", so any new column arrives silently hidden for every returning visitor. Keys absent from `known-cols` get the default treatment; keys the visitor actually hid stay hidden — there is a test for that second half specifically. ## Tests Each watched fail first. **Go** (`cmd/server/packet_scope_name_test.go`) - `txToMap` unit tests for all three states, including a JSON round-trip so a typed nil `*string` cannot pass as `null` - end-to-end through `/api/packets/{hash}` - `groupedTxsToPage` unit + end-to-end through `/api/packets?groupByHash=true`, across **both** the store-backed and DB-fallback paths - `transported_scopes_1751_test.go`: the "no scope" guard now covers both non-values (nil and a pointer to `""`) **Frontend** - `test-frontend-helpers.js`: `scopeCellHtml` three states + escaping - `test-packet-filter.js`: `scope` matching, case-insensitivity, and `FIELDS` registration - `test-packets-scope-column.js` (new Playwright e2e): header position, default visibility, one cell per row, em dash on non-transport rows, empties-last sorting, the Columns toggle, and the prefs backfill ## Verification Deployed and checked against a live instance: ``` /api/packets?groupByHash=true&limit=500 → scope_name present on 500/500, 59 with a matched region, 1 unknown-scope test-packets-scope-column.js → 7 passed, 0 failed cd cmd/server && go test ./... → ok ``` Two pre-existing failures, unrelated and equally red on an unmodified checkout: `test-e2e-playwright.js` "Customizer open does not overwrite server home config" and `test-observer-iata-1188-e2e.js` (timeout on `[data-loaded="true"]`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
fa15ab0a30 |
fix(#1809): gate background loader on LoadChunked completion (#1811)
Partial fix for #1809. Red commit: |
||
|
Michael J. Arcan
|
ae2e3933dd |
feat(server): store memory diagnostics + drop redundant obs.RawHex (#1773)
Drops the redundant per-observation RawHex (~98MB on a live store; reader already falls back to tx.RawHex #881) and adds an opt-in /api/perf?mem=1 memory breakdown (flood-forward share + per-component bytes). Profiled against a live instance. **Savings substantiation:** live-instance profiling shows ~1.66M observations in the store, each previously carrying its own per-observation `raw_hex` (avg ≈118 hex chars ≈59 bytes) that exactly duplicates the parent transmission's `raw_hex`. Dropping the duplicate on every load/ingest path eliminates ≈98 MB of redundant in-memory storage plus ~1.66M string allocations, with no data loss — the read path (`enrichObs`) already falls back to `tx.RawHex` when `obs.RawHex` is empty (verified by the new safety-gate test). The patched build cannot be run against the live instance here; instead the new opt-in `/api/perf?mem=1` diagnostic lets operators measure the real before/after (`trackedMB` and the per-component breakdown) directly after deploy. |
||
|
Michael J. Arcan
|
fc26fb6b3a |
feat(#1751): show transported region scopes in repeater sidebar (#1752)
Closes #1751. |
||
|
|
048143f54f |
fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691)
## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com> |
||
|
|
938153dd92 |
fix(nodes): rebuild relay-hop history on startup from path_json (#1643)
## Problem A relay node's **activity timeline** — and its per-node `packetsToday` / observer counts — collapses to *"only the hour the server restarted"* after every restart. Before the restart the timeline shows only the node's own adverts (~1–2/hr); all of its relay activity piles into the single post-restart hour. ## Root cause All DB cold-load paths (`Load`, `loadChunk`, `scanAndMergeChunk`) index relay-hop attribution into `byNode` **only** from `observations.resolved_path`. But since #1287 the ingestor persists relay data as aggregate `neighbor_edges` and **never writes `resolved_path`** — it is `NULL` on every deployment (verified on a live DB: 0 of ~440k rows populated). So relay attribution is never reconstructed on startup; it only re-accumulates from live traffic (`IngestNew*`, which re-resolves from `path_json` + the neighbor graph), piling a relay node's whole history into the post-restart window. ## Fix Server read-side only — **no schema / ingestor / migration change**. When `resolved_path` is empty, re-resolve relay hops from the already-persisted `path_json` using the in-memory prefix map + neighbor graph (the same `resolvePathForObs` compute the live ingest path already runs). `main.go` now loads the persisted neighbor graph *before* the packet load so resolution has the graph available. Two correctness details worth a close look: 1. **Fetch the prefix-map/graph snapshot BEFORE opening each load cursor.** `getCachedNodesAndPM` issues its own DB query; doing so while a load cursor is open deadlocks on a single-connection SQLite pool (the test harness uses one). 2. **Index into `byNode` ONLY** — not the `resolved_path` / path-hop indexes. Those are cross-checked by `handleNodePaths` against the persisted `resolved_path` column (NULL here); populating them from an in-memory re-resolution would make that SQL confirmation fail and wrongly drop the tx from paths-through (#1352). ## Tests New coverage asserts a relay pubkey reachable *only* via `path_json` lands in `byNode` after a restart-style load, for both the hot-window (`LoadChunked`) and background-window (`loadChunk`) paths. Existing #1558 (`resolved_path`) and #1352 (paths-through) tests still pass. Full `cd cmd/server && go test ./...` is green under `-race`. ## Perf The fallback runs `resolvePathForObs` per observation with a non-empty `path_json` during cold load — the same per-packet compute the live ingest path already performs, so no new asymptotic cost. The prefix map + graph are snapshotted **once per load** (not per row); `getCachedNodesAndPM` is 30s-cached. In `loadChunk` the resolution runs in the existing lock-free scan and is accumulated locally, matching that function's "build local, merge under lock" design. ## Note on a pre-existing flaky test `TestDistanceConcurrentRequestsDuringBuildReturn202` is timing-fragile (fails ~1/15 on `master` without this change). It relies on the lazy distance build being slow because it's the first caller of `getCachedNodesAndPM` (cold cache). This PR pre-warms that cache during `Load`, narrowing the build window, so the test fails more often in **non-race** local runs. It passes reliably under `-race` (CI mode), where the build stays slow. Flagging in case you want to harden the test separately. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
|
bc1822e46c |
perf(load): chunked Load with early HTTP readiness (#1009) (#1596)
## What Switches the server's startup from a synchronous full-scan `PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that: 1. Streams transmissions+observations from SQLite in id-ordered chunks (default `chunkSize=10000`, configurable via `db.load.chunkSize`). 2. Closes `FirstChunkReady()` after the first chunk is merged — `main.go` binds the HTTP listener on that signal instead of blocking on the full multi-minute load. 3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every response while LoadChunked is in flight, flipping to `ready` once it completes (via `loadStatusMiddleware`). 4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB` clamps and the post-load index rebuild (`pickBestObservation` / `buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`). ## Why Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked load the listener binds within seconds; dashboards and probes can read partial data and see the `loading` status header until the background load finishes. ## Notes - `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`) is unchanged — it still waits for neighbor-graph build + initial `pickBestObservation` before reporting `ready:true`. `LoadChunked` only changes when the listener BINDS, not when it advertises ready. - `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on a tiny DB) before proceeding, and drains the load goroutine in the background with a logged error path. - Config Documentation Rule: `config.example.json` now documents `db.load.chunkSize` with a nested `_comment` describing the trade-off. ## Tests - `cmd/server/chunked_load_test.go` asserts: - (a) `FirstChunkReady` fires before `LoadChunked` returns - (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` → `ready` - (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via `OnChunkLoaded`) - (d) `Config.DBLoadChunkSize()` default 10000 + override - Red commit (`102a4c84`) lands the tests with stubs that fail on assertion — verified locally before the green commit. - Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite green (47s locally). Closes #1009 ## TDD red-commit exemption The original red commit `f878e15e` ("test(load): failing tests for chunked Load + early HTTP readiness") fails to **compile** rather than failing on an assertion, because it references symbols (`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`, `Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A compile error is NOT a valid red commit." This is claimed under the **net-new surface** exemption with the following justification: - LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize are all introduced by this PR — no prior implementation existed to refactor. There is no behaviour on master that the red commit could meaningfully assert against without first declaring the new symbols. - The cheapest "proper" alternative (split the red into two commits: stub-first + assertion-fail) was deferred because the test file unambiguously fails on missing-symbol — there is no risk of the test becoming a tautology against a pre-existing stub. - **Behaviour gating IS proven elsewhere on this branch.** Commit `799bde49` ("test(load): red — LoadChunked must mark indexes ready + not flip Complete on error") is a proper assertion-fail red against the same package, and commit `92cadd1d` is the matching green. Reviewers can verify the red→green pattern there. If a future reviewer wants the strict pattern, the follow-up is mechanical: split `f878e15e` into a stub-only commit followed by the assertion commit. Not done here to keep the rework cost proportional to the risk (zero, in this case). ## Preflight overrides - check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and `cmd/server/chunked_load_oldest_test.go` only. They run against per-test `t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single test) — they are NOT production schema migrations. No prod table is touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir fixture). --------- Co-authored-by: CoreScope Bot <bot@corescope.local> Co-authored-by: clawbot <bot@noreply.example.com> Co-authored-by: Kpa-clawbot <bot@example.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |