From c98c2f53e2921bed017fa1d0ecc2016f20a5c374 Mon Sep 17 00:00:00 2001 From: gadgethd <111318106+gadgethd@users.noreply.github.com> Date: Sun, 11 Oct 2026 21:47:08 +0000 Subject: [PATCH] docs: record verified F06-F08/F17 repair checkpoint --- FIX-PLAN.md | 19 ++++++++++++++++- FIX-REPORT.md | 59 +++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 71 insertions(+), 7 deletions(-) diff --git a/FIX-PLAN.md b/FIX-PLAN.md index 937448e..5f3d557 100644 --- a/FIX-PLAN.md +++ b/FIX-PLAN.md @@ -68,7 +68,24 @@ migration, potentially deleting distinct observations. Schema reapplication now preserves upstream IDs and startup repairs actual missing/legacy indexes even when an earlier migration marker says done. Both source orders pass active-last-job process crash, SQLite failure, cursor hold and replay/count -checks. Full CoreScope suites are running before its local commit. +checks. Full ingestor/server/shared-schema suites pass. CoreScope is committed +locally at `ff9eb3b` (not pushed); both unrelayed commits are exported in the +owned bench verification evidence directory. RAM `6ed4296` is pushed; its +source-labelled image passed all eight compiled PostgreSQL/Redis regression +tests with zero skips. Both code-head CI workflows `38176440656` and +`38176443793` concluded success in all four jobs. +Fresh full local gates also pass with two CPUs per suite to bound fixture +startup contention: perf 418/418, RAM 469/469, and both typechecks. +Existing bench services and production remain untouched. Next prioritize F01 +copied-data revocation, then the F09/F12–F16 operational/read gates and the ready +WebSocket gap. Preserve the new fixtures and logs in `verification-sol-20261011`; +Docker cannot bind the tool session's private `/tmp` files directly. + +For F01, also test the SQL snapshot's aggregate observation/observer fields +against a newer authorization generation. The current merge reclassifies the +representative packet, so a privacy transition after SQL must not preserve +older aggregate contributions from other observers or paths. Carry and check +the SQL snapshot generation as part of the current-authority contract. ### Owner handoff at 20:55 UTC diff --git a/FIX-REPORT.md b/FIX-REPORT.md index caa85c1..4df17be 100644 --- a/FIX-REPORT.md +++ b/FIX-REPORT.md @@ -16,8 +16,44 @@ The complete ingest integration command passes **6 PostgreSQL + 2 Redis tests** using a newly created disposable PostgreSQL database and an independently created Redis fixture (`/tmp/ukmesh-sol-full-ingest-integration.log`). Focused CoreScope F07/F08/F17 tests pass with real PostgreSQL/Redis/SQLite -(`/tmp/core-sol-f08-integration-after.log`); full Go suites are still running. -No new repair-wave push or CI success is claimed yet. +(`/tmp/core-sol-f08-integration-after.log`). Full successor Go suites pass: +ingestor **222.972s**, server **116.673s**, shared schema **0.530s**. Logs are +`/tmp/core-sol-full-ingestor.log`, `/tmp/core-sol-full-server.log` and +`/tmp/core-sol-full-dbschema.log`. + +RAM `6ed4296e988f1ec69640d02443f551d37c619400` is pushed and verified by +`git ls-remote`. Its workflows +[38176440656](https://github.com/gadgethd/ukmesh/actions/runs/38176440656) and +[38176443793](https://github.com/gadgethd/ukmesh/actions/runs/38176443793) +both concluded **success in all four jobs**, including full Backend and +Workers/Compose/E2E/empty-volume gates. Both PR descriptions are updated; #122 remains +open/non-draft and #125 remains open/draft. #122's title now describes the +supported N=1 repair instead of obsolete multi-connection performance. + +CoreScope is locally committed at `ff9eb3b93c9addbb29feb9debcc12e6cb99d5c8d`. +Both `e9afb3a` and `ff9eb3b` await relay; CoreScope was not pushed. Export with +`git -C /home/ben/corescope-ukmesh/src format-patch --stdout 90c104eab51618894a3b69ab521829f39313e261..ff9eb3b93c9addbb29feb9debcc12e6cb99d5c8d`. +The ready patch is `/tmp/corescope-sol-relay-20261011.patch` (2383 lines). +Evidence logs and the patch are also preserved in the new owned +`/home/ben/ukmesh-bench/mqtt-ingest-perf-20261011/verification-sol-20261011/evidence/` +directory so tool-session temporary files are not the sole handoff artifacts. + +Both uniquely tagged verification images built successfully: RAM +`mqttbench-backend:sol-handoff-6ed4296-20261011` has image ID +`sha256:563424ed79ac51abc2eb70cf8f2080bec9961e6c0622f1948233bfeaf92b0859` +and revision label `6ed4296`; durable +`mqttbench-backend:sol-durable-8968078-20261011` has image ID +`sha256:f501da065bf98e219c54d335dfad6703d430b51e6437d49a63140341dc72a637` +and label `8968078`. Existing bench services were unchanged. The completed +compiled-regression gate against the RAM image was launched by the new owned +`verification-sol-20261011/run-image-integrity.sh` under the bench directory. +It mounts only tests, executes shipped modules and requires eight passing, +zero-skipped tests; exit-zero/zero-tests cannot pass. +The `6ed4296` image passed **8/8**, zero skips, in **72.190s**, including real +snapshot/expiry, historical/compressed/contended backfill, refused-write, +replay and quarantine cases. Its evidence is +`logs/sol-image-integrity-shared-path-6ed4296-20261011.log` under the bench +directory (`IMAGE-INTEGRITY-END code=0 complete=true`). F06's new unit failed on the checkpoint (`/tmp/ukmesh-f06-before.log`), then passed with `readRecentWithPending`. The real fixture captures Redis before @@ -49,10 +85,21 @@ and startup checks/repairs the actual index state despite a done marker. The first local backend suite failed a real-time 50ms slow-mode assertion under CPU load; its clock is now deterministic and the full suite passes. +Concurrent fresh suite reruns then hit the existing two-second worker-fixture +startup timeout on both branches. Complete suites are being rerun with two +CPUs per suite (same test files/assertions) to bound test-process contention. +Both complete reruns passed: perf **418/418** in **138.715s**, RAM **469/469** +in **210.111s**, with typechecks and the alert receiver gate passing. Logs are +`evidence/perf-8968078-two-cpu-tests.log` and +`evidence/ram-6ed4296-two-cpu-tests.log` in the owned verification directory. The first full PG invocation used an already initialized bench database and failed its final empty-schema fixture; the successful rerun used a new owned -database. One intentionally stopped diagnostic CoreScope test left its owned -fixture schema for cleanup; no production data or services were involved. +database. One intentionally stopped diagnostic CoreScope test left an owned +fixture schema; that exact schema was removed. No production data or services +were involved. Preliminary image-fixture packaging had missing assets or zero +tests; neither is accepted as a pass. Docker's daemon does not share the tool +session's `/tmp`, so fixtures now use the shared, newly owned bench verification +directory. The corrected image run's exact eight-test result is verified above. Remaining mission work: F01 copied-data revocation and generation transitions; F09/F14 memory/accounting and metric semantics; F12/F13 SQL read parity; @@ -80,7 +127,7 @@ by this successor evidence. The full objective is still active. | F05 | Implemented for UKMesh; final-head gates pass | Delivery-key replay ledger in `ram/liveStore.ts`, stable observation ID and PG ledger; lost-reply and same-ms unit/PG integration regressions added. Real-PG and real-Redis CI tests pass. Final-head crash and ambiguous SQL-commit replay pass; a deterministic real-Redis XADD followed by a lost reply recovers the original observation ID with stream length one. | | F06 | Implemented; real snapshot/expiry regression passes | Per-entry commit state, retained overlap and stable-ID dedupe plus `readRecentWithPending` before-SQL capture on REST/WS. `F06 real PG snapshot and Redis expiry preserve the captured handoff` covers concurrent admission and retention expiry. Final image gate remains. | | F07 | Implemented; historical/late-commit/cursor-loss regressions pass | Migration 059 adds atomic compressed-history backfill and coverage fence without changing 058. CoreScope requires coverage and follows `seq > cursor`. `F07 historical and compressed observations enter the complete ledger atomically` and `TestF07RealLedgerHistoricalCoverageLateCommitAndCursorLoss` pass. Final image/paired offline gate remains. | -| F08 | Implemented; real paired crash/failure regression passes | Both CoreScope sources use `finishSourceBatch`; `TestF08RealPairedSourcesCrashFailureAndRestartBothOrders` proves active-last-job process crash, same-SQLite/WAL restart, actual observation insert failure, held cursors and successful replay. Full successor Go suites are pending. | +| F08 | Implemented; real paired crash/failure and full Go suites pass | Both CoreScope sources use `finishSourceBatch`; `TestF08RealPairedSourcesCrashFailureAndRestartBothOrders` proves active-last-job process crash, same-SQLite/WAL restart, actual observation insert failure, held cursors and successful replay. Full successor ingestor/server/schema suites pass at `ff9eb3b`. | | F09 | Partial | One-time atomic cleanup, bounded reconciliation and retained accounting in `ram/liveStore.ts`, `writer.ts`; replay, cleanup and real-Redis lost-counter reconciliation tests pass. Final-head concurrent byte admission passes at the enforced 1 MiB minimum. Startup/retention accounting in the final-head bench remains. | | F10 | Implemented; group-loss integration passes | Supervised startup, NOGROUP repair, periodic bounded foreign PEL reclaim, and stale-PEL/progress readiness in `index.ts`, `ram/writer.ts`, `writerRuntime.ts`; unit regressions and a final-head real-Redis group-destruction/recovery gate pass. | | F11 | Implemented; final-head fault gate passes | Real ioredis XADD quarantine, common validated decode, and rowwise permanent SQL poison isolation in `ram/writer.ts`, `protocol.ts`; malformed PEL, poison-neighbor and real-Redis quarantine tests pass. A final-head simulated permanent SQLSTATE 23514 in a two-row batch quarantined only the poison row through real ioredis, committed its valid neighbor in PG, and left PEL zero. | @@ -89,7 +136,7 @@ by this successor evidence. The full objective is still active. | F14 | Partial | #122 metric registration fixed. #125 separates pending/retained/PEL gauges, byte budget and minimum admitted-age ordering; commit batch histogram now counts actual inserted PG rows. A final-head live metric scrape confirms all those series are served under load. Final load validation remains. | | F15 | Implemented, boundary still partly unit-tested | #122 deadline fixed. #125 shares one shutdown deadline, joins writer loop, detects PEL on drain and closes clients; `drainLifecycle.test.ts` holds an active persist past shutdown. Isolated runtime SIGTERM logged graceful completion in about two seconds and its 300/300 exact identities, side effects, queue, PEL and bytes passed a manual post-restart gate. The bench's stock wait script missed this because `processed` has no zero-valued series after restart; no held active persist was injected into that runtime run. | | F16 | Partial | Compose variables and CI render assertion, `docs/ram-first-ingestion.md`, and `node dist/ram/drain.js` added. The drain waits for Redis readiness, reuses the ready client for final byte reconciliation, closes path-resolution workers, and refuses a nonempty quarantine as well as nonempty stream/PEL/byte reservation. Isolated standalone drain, quarantine refusal and durable rollback passed on `f70fe05`. Raw-stream Redis ACL isolation remains. | -| F17 | Implemented, full Go suites pass | CoreScope preserves upstream UUID in SQLite and updates observer count only for a newly inserted observation. Async indexes hold paired sources until ready. `TestF17TwoSourceObservationIdentityAndCountInBothOrders` and migration readiness test pass. | +| F17 | Implemented including restart repair; full Go suites pass | CoreScope preserves upstream UUID and increments observer count once. `ff9eb3b` prevents schema reapply from rebuilding the broad legacy index and deleting distinct IDs, and repairs actual missing/legacy index state despite done migration markers. `TestF17UpstreamIdentitySurvivesSchemaReapply`, both-order identity/count and real crash/restart tests pass. | ## Successor handoff — 2026-10-11 20:55 UTC