2 Commits

Author	SHA1	Message	Date
Kpa-clawbot	f4cf2acbc0	perf: cancelled writes + ingestor I/O + threshold tests (#1120 follow-up) (#1167) ... Red commit: e964ec9c46 (CI run: pending — workflow only triggers on PR open) Partial fix for #1120 — finishes the four follow-up items left open after PR #1123 (cancelled writes, ingestor I/O, threshold-flag tests, docs). ## What's done - **`cancelledWriteBytesPerSec`** — server `/proc/self/io` parser handles `cancelled_write_bytes`; `/api/perf/io` exposes the per-second rate; Perf page renders it next to Read/Write with ⚠️ when sustained >1 MB/s. - **Ingestor `/proc/<pid>/io`** — `cmd/ingestor/stats_file.go` samples its own `/proc/self/io` each tick and includes `procIO` in the snapshot. The server's `/api/perf/io` reads it and surfaces `.ingestor`. Frontend renders an `Ingestor process` Disk I/O block alongside the existing `server process` block (issue mockup: "Both ingestor and server"). - **Threshold + anomaly tests** — `test-perf-disk-io-1120.js` now asserts ⚠️ fires/suppresses on WAL>100MB, cache_hit<90%, and the backfill-rate-vs-tx-rate guard with the `tx_inserted >= 100` baseline floor. Drops the tautological `|| ... === false` short-circuits flagged in MINOR m4. - **Docs (m8)** — `config.example.json` adds `_comment_ingestorStats` (env var, default path, shared-tmp security note); `cmd/ingestor/README.md` adds `CORESCOPE_INGESTOR_STATS` to the env-var table plus a `Stats file` section. ## What's NOT done (deferred) m1 sync.Map → map+RWMutex, m2 perfIOMu rate caching, m3 negative cacheSize translation, m5 deterministic-write test, m7 ctx-aware shutdown — pure polish; will file a follow-up issue if the operator wants them tracked. ## TDD - Red: `e964ec9` — adds failing tests + stub field/handler shape (cancelled missing from struct, ingestor stub returns nil, ingestor procIO absent). - Green: `1240703` — wires up the parser case, ingestor sampler, frontend rendering, docs. E2E assertion added: test-perf-disk-io-1120.js:108 --------- Co-authored-by: clawbot <clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot>	2026-05-08 16:29:23 -07:00
Kpa-clawbot	74dffa2fb7	feat(perf): per-component disk I/O + write source metrics on Perf page (#1120) (#1123) ... ## Summary Implements per-component disk I/O + write source metrics on the Perf page so operators can self-diagnose write-volume anomalies (cf. the BackfillPathJSON loop debugged in #1119) without SSHing in to run iotop/fatrace. Partial fix for #1120 ## What's done (4/6 ACs) - ✅ `/api/perf/io` — server-process `/proc/self/io` delta rates (read/write bytes per sec, syscalls) - ✅ `/api/perf/sqlite` — WAL size, page count, page size, cache hit rate - ✅ `/api/perf/write-sources` — per-component counters from ingestor (tx/obs/upserts/backfill_*) - ✅ Frontend Perf page — three new sections with anomaly thresholds + per-second rate columns ## What's NOT done (deferred to follow-up) - ❌ `cancelledWriteBytesPerSec` field — issue #1120 lists this under server-process I/O ("writes the kernel discarded — interesting signal"); not exposed in this PR - ❌ Ingestor `/proc/<pid>/io` — issue #1120 says "Both ingestor and server"; only server-process I/O lands here. Adding ingestor I/O requires either a unix socket back to the server, or surfacing the ingestor pid through the stats file. Doable without changing the existing API shape. - ❌ Adaptive baselining — anomaly thresholds remain static (10×, 100 MB, 90%); steady-state baselining can come once we have enough deployed Perf-page telemetry Per AGENTS.md rule 34, this PR uses "Partial fix for #1120" rather than "Fixes #1120" so the issue stays open until the remaining ACs land. ## Backend **Server (`cmd/server/perf_io.go`)** - `GET /api/perf/io` — reads `/proc/self/io` and returns delta-rate `{readBytesPerSec, writeBytesPerSec, syscallsRead, syscallsWrite}` since last call (in-memory tracker, no allocation per sample). - `GET /api/perf/sqlite` — returns `{walSize, walSizeMB, pageCount, pageSize, cacheSize, cacheHitRate}`. `cacheHitRate` is proxied from the in-process row cache (closest available signal under the modernc sqlite driver). - `GET /api/perf/write-sources` — reads the ingestor's stats JSON file and returns a flat `{sources: {...}, sampleAt}` payload. **Ingestor (`cmd/ingestor/`)** - `DBStats` gains `WALCommits atomic.Int64` (incremented on every successful `tx.Commit()` and on every auto-commit `InsertTransmission` write) and `BackfillUpdates sync.Map` keyed by backfill name with `IncBackfill(name)` / `SnapshotBackfills()` helpers. - `BackfillPathJSONAsync` now increments `BackfillUpdates["path_json"]` per row write — the BackfillPathJSON-style infinite loop becomes immediately visible at `backfill_path_json` in the Write Sources table. - New `StartStatsFileWriter` publishes a JSON snapshot to `/tmp/corescope-ingestor-stats.json` (override via `CORESCOPE_INGESTOR_STATS`) every second using atomic tmp+rename. The tmp file is opened with `O_CREATE|O_WRONLY|O_TRUNC|O_NOFOLLOW` mode `0o600` so a pre-planted symlink in a world-writable `/tmp` cannot redirect the write to an arbitrary file. ## Frontend (`public/perf.js`) Three new sections on the Perf page, all auto-refreshed via the existing 5s interval: - **Disk I/O (server process)** — read/write rates (formatted B/KB/MB-per-sec) + syscall counts. Write rate >10 MB/s flags ⚠️. - **Write Sources** — sorted table of per-component counters with a per-second rate column derived from snapshot deltas. Backfill rows show ⚠️ only when `tx_inserted >= 100` (meaningful baseline) AND the backfill's per-second rate exceeds 10× the live tx rate. Avoids the startup-spurious-alarm where cumulative-vs-cumulative was a tautology. - **SQLite (WAL + Cache Hit)** — WAL size (⚠️ when >100 MB), page count, page size, cache hit rate (⚠️ when <90%). ## Tests - **Backend** (`cmd/server/perf_io_test.go`) — `TestPerfIOEndpoint_ReturnsValidJSON`, `TestPerfSqliteEndpoint_ReturnsValidJSON`, `TestPerfWriteSourcesEndpoint_ReturnsSources` exercise the three new endpoints. Skips the `/proc/self/io` non-zero-rate assertion when `/proc` is unavailable. - **Frontend** (`test-perf-disk-io-1120.js`) — vm-sandbox runs `perf.js` with stubbed `fetch`, asserts the three new sections render with their headings + values. E2E assertion added: test-perf-disk-io-1120.js:91 ## TDD 1. Red commit (`21abd22`) — added the three handlers as no-op stubs returning empty values; tests fail on assertion mismatches (non-zero rate, `pageSize > 0`, headings present). 2. Green commit (`d8da54c`) — fills in the real `/proc/self/io` parser, PRAGMA queries, ingestor stats writer, and Perf page rendering. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>	2026-05-05 17:56:56 -07:00