Commit Graph

1827 Commits

Author SHA1 Message Date
Kpa-clawbot 268f9ae5a4 ci: update go-ingestor-coverage.json [skip ci] 2026-05-06 02:31:46 +00:00
Kpa-clawbot 93f7a11906 ci: update frontend-tests.json [skip ci] 2026-05-06 02:31:45 +00:00
Kpa-clawbot 13e2d349c2 ci: update frontend-coverage.json [skip ci] 2026-05-06 02:31:44 +00:00
Kpa-clawbot 072a8f2ff9 ci: update e2e-tests.json [skip ci] 2026-05-06 02:31:43 +00:00
Kpa-clawbot b03ef4abd3 fix(packets): resolve --surface undefined + z-index scale + path chip re-measure + +N popover (#1128) (#1131)
## Summary

Resolves the **5 layout bugs** documented in
`specs/packets-layout-audit.md` from the issue investigation. All fixes
shipped in one PR per the audit's recommended fix order.

Fixes #1128

### Bug 4 (P0, 1 line) — `--surface` undefined
`var(--surface)` was referenced in **8** rules across `style.css`
(`.fux-saved-menu`, `.fux-popover`, `.path-popover`, `.fux-ac-dropdown`,
`.fux-ctx-menu`, `.path-overflow-pill:hover`,
`.fux-saved-trigger:hover`, `.fux-popover-header sticky`) but the
variable was **never defined** — every caller resolved to `transparent`
and row content bled through. Aliased `--surface: var(--surface-1);` in
the `:root`, `@media (prefers-color-scheme: dark)`, and
`[data-theme="dark"]` blocks.

### Z-index scale (foundational)
Added documented custom properties at the top of `style.css`:

```css
--z-base: 0;
--z-dropdown: 100;
--z-popover: 300;
--z-modal-backdrop: 9000;
--z-modal: 9100;
--z-tooltip: 9200;
```

New code uses these tokens. Existing working values left in place to
avoid behavioural risk.

### Bug 1 — path chip re-measure
`_finalizePathOverflow` runs **before** `hop-resolver` mutates chip text
from hex prefix → longer node name. Chips that fit on first measurement
overflow once names resolve, but the `+N` pill never gets appended.
Cleared the per-host `overflowChecked` guard and re-ran finalize on a
120 ms debounced timer, so post-resolution overflow is detected.

### Bug 2 — `+N` popover position + z-index
`.path-popover` was `z-index: 10500` (above the modal stack) and only
ever positioned **below** the pill — when near the bottom of the
viewport it hung over adjacent rows. Lowered to `var(--z-popover)`
(300), capped `max-height` from `60vh` → `240px`, and added flip-above
logic when there isn't room below.

### Bug 3 — filter-bar gap + multi-select truncation
`.filter-bar { row-gap: 6px }` was too tight for the 34px controls;
bumped to `12px`. `.multi-select-trigger` had no `max-width`, so a
selection like `"TRACE,MULTIPART,GRP_TXT"` ballooned the row and
overlapped toolbar buttons. Capped `max-width: 180px` with
`text-overflow: ellipsis` and surfaced the full selection in the
trigger's `title` attribute (so the value remains discoverable).

### Bug 5 — already addressed in #1124
Verified `.filter-group` structure prevents mid-cluster wrap; no further
change needed here.

## TDD

Branch shows the required **red → green** sequence:

| commit | result |
|---|---|
| `8ad6394` test(packets): red E2E for issue #1128 layout chaos | ✗ Bug
4 (alpha=0), ✗ Bug 2 (z=10500), ✗ Bug 3 (gap=6) |
| `eacadc1` fix(packets): resolve --surface undefined + z-index scale +
... | ✓ 5/5 |

Test file: `test-issue-1128-packets-layout-e2e.js` — asserts opaque
dropdown background, every overflowing `.path-hops` has a `+N` pill,
popover z-index ≤ 9000 + anchored to pill, filter-bar gap ≥ 10px,
trigger `max-width` bounded.

## E2E

Local run against the e2e fixture:

```
=== #1128 packets layout E2E ===
  ✓ navigate to /packets and wait for table + rows
  ✓ Bug 4: Saved-filter dropdown background is OPAQUE (alpha ≥ 0.99)
  ✓ Bug 1: every overflowing .path-hops has a .path-overflow-pill
  ✓ Bug 2: +N popover anchored to pill + z-index ≤ 9000
  ✓ Bug 3: .filter-bar row-gap ≥ 10px AND .multi-select-trigger has bounded max-width
=== Results: passed 5 failed 0 ===
```

CI hookup: please add `node test-issue-1128-packets-layout-e2e.js`
alongside the other `test-issue-XXXX-*-e2e.js` invocations in
`.github/workflows/deploy.yml` (line ~226).

## Files

- `public/style.css` — `--surface` definition × 3 blocks, z-index scale
tokens, `.path-popover`, `.filter-bar`, `.multi-select-trigger`
- `public/packets.js` — flip-above popover logic, debounced re-finalize,
trigger `title`
- `test-issue-1128-packets-layout-e2e.js` — new E2E (red → green)

---------

Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>
2026-05-05 19:19:19 -07:00
Kpa-clawbot 6b9154df3a ci: update go-server-coverage.json [skip ci] 2026-05-06 02:13:21 +00:00
Kpa-clawbot 19e58c97b6 ci: update go-ingestor-coverage.json [skip ci] 2026-05-06 02:13:20 +00:00
Kpa-clawbot e281871971 ci: update frontend-tests.json [skip ci] 2026-05-06 02:13:19 +00:00
Kpa-clawbot 642aac472f ci: update frontend-coverage.json [skip ci] 2026-05-06 02:13:18 +00:00
Kpa-clawbot 5e6dfaa496 ci: update e2e-tests.json [skip ci] 2026-05-06 02:13:17 +00:00
Kpa-clawbot 5a5df5d92b revert: group commit M1 (#1117) — starves MQTT, refs #1129 (#1130)
## Why

Diagnostic on #1129 shows PR #1117 (group commit M1 for #1115) is
fundamentally broken: it starves the MQTT goroutine via `gcMu` lock
contention, causing pingresp disconnects and lost packets at modest
ingest rates.

## Three structural defects

1. **Lock held across `sql.Stmt.Exec`** — every concurrent
`InsertTransmission` blocks for the full SQLite write latency, not just
the brief queue mutation.
2. **Lock held across `tx.Commit`** — the WAL fsync runs *under* `gcMu`,
so any backlog blocks all ingest writers AND the flusher ticker,
snowballing under load.
3. **Single-conn DB** (`MaxOpenConns=1`) — the flusher and the ingest
path serialise on one connection, turning the lock into a global ingest
stall.

Net effect: at modest packet rates the MQTT client loop misses its own
pingresp deadline, the broker drops the connection, and packets received
during the stall are lost.

## What this PR removes

- `Store.SetGroupCommit`, `Store.FlushGroupTx`, `Store.flushLocked`,
`Store.GroupCommitMs`
- `gcMu`, `activeTx`, `pendingRows`, `groupCommitMs`,
`groupCommitMaxRows` Store fields
- `groupCommitMs` / `groupCommitMaxRows` config fields and
`GroupCommitMsOrDefault` / `GroupCommitMaxRowsOrDefault` accessors
- The flusher goroutine in `cmd/ingestor/main.go`
- `cmd/ingestor/group_commit_test.go`
- The `if s.activeTx != nil { … pendingRows … }` branch in
`InsertTransmission` — reverts to plain prepared-stmt usage

## What this PR keeps (merged after #1117)

- #1119 `BackfillPathJSON` `path_json='[]'` fix
- #1120/#1123 perf metrics endpoints — `WALCommits` counter retained
- `GroupCommitFlushes` JSON field on `/api/perf/write-sources` is kept
as always-0 for API stability (server `perf_io.go` references it as a
string field name; no client breakage)
- `DBStats.GroupCommitFlushes` atomic field is removed from the Go
struct

## Tests

`cd cmd/ingestor && go test ./... -run "Test"` → `ok` (47.8s).
`cd cmd/server && go build ./...` → clean.

## #1115 stays open

The group-commit *idea* is sound — batching observation INSERTs would
meaningfully reduce WAL fsync rate. But it needs a redesign that does
**not** hold a mutex across blocking SQLite calls. Suggested directions
for a future M1:
- Channel-fed writer goroutine (single owner of the tx, ingest path is
non-blocking enqueue)
- Per-batch DB handle so the flusher doesn't serialise the ingest
connection
- Bounded queue with backpressure rather than a shared lock

Refs #1117 #1129
2026-05-05 19:02:43 -07:00
Kpa-clawbot ee7ebd988b ci: update go-server-coverage.json [skip ci] 2026-05-06 01:24:18 +00:00
Kpa-clawbot 1559dcae2b ci: update go-ingestor-coverage.json [skip ci] 2026-05-06 01:24:17 +00:00
Kpa-clawbot 0d47a2e295 ci: update frontend-tests.json [skip ci] 2026-05-06 01:24:16 +00:00
Kpa-clawbot 96cf279fdc ci: update frontend-coverage.json [skip ci] 2026-05-06 01:24:14 +00:00
Kpa-clawbot fdce338b92 ci: update e2e-tests.json [skip ci] 2026-05-06 01:24:13 +00:00
Kpa-clawbot f86a44d6b5 fix(packets): filter UX disaster — modal help, single header, bounded path rows (#1122) (#1124)
## Summary
Fixes #1122 — Packets page filter UX repairs.

## Bugs addressed
1. **Filter syntax help no longer floats over the packet table.** It now
opens inside a real `.modal-overlay` backdrop and is centered via the
existing `.modal` flex pattern (same pattern as BYOP).
2. **Duplicate "Filter syntax" header removed.** The inner `<h3>` was
redundant with the popover header `<strong>`. Help body now contains
exactly one occurrence.
3. **Path column chips no longer spill into adjacent rows.**
`.path-hops` now uses `flex-wrap: nowrap` + `max-height: 22px` +
`overflow: hidden`; individual `.hop-named` chips cap at `max-width:
120px` with ellipsis. `td.col-path` itself caps at `max-height: 28px` so
a long hop chain can never push the row past 28px regardless of hop
count.
4. **Toolbar grouping documented.** Added a fenced section comment in
`style.css` enumerating the four logical clusters (quick filters /
toggles / time window / sort & view), bumped `.filter-bar` gap from
6→8px and added `row-gap: 6px` so wrapped controls stay readable at
narrow widths.

## Test
TDD red→green. New Playwright E2E
`test-issue-1122-packets-filter-ux-e2e.js` asserts:
- Help panel rect does not overlap any visible `#pktBody` row, and a
`.modal-overlay` backdrop is present.
- Help panel contains exactly 1 `Filter syntax` occurrence (not 2).
- Every rendered `.col-path` cell stays under 60px height.

Wired into `.github/workflows/deploy.yml` Playwright fail-fast step. Red
commit: `bd58634` (test only). Green commit: `c580254` (impl).

## Files
- `public/filter-ux.js` — `_showHelp` wraps the popover in
`.modal-overlay`; `_buildHelpHtml` drops the duplicate `<h3>Filter
syntax</h3>`.
- `public/style.css` — `.modal-overlay > .fux-popover` reset,
`.path-hops` clipping, `td.col-path` height cap, `.filter-bar` section
comment.
- `test-issue-1122-packets-filter-ux-e2e.js` — new Playwright E2E.
- `.github/workflows/deploy.yml` — runs the new E2E.

---------

Co-authored-by: clawbot <clawbot@example.com>
Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
2026-05-05 18:12:33 -07:00
Kpa-clawbot 0a3590358e ci: update go-server-coverage.json [skip ci] 2026-05-06 01:08:32 +00:00
Kpa-clawbot f8fdfe38f3 ci: update go-ingestor-coverage.json [skip ci] 2026-05-06 01:08:31 +00:00
Kpa-clawbot 0ee26b4fcf ci: update frontend-tests.json [skip ci] 2026-05-06 01:08:30 +00:00
Kpa-clawbot 9827e751aa ci: update frontend-coverage.json [skip ci] 2026-05-06 01:08:29 +00:00
Kpa-clawbot 33e8263b9e ci: update e2e-tests.json [skip ci] 2026-05-06 01:08:28 +00:00
Kpa-clawbot 74dffa2fb7 feat(perf): per-component disk I/O + write source metrics on Perf page (#1120) (#1123)
## Summary

Implements per-component disk I/O + write source metrics on the Perf
page so operators can self-diagnose write-volume anomalies (cf. the
BackfillPathJSON loop debugged in #1119) without SSHing in to run
iotop/fatrace.

Partial fix for #1120

## What's done (4/6 ACs)
-  `/api/perf/io` — server-process `/proc/self/io` delta rates
(read/write bytes per sec, syscalls)
-  `/api/perf/sqlite` — WAL size, page count, page size, cache hit rate
-  `/api/perf/write-sources` — per-component counters from ingestor
(tx/obs/upserts/backfill_*)
-  Frontend Perf page — three new sections with anomaly thresholds +
per-second rate columns

## What's NOT done (deferred to follow-up)
-  `cancelledWriteBytesPerSec` field — issue #1120 lists this under
server-process I/O ("writes the kernel discarded — interesting signal");
not exposed in this PR
-  Ingestor `/proc/<pid>/io` — issue #1120 says "Both ingestor and
server"; only server-process I/O lands here. Adding ingestor I/O
requires either a unix socket back to the server, or surfacing the
ingestor pid through the stats file. Doable without changing the
existing API shape.
-  Adaptive baselining — anomaly thresholds remain static (10×, 100 MB,
90%); steady-state baselining can come once we have enough deployed
Perf-page telemetry

Per AGENTS.md rule 34, this PR uses "Partial fix for #1120" rather than
"Fixes #1120" so the issue stays open until the remaining ACs land.

## Backend

**Server (`cmd/server/perf_io.go`)**
- `GET /api/perf/io` — reads `/proc/self/io` and returns delta-rate
`{readBytesPerSec, writeBytesPerSec, syscallsRead, syscallsWrite}` since
last call (in-memory tracker, no allocation per sample).
- `GET /api/perf/sqlite` — returns `{walSize, walSizeMB, pageCount,
pageSize, cacheSize, cacheHitRate}`. `cacheHitRate` is proxied from the
in-process row cache (closest available signal under the modernc sqlite
driver).
- `GET /api/perf/write-sources` — reads the ingestor's stats JSON file
and returns a flat `{sources: {...}, sampleAt}` payload.

**Ingestor (`cmd/ingestor/`)**
- `DBStats` gains `WALCommits atomic.Int64` (incremented on every
successful `tx.Commit()` and on every auto-commit `InsertTransmission`
write) and `BackfillUpdates sync.Map` keyed by backfill name with
`IncBackfill(name)` / `SnapshotBackfills()` helpers.
- `BackfillPathJSONAsync` now increments `BackfillUpdates["path_json"]`
per row write — the BackfillPathJSON-style infinite loop becomes
immediately visible at `backfill_path_json` in the Write Sources table.
- New `StartStatsFileWriter` publishes a JSON snapshot to
`/tmp/corescope-ingestor-stats.json` (override via
`CORESCOPE_INGESTOR_STATS`) every second using atomic tmp+rename. The
tmp file is opened with `O_CREATE|O_WRONLY|O_TRUNC|O_NOFOLLOW` mode
`0o600` so a pre-planted symlink in a world-writable `/tmp` cannot
redirect the write to an arbitrary file.

## Frontend (`public/perf.js`)

Three new sections on the Perf page, all auto-refreshed via the existing
5s interval:

- **Disk I/O (server process)** — read/write rates (formatted
B/KB/MB-per-sec) + syscall counts. Write rate >10 MB/s flags ⚠️.
- **Write Sources** — sorted table of per-component counters with a
per-second rate column derived from snapshot deltas. Backfill rows show
⚠️ only when `tx_inserted >= 100` (meaningful baseline) AND the
backfill's per-second rate exceeds 10× the live tx rate. Avoids the
startup-spurious-alarm where cumulative-vs-cumulative was a tautology.
- **SQLite (WAL + Cache Hit)** — WAL size (⚠️ when >100 MB), page count,
page size, cache hit rate (⚠️ when <90%).

## Tests

- **Backend** (`cmd/server/perf_io_test.go`) —
`TestPerfIOEndpoint_ReturnsValidJSON`,
`TestPerfSqliteEndpoint_ReturnsValidJSON`,
`TestPerfWriteSourcesEndpoint_ReturnsSources` exercise the three new
endpoints. Skips the `/proc/self/io` non-zero-rate assertion when
`/proc` is unavailable.
- **Frontend** (`test-perf-disk-io-1120.js`) — vm-sandbox runs `perf.js`
with stubbed `fetch`, asserts the three new sections render with their
headings + values.

E2E assertion added: test-perf-disk-io-1120.js:91

## TDD

1. Red commit (`21abd22`) — added the three handlers as no-op stubs
returning empty values; tests fail on assertion mismatches (non-zero
rate, `pageSize > 0`, headings present).
2. Green commit (`d8da54c`) — fills in the real `/proc/self/io` parser,
PRAGMA queries, ingestor stats writer, and Perf page rendering.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>
2026-05-05 17:56:56 -07:00
Kpa-clawbot 4b3b01b50a ci: update go-server-coverage.json [skip ci] 2026-05-06 00:46:09 +00:00
Kpa-clawbot 6eea05960d ci: update go-ingestor-coverage.json [skip ci] 2026-05-06 00:46:08 +00:00
Kpa-clawbot 330942f661 ci: update frontend-tests.json [skip ci] 2026-05-06 00:46:07 +00:00
Kpa-clawbot acb7e15bec ci: update frontend-coverage.json [skip ci] 2026-05-06 00:46:06 +00:00
Kpa-clawbot 91522ef738 ci: update e2e-tests.json [skip ci] 2026-05-06 00:46:05 +00:00
Kpa-clawbot 76d89e6578 fix(ingestor): exclude path_json='[]' rows from backfill WHERE (#1119) (#1121)
## Summary

`BackfillPathJSONAsync` re-selected observations whose `path_json` was
already `'[]'`, rewrote them to `'[]'`, and looped forever. The
`len(batch) == 0` exit condition was never reached, the migration marker
was never recorded, and the ingestor sustained 2–3 MB/s WAL writes at
idle (76% of CPU in `sqlite.Exec` per pprof).

## Fix

Drop `'[]'` from the WHERE clause:

```diff
WHERE o.raw_hex IS NOT NULL AND o.raw_hex != ''
- AND (o.path_json IS NULL OR o.path_json = '' OR o.path_json = '[]')
+ AND (o.path_json IS NULL OR o.path_json = '')
```

`'[]'` is the "already attempted, no hops" sentinel (still written at
line 994 of `cmd/ingestor/db.go` when `DecodePathFromRawHex` returns no
hops). Excluding it from the WHERE lets the loop terminate after one
full pass and the migration marker `backfill_path_json_from_raw_hex_v1`
to be recorded.

## TDD

- **Red commit** (`19f8004`):
`TestBackfillPathJSONAsync_BracketRowsTerminate` — seeds 100
observations with `path_json='[]'` and a `raw_hex` that decodes to zero
hops, asserts the migration marker is written within 5s. Fails on master
with *"backfill never recorded migration marker within 5s — infinite
loop on path_json='[]' rows"*.
- **Green commit** (`7019100`): WHERE-clause fix + updates
`TestBackfillPathJsonFromRawHex` row 1 expectation (the pre-seeded
`'[]'` row is now correctly skipped instead of being re-decoded).

## Test results

```
ok  	github.com/corescope/ingestor	49.656s
```

## Acceptance criteria from #1119

- [x] Backfill terminates within 1 polling cycle of having no progress
to make
- [x] Migration marker `backfill_path_json_from_raw_hex_v1` written
after termination
- [x] On restart, backfill recognizes migration done and exits
immediately (existing behavior — the migration check at the top of
`BackfillPathJSONAsync` was always correct; the bug was that the marker
never got written)
- [x] Test: seed DB with N observations all having `path_json = '[]'` →
backfill runs once → no UPDATEs issued, migration marker written
- [ ] Disk write rate on idle staging drops from 2–3 MB/s to <100 KB/s —
to be verified by the user post-deploy

Fixes #1119.

---------

Co-authored-by: OpenClaw Bot <bot@openclaw.local>
2026-05-05 17:35:16 -07:00
Kpa-clawbot 90c3d46e8f ci: update go-server-coverage.json [skip ci] 2026-05-05 23:51:42 +00:00
Kpa-clawbot 0497d98bc9 ci: update go-ingestor-coverage.json [skip ci] 2026-05-05 23:51:41 +00:00
Kpa-clawbot 4973b18e9d ci: update frontend-tests.json [skip ci] 2026-05-05 23:51:40 +00:00
Kpa-clawbot d1f72c7c5a ci: update frontend-coverage.json [skip ci] 2026-05-05 23:51:39 +00:00
Kpa-clawbot 5a901e4ccf ci: update e2e-tests.json [skip ci] 2026-05-05 23:51:38 +00:00
Kpa-clawbot 45f2607f75 perf(ingestor): group commit observation INSERTs by time window (M1, refs #1115) (#1117)
## Summary

Implements **M1 from #1115**: batches observation/transmission INSERTs
into a single SQLite `BEGIN/COMMIT` window instead of fsyncing per
packet. At ~250 obs/sec this drops WAL fsync rate from ~20/s to ~1/s and
eliminates the `obs-persist skipped` / `SQLITE_BUSY` log spam that the
issue documents.

This is a **partial fix** — it ships the group-commit mechanism.
Acceptance items 6–7 (measured fsync rate / measured `obs-persist
skipped` rate at staging steady-state) require post-deploy observation,
and M2 (per-`tx_hash` observation buffering) is intentionally deferred.
The issue stays open for the user to verify on staging.

> Partial fix for #1115 — does not auto-close. Refs #1115.

## Mechanism

- `Store` gains an active `*sql.Tx`, `pendingRows` counter, `gcMu`, and
the `groupCommitMs` / `groupCommitMaxRows` knobs. `SetGroupCommit(ms,
maxRows)` enables the mode; `FlushGroupTx()` commits the in-flight tx.
- `InsertTransmission` lazily opens a tx on the first call after each
flush, then issues all writes through `tx.Stmt()` bindings of the
existing prepared statements. With `MaxOpenConns(1)` the connection is
already serialized; `gcMu` serializes group-commit state without
contention.
- A goroutine in `cmd/ingestor/main.go` calls `FlushGroupTx()` every
`groupCommitMs` ms. `pendingRows >= groupCommitMaxRows` triggers an
eager flush. `Close()` flushes before the WAL checkpoint so no rows are
lost on graceful shutdown.
- `groupCommitMs == 0` short-circuits to the legacy per-call auto-commit
path (statements bound to `s.db`, no tx) — current behavior preserved
byte-for-byte for operators who opt out.

## Config

Two new optional fields (ingestor-only), both documented in
`config.example.json`:

| Field | Default | Effect |
|---|---|---|
| `groupCommitMs` | `1000` | Flush window in ms. `0` disables batching
(legacy per-packet auto-commit). |
| `groupCommitMaxRows` | `1000` | Safety cap; when exceeded the queue
flushes immediately to bound memory and the crash-loss window. |

No DB schema change. No required config change on upgrade.

## Tests (TDD red → green visible in commits)

`cmd/ingestor/group_commit_test.go` — three assertions, written first as
the red commit:

- `TestGroupCommit_BatchesInsertsIntoOneTx` — 50 `InsertTransmission`
calls inside a wide window produce **0** commits until `FlushGroupTx`,
then exactly **1**; all 50 rows visible after flush. (This is the spec's
"50 observations → 1 SQLite write transaction" assertion.)
- `TestGroupCommit_Disabled` — `groupCommitMs=0` keeps every insert
immediately visible and `GroupCommitFlushes` never advances. (Spec's
"groupCommitMs=0 reverts to per-packet behavior" assertion.)
- `TestGroupCommit_MaxRowsForcesEarlyFlush` — cap=3, 7 inserts → 2
auto-flushes from the cap + 1 final manual flush = 3 total.

Red commit: `e2b0370` (stubs `SetGroupCommit` / `FlushGroupTx` so the
tests compile and fail on **assertions**, not import errors).
Green commit: `73f3559`.

Full ingestor suite (`go test ./...` in `cmd/ingestor`) stays green, ~49
s.

## Performance

This PR is the perf change itself. Local micro-test (the new
`TestGroupCommit_BatchesInsertsIntoOneTx`) shows the structural
property: 50 inserts → 1 commit. The fsync-rate measurement called out
in the M1 acceptance criteria (`~20/s → ~1/s` at 250 obs/sec) requires
staging deployment to confirm — that's the remaining open item that
keeps #1115 open after this merges.

No hot-path regressions: when `groupCommitMs > 0` we acquire one mutex
per insert (uncontended in the steady state — the connection was already
single-threaded via `MaxOpenConns(1)`). When `groupCommitMs == 0` the
code path is identical to before plus one nil-tx check.

## What this PR does NOT do (per spec)

- Does not collapse "30 observations of one packet" into 1 row write —
that's M2.
- Does not eliminate dual-writer contention with `cmd/server`'s
`resolved_path` writes.
- Does not change observation ordering or live broadcast latency.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
2026-05-05 16:38:43 -07:00
Kpa-clawbot 433f1b544e ci: update go-server-coverage.json [skip ci] 2026-05-05 21:42:57 +00:00
Kpa-clawbot d78ccb4bd4 ci: update go-ingestor-coverage.json [skip ci] 2026-05-05 21:42:55 +00:00
Kpa-clawbot 33ee1659de ci: update frontend-tests.json [skip ci] 2026-05-05 21:42:54 +00:00
Kpa-clawbot 51ce1b1085 ci: update frontend-coverage.json [skip ci] 2026-05-05 21:42:52 +00:00
Kpa-clawbot 25b3e4ec2b ci: update e2e-tests.json [skip ci] 2026-05-05 21:42:51 +00:00
Kpa-clawbot 50676d5e65 fix(live): #1110 node filter — autocomplete, theming, no reload (#1113)
## Summary

Fixes the broken **Filter by node** input on the Live page.

The previous implementation used a native `<datalist>` (no consistent
styling, no real autocomplete UX), only applied on `change` (Enter), and
mutated `location.hash` on commit — which the SPA router treated as a
navigation, triggering a full re-init.

## What changed

- **Markup** (`public/live.js`): replaces the `<datalist>` with a styled
custom `#liveNodeFilterDropdown` and adds combobox/listbox ARIA wiring.
- **Styling** (`public/live.css`): new `.live-node-filter-input` rules
use `color-mix` on `var(--text)` for the background and `var(--border)`
/ `var(--text)` for border + foreground — fully theme-aware. Dropdown
uses `var(--surface-1)` + `var(--border)`.
- **Behavior**: 200 ms debounced `/api/nodes/search` call as the user
types. Suggestions render with name + 8-char pubkey prefix. Clicking a
suggestion (`mousedown` so it beats blur) sets the filter to the pubkey.
- **No reload**: `applyFilterFromInput` and the clear button now use
`history.replaceState` instead of mutating `location.hash`, so the SPA
router never re-runs and the page never reloads. Enter is
`preventDefault`-ed and either selects the highlighted suggestion or
commits the typed text.
- **Keyboard**: ArrowUp/Down navigate suggestions, Esc closes, Enter
selects.

## TDD

Per `AGENTS.md`, the failing E2E test landed first (commit `74f3e92`),
then the fix made it green (commit `a5c5c65`).

The test file `test-1110-live-filter.js` (and an integrated block in
`test-e2e-playwright.js`) asserts:

1. The input's computed `background-color` is **not** hardcoded white
when `data-theme="dark"` is set.
2. The input is not vastly larger than the surrounding toolbar row.
3. Typing `"te"` shows a visible `#liveNodeFilterDropdown` with at least
one `.live-node-filter-option`.
4. Clicking a suggestion sets `_liveGetNodeFilterKeys()` to a non-empty
list **without** reloading the page (verified via a `window.__m` marker
that survives) and **without** navigating away from `#/live`.
5. Pressing **Enter** in the filter input never reloads or navigates.

### How to run the E2E

```
go build -o /tmp/corescope-server ./cmd/server
/tmp/corescope-server -port 13581 -db test-fixtures/e2e-fixture.db -public public &
CHROMIUM_PATH=/usr/bin/chromium-browser BASE_URL=http://localhost:13581 \
  node test-1110-live-filter.js
# 4/4 passed
```

## Acceptance criteria from #1110

- [x] Filter input visually matches Live page toolbar (theme-aware bg,
border, padding)
- [x] Typing 1+ characters shows dropdown of matching node names
- [x] Selecting a suggestion filters the live feed immediately
- [x] Clearing input restores unfiltered view
- [x] No page reload on any interaction with the input
- [x] E2E test asserts: type → suggestions appear → click suggestion →
feed filters → no navigation

Fixes #1110

---------

Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>
2026-05-05 14:28:12 -07:00
Kpa-clawbot 2634e4b4b7 ci: update go-server-coverage.json [skip ci] 2026-05-05 21:12:56 +00:00
Kpa-clawbot 33fab978f4 ci: update go-ingestor-coverage.json [skip ci] 2026-05-05 21:12:55 +00:00
Kpa-clawbot f8e9964915 ci: update frontend-tests.json [skip ci] 2026-05-05 21:12:55 +00:00
Kpa-clawbot 7e6fb013a3 ci: update frontend-coverage.json [skip ci] 2026-05-05 21:12:54 +00:00
Kpa-clawbot 4b91035f4d ci: update e2e-tests.json [skip ci] 2026-05-05 21:12:53 +00:00
Kpa-clawbot 12d96a9d15 fix(#1111): hide My Channels section entirely when empty (#1112)
Fixes #1111.

## Problem
When the user has no PSK channels added, `public/channels.js` still
renders the "My Channels 🖥️ (this browser)" section header plus an
empty-state placeholder ("No channels yet — click [+ Add Channel] to
add one."). The section should not exist in the DOM at all when empty.

## Fix
Wrap the entire My Channels section render in a `mine.length > 0`
guard. When `mine.length === 0`: no section, no header, no placeholder.

## TDD
- **Red commit** (`b8bf938`): adds `test-channel-issue-1111-e2e.js`,
  which fails on the current renderer because the section always
  emits — the test reproduces the bug.
- **Green commit** (`776653d`): the conditional render in
  `public/channels.js` makes the test pass.

## E2E
New test: `test-channel-issue-1111-e2e.js` (wired into the deploy
workflow alongside the other channel E2Es).
- Case 1: clear `localStorage` → asserts `.ch-section-mychannels`
  absent and no "My Channels" text in `#chList`.
- Case 2: seed `corescope_channel_keys` with one PSK key → asserts
  `.ch-section-mychannels` exists with the "My Channels" header.

## Acceptance criteria
- [x] No "My Channels" section when empty (no header, no placeholder)
- [x] Section + header + channel row render with ≥1 stored PSK key
- [x] E2E covers both states

## Performance
None — single conditional around an existing render path.

---------

Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com>
Co-authored-by: clawbot <bot@kpabap.invalid>
2026-05-05 14:01:09 -07:00
Kpa-clawbot 10caa71ccf ci: update go-server-coverage.json [skip ci] 2026-05-05 19:59:14 +00:00
Kpa-clawbot 43321e8eae ci: update go-ingestor-coverage.json [skip ci] 2026-05-05 19:59:13 +00:00
Kpa-clawbot b62d00f4fd ci: update frontend-tests.json [skip ci] 2026-05-05 19:59:12 +00:00