ci: frontend coverage from every E2E suite, merged across the shards (#2204)

Fixes #2203.

## Problem

The frontend coverage badge (35.16% on run 38413268001) came from two
sources on shard 2: one dump part-way through `test-e2e-playwright.js`
and the `collect-frontend-coverage.js` tour. The other E2E suites run
against the instrumented build but never save `window.__coverage__`,
including the suites written for single pages (`path-inspector.js`
4.72%, `touch-gestures.js` 14% on that run).

## Change

1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with
fake pages, contexts and browsers (no Chromium needed). On the
placeholder module the first assertion fails.
2. `c00bbb12` `feat(e2e):` the collector.
- `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that
wraps `chromium.launch` from `playwright`. `playwright` re-exports
`playwright-core`, so a suite's own `const { chromium } =
require('playwright')` gets the same, wrapped object.
- `scripts/e2e-coverage.js` reads `window.__coverage__` before every
call that replaces or ends a page's document: `goto`, `reload`,
`setContent`, `goBack`, `goForward`, `close`, and context and browser
`close`. Pages from `newPage`, `newContext` and the context `page` event
are all tracked.
- Each read zeroes the page's counters, so a document read twice is not
counted twice. Maps travel once per file; later reads carry counts only.
- The page returns a JSON string. Returning the object cost about 10 s
per suite, because Playwright serialises it node by node.
- A read is given up after 5 s, so a busy page cannot hold up a suite's
cleanup.
- On exit the hook writes one file per suite to `.nyc_output/`, or
nothing without coverage. A failed write is logged and never changes the
exit code.
3. `64487c4c` `ci:` the workflow.
- On master pushes the E2E run step sets `NODE_OPTIONS=--require
./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are
unchanged.
- Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly
the same maps) and uploads `coverage-shard.json` with its results
instead of `.badges/`.
- The `e2e-test` job merges the three shards and makes the frontend
badge (it used to come from shard 2 alone).
- The merge, setup-node and badge steps are `continue-on-error`. When
`nyc report` gives no statement percentage the step warns and writes no
badge, so a registry hiccup cannot publish 0%.
- `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`:
the hook covers that suite completely, including the tests after the old
dump. `collect-frontend-coverage.js` stays as an extra source.

## Performance

Measured locally against an instrumented server (Windows, Chrome):

| Suite | Without hook | With hook |
|---|---|---|
| `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s |
| `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s |
| `test-e2e-playwright.js` (stops at the same local-fixture failure in
both runs) | 52 s | 52 s |
| `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s |

The hook only loads on master pushes.

## Testing

- `tests/unit/test-e2e-coverage.js` covers:
  - reads before each document-replacing call and before close;
  - pages opened by the page itself;
  - context and browser close;
  - summing without double counts;
  - a failed read, and a read that never answers;
  - the file writer, including a failing write.
- Nine deliberate breaks of the library each turn it red:
  - dropping the counter reset;
- dropping the `page` event, the `goto` read, the `goForward` read, the
`close` read or the browser `close` read;
  - removing the timeout or the write guard, or writing an empty store.
- The test also fails if it stops before its end. Without that check, a
read that never returns would let node exit 0 half-way.
- Real suites with the hook, measured on their own (the badge run had
all suites):

| File | Badge run 38413268001 | Own suite with the hook |
|---|---|---|
| `path-inspector.js` | 4.72% | 52.75% |
| `touch-gestures.js` | 14% | 72.96% |

- Exit codes and pass counts were unchanged with the hook for these
suites (the first three were run before the 5 s read timeout was added):
  - `test-slideover-1056-e2e.js` (6 contexts);
  - `test-a11y-axe-1668.js` (`@axe-core/playwright`);
  - `test-issue-1150-404-state-e2e.js`;
  - the three above.
- The badge step, run locally against two merged shard files:
  - It gives the same per-file numbers as `nyc report` on the inputs.
- With no shard coverage, or with `nyc report` failing, it warns and
writes no badge.
- `sh test-all.sh` passes locally, apart from
`test-issue-1956-release-routing.js`, which needs the workflow's
`GITHUB_REF_NAME` and was skipped. The test inventory classifies 321
suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as
on master.

**Not tested:** the shard and badge steps in CI. They only run on a
master push, so the first master run after merge is the real test. I do
not know what the merged number will be. I will compare the badge with
the merged report in that run's `e2e-test` log.

**Not counted:** a full page load the suite does not start through one
of the wrapped calls, such as a link that leaves the SPA or
`location.reload()` from page code.

## Not in this PR

- The `tests/unit/` suites, which load frontend files through
`vm.createContext` without instrumentation (noted in #2203).
- Coverage on PR runs. The overhead above is small, so this could follow
if wanted.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

scripts/e2e-coverage-hook.js

@@ -0,0 +1,18 @@
+'use strict';
+// Preload for the E2E shards (#2203): `NODE_OPTIONS=--require ./scripts/e2e-coverage-hook.js`.
+// Every Chromium the suite launches is tracked by scripts/e2e-coverage.js, and
+// the summed coverage is written to .nyc_output/ when the process exits. A
+// suite that never loads an instrumented page writes nothing.
+
+const fs = require('fs');
+const path = require('path');
+const { chromium } = require('playwright');
+const { createCollector, writeCoverage } = require('./e2e-coverage');
+
+const collector = createCollector();
+collector.wrapLaunch(chromium);
+
+process.on('exit', () => {
+  writeCoverage(fs, path, collector.store, path.resolve(__dirname, '..', '.nyc_output'),
+    path.basename(process.argv[1] || 'node', '.js'), process.pid);
+});

scripts/e2e-coverage.js

@@ -0,0 +1,154 @@
+'use strict';
+// Collects window.__coverage__ from every page a Playwright E2E suite opens
+// (#2203). Loaded through scripts/e2e-coverage-hook.js; kept free of
+// Playwright and fs so tests/unit/test-e2e-coverage.js can drive it with fakes.
+//
+// A page's counters live until its document is replaced, so they are read
+// (and zeroed) before every call that can replace or end the document. A
+// full load the suite does not start through one of these calls, such as a
+// click on a link that leaves the SPA, is not counted.
+
+const PAGE_CALLS = ['goto', 'reload', 'setContent', 'goBack', 'goForward', 'close'];
+
+// A read must never hold up the call it runs before: a page stuck in a busy
+// loop would otherwise keep browser.close() in a suite's cleanup waiting for
+// as long as the loop runs. Past this, the read is given up.
+const READ_TIMEOUT_MS = 5000;
+
+// Runs inside the page. Returns the hit counts since the last read and zeroes
+// them, so a document read twice is not counted twice. A file the caller has
+// not seen yet comes back whole (with its statement, function and branch maps).
+// The result is a JSON string: Playwright serialises a returned object node by
+// node, which took about 10 s for the roughly 6 MB of maps of the first read.
+function readAndReset(known) {
+  const cov = window.__coverage__;
+  if (!cov) return null;
+  const seen = new Set(known);
+  const out = {};
+  for (const file of Object.keys(cov)) {
+    const fc = cov[file];
+    out[file] = seen.has(file) ? { s: fc.s, f: fc.f, b: fc.b } : fc;
+  }
+  const json = JSON.stringify(out);
+  for (const file of Object.keys(cov)) {
+    const fc = cov[file];
+    for (const k in fc.s) fc.s[k] = 0;
+    for (const k in fc.f) fc.f[k] = 0;
+    for (const k in fc.b) fc.b[k].fill(0);
+  }
+  return json;
+}
+
+// Adds a snapshot from readAndReset to store, summing the hit counts.
+function mergeInto(store, snapshot) {
+  for (const file of Object.keys(snapshot)) {
+    const add = snapshot[file];
+    const into = store[file];
+    if (!into) {
+      store[file] = add;
+      continue;
+    }
+    for (const k in add.s) into.s[k] = (into.s[k] || 0) + add.s[k];
+    for (const k in add.f) into.f[k] = (into.f[k] || 0) + add.f[k];
+    for (const k in add.b) {
+      const sum = into.b[k] || add.b[k].map(() => 0);
+      add.b[k].forEach((n, i) => { sum[i] = (sum[i] || 0) + n; });
+      into.b[k] = sum;
+    }
+  }
+}
+
+function createCollector({ readTimeoutMs = READ_TIMEOUT_MS } = {}) {
+  const store = {};
+  const pages = new WeakSet();
+  const contexts = new WeakSet();
+
+  async function collect(page) {
+    if (page.isClosed()) return;
+    let timer;
+    const timeout = new Promise(resolve => { timer = setTimeout(resolve, readTimeoutMs, null); });
+    let snapshot;
+    try {
+      snapshot = await Promise.race([page.evaluate(readAndReset, Object.keys(store)), timeout]);
+    } catch {
+      return; // page crashed or navigated away mid-read: nothing to add
+    } finally {
+      clearTimeout(timer);
+    }
+    if (snapshot) mergeInto(store, JSON.parse(snapshot));
+  }
+
+  function readFirst(obj, names, read) {
+    for (const name of names) {
+      const orig = obj[name];
+      if (typeof orig !== 'function') continue;
+      obj[name] = async function (...args) {
+        await read();
+        return orig.apply(this, args);
+      };
+    }
+  }
+
+  function trackPage(page) {
+    if (pages.has(page)) return page;
+    pages.add(page);
+    readFirst(page, PAGE_CALLS, () => collect(page));
+    return page;
+  }
+
+  function trackContext(context) {
+    if (contexts.has(context)) return context;
+    contexts.add(context);
+    context.pages().forEach(trackPage);
+    context.on('page', trackPage);
+    readFirst(context, ['close'], () => Promise.all(context.pages().map(collect)));
+    const newPage = context.newPage;
+    context.newPage = async function (...args) {
+      return trackPage(await newPage.apply(this, args));
+    };
+    return context;
+  }
+
+  function trackBrowser(browser) {
+    const newContext = browser.newContext;
+    browser.newContext = async function (...args) {
+      return trackContext(await newContext.apply(this, args));
+    };
+    const newPage = browser.newPage;
+    browser.newPage = async function (...args) {
+      const page = await newPage.apply(this, args);
+      trackContext(page.context());
+      return trackPage(page);
+    };
+    readFirst(browser, ['close'], () =>
+      Promise.all(browser.contexts().flatMap(c => c.pages()).map(collect)));
+    return browser;
+  }
+
+  function wrapLaunch(browserType) {
+    const launch = browserType.launch;
+    browserType.launch = async function (...args) {
+      return trackBrowser(await launch.apply(this, args));
+    };
+  }
+
+  return { store, collect, trackPage, trackContext, trackBrowser, wrapLaunch };
+}
+
+// Writes store as one nyc input file, or nothing when it is empty. Never
+// throws: it runs in an exit handler, where a throw would change the suite's
+// exit code. Returns the file written, or null.
+function writeCoverage(fs, path, store, outDir, suite, pid) {
+  if (Object.keys(store).length === 0) return null;
+  try {
+    fs.mkdirSync(outDir, { recursive: true });
+    const file = path.join(outDir, `e2e-${suite}-${pid}.json`);
+    fs.writeFileSync(file, JSON.stringify(store));
+    return file;
+  } catch (e) {
+    console.error(`e2e-coverage: not written: ${e.message}`);
+    return null;
+  }
+}
+
+module.exports = { readAndReset, mergeInto, createCollector, writeCoverage };