dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-10-15 21:29:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci: frontend coverage from every E2E suite, merged across the shards (#2204)

Browse Source 
Fixes #2203.

## Problem

The frontend coverage badge (35.16% on run 38413268001) came from two
sources on shard 2: one dump part-way through `test-e2e-playwright.js`
and the `collect-frontend-coverage.js` tour. The other E2E suites run
against the instrumented build but never save `window.__coverage__`,
including the suites written for single pages (`path-inspector.js`
4.72%, `touch-gestures.js` 14% on that run).

## Change

1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with
fake pages, contexts and browsers (no Chromium needed). On the
placeholder module the first assertion fails.
2. `c00bbb12` `feat(e2e):` the collector.
- `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that
wraps `chromium.launch` from `playwright`. `playwright` re-exports
`playwright-core`, so a suite's own `const { chromium } =
require('playwright')` gets the same, wrapped object.
- `scripts/e2e-coverage.js` reads `window.__coverage__` before every
call that replaces or ends a page's document: `goto`, `reload`,
`setContent`, `goBack`, `goForward`, `close`, and context and browser
`close`. Pages from `newPage`, `newContext` and the context `page` event
are all tracked.
- Each read zeroes the page's counters, so a document read twice is not
counted twice. Maps travel once per file; later reads carry counts only.
- The page returns a JSON string. Returning the object cost about 10 s
per suite, because Playwright serialises it node by node.
- A read is given up after 5 s, so a busy page cannot hold up a suite's
cleanup.
- On exit the hook writes one file per suite to `.nyc_output/`, or
nothing without coverage. A failed write is logged and never changes the
exit code.
3. `64487c4c` `ci:` the workflow.
- On master pushes the E2E run step sets `NODE_OPTIONS=--require
./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are
unchanged.
- Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly
the same maps) and uploads `coverage-shard.json` with its results
instead of `.badges/`.
- The `e2e-test` job merges the three shards and makes the frontend
badge (it used to come from shard 2 alone).
- The merge, setup-node and badge steps are `continue-on-error`. When
`nyc report` gives no statement percentage the step warns and writes no
badge, so a registry hiccup cannot publish 0%.
- `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`:
the hook covers that suite completely, including the tests after the old
dump. `collect-frontend-coverage.js` stays as an extra source.

## Performance

Measured locally against an instrumented server (Windows, Chrome):

| Suite | Without hook | With hook |
|---|---|---|
| `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s |
| `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s |
| `test-e2e-playwright.js` (stops at the same local-fixture failure in
both runs) | 52 s | 52 s |
| `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s |

The hook only loads on master pushes.

## Testing

- `tests/unit/test-e2e-coverage.js` covers:
  - reads before each document-replacing call and before close;
  - pages opened by the page itself;
  - context and browser close;
  - summing without double counts;
  - a failed read, and a read that never answers;
  - the file writer, including a failing write.
- Nine deliberate breaks of the library each turn it red:
  - dropping the counter reset;
- dropping the `page` event, the `goto` read, the `goForward` read, the
`close` read or the browser `close` read;
  - removing the timeout or the write guard, or writing an empty store.
- The test also fails if it stops before its end. Without that check, a
read that never returns would let node exit 0 half-way.
- Real suites with the hook, measured on their own (the badge run had
all suites):

| File | Badge run 38413268001 | Own suite with the hook |
|---|---|---|
| `path-inspector.js` | 4.72% | 52.75% |
| `touch-gestures.js` | 14% | 72.96% |

- Exit codes and pass counts were unchanged with the hook for these
suites (the first three were run before the 5 s read timeout was added):
  - `test-slideover-1056-e2e.js` (6 contexts);
  - `test-a11y-axe-1668.js` (`@axe-core/playwright`);
  - `test-issue-1150-404-state-e2e.js`;
  - the three above.
- The badge step, run locally against two merged shard files:
  - It gives the same per-file numbers as `nyc report` on the inputs.
- With no shard coverage, or with `nyc report` failing, it warns and
writes no badge.
- `sh test-all.sh` passes locally, apart from
`test-issue-1956-release-routing.js`, which needs the workflow's
`GITHUB_REF_NAME` and was skipped. The test inventory classifies 321
suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as
on master.

**Not tested:** the shard and badge steps in CI. They only run on a
master push, so the first master run after merge is the real test. I do
not know what the merged number will be. I will compare the badge with
the merged report in that run's `e2e-test` log.

**Not counted:** a full page load the suite does not start through one
of the wrapped calls, such as a link that leaves the SPA or
`location.reload()` from page code.

## Not in this PR

- The `tests/unit/` suites, which load frontend files through
`vm.createContext` without instrumentation (noted in #2203).
- Coverage on PR runs. The overhead above is small, so this could follow
if wanted.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
efiten
2026-10-13 22:04:03 +02:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 59708ee72e
commit fbeb9e4d71
6 changed files with 451 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/deploy.yml
+55 -21
View File
@@ -543,6 +543,10 @@ jobs:
done
- name: Run Playwright E2E tests (fail-fast)
env:
# On master pushes every suite records the frontend coverage of the
# pages it opens into .nyc_output/ (#2203). Off on PRs to keep them fast.
NODE_OPTIONS: ${{ github.event_name == 'push' && '--require ./scripts/e2e-coverage-hook.js' || '' }}
run: |
# suite <shard> <file> [VAR=value ...]: runs tests/e2e/<file> with
# exactly those variables, but only on its own shard. The shard numbers
@@ -720,29 +724,26 @@ jobs:
done
echo "3 passed"
# Shard 2 runs test-e2e-playwright.js, the one suite that writes
# .nyc_output, so the coverage collection and its badge live there too.
# An extra coverage source on top of the suites: a tour of the main
# page groups. Its files land in .nyc_output/ beside the suites' files.
- name: Collect frontend coverage (parallel)
if: success() && github.event_name == 'push' && matrix.shard == 2
run: |
BASE_URL=http://localhost:13581 node scripts/collect-frontend-coverage.js 2>&1 | tee fe-coverage-output.txt || true
- name: Generate frontend coverage badge
if: success()
# One file per suite is about 6 MB, mostly the same maps, so the shard
# merges them before upload. e2e-test merges the three shards and makes
# the frontend coverage badge.
# Coverage never decides the outcome: a failure here must not skip the
# shard upload below (if: success()) and so fail the E2E gate.
- name: Merge this shard's frontend coverage
if: success() && github.event_name == 'push'
continue-on-error: true
run: |
# .badges/ is tracked on master, so the checkout already holds the
# old badge files; the shard upload would ship them as if this run
# had produced them (#2136). Start empty.
rm -rf .badges && mkdir .badges
if [ -f .nyc_output/frontend-coverage.json ] || [ -f .nyc_output/e2e-coverage.json ]; then
npx nyc report --reporter=text-summary --reporter=text 2>&1 | tee fe-report.txt
FE_COVERAGE=$(grep 'Statements' fe-report.txt | head -1 | grep -oP '[\d.]+(?=%)' || echo "0")
FE_COVERAGE=${FE_COVERAGE:-0}
FE_COLOR="red"
[ "$(echo "$FE_COVERAGE > 50" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="yellow"
[ "$(echo "$FE_COVERAGE > 80" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="brightgreen"
echo "{\"schemaVersion\":1,\"label\":\"frontend coverage\",\"message\":\"${FE_COVERAGE}%\",\"color\":\"${FE_COLOR}\"}" > .badges/frontend-coverage.json
echo "## Frontend: ${FE_COVERAGE}% coverage" >> $GITHUB_STEP_SUMMARY
if ls .nyc_output/*.json >/dev/null 2>&1; then
npx nyc merge .nyc_output coverage-shard.json
else
echo "::warning::no frontend coverage was recorded on shard ${{ matrix.shard }}"
fi
- name: Stop test server
@@ -764,7 +765,7 @@ jobs:
name: e2e-shard-${{ matrix.shard }}
path: |
e2e-output.txt
.badges/
coverage-shard.json
retention-days: 1
if-no-files-found: ignore
include-hidden-files: true
@@ -803,10 +804,9 @@ jobs:
ls shards
[ "$(ls -d shards/e2e-shard-* | wc -l)" -eq 3 ] || { echo "::error::expected results from 3 shards"; exit 1; }
cat shards/e2e-shard-*/e2e-output.txt > e2e-output.txt
# Start empty, as in the shards: the checkout holds master's old
# .badges/ files, which would otherwise go out in e2e-badges (#2136).
# Start empty: the checkout holds master's old .badges/ files,
# which would otherwise go out in e2e-badges (#2136).
rm -rf .badges && mkdir .badges
cp shards/e2e-shard-*/.badges/*.json .badges/ 2>/dev/null || true
# Aggregate per-suite PASS/FAIL across every test-*-e2e.js summary.
# The previous regex (grep -oP '[0-9]+(?=/)' | tail -1) caught a
# stray digits-before-slash like the '2' in '2/3 tests passed' from
@@ -824,6 +824,40 @@ jobs:
echo "{\"schemaVersion\":1,\"label\":\"e2e tests\",\"message\":\"${E2E_MSG}\",\"color\":\"${E2E_COLOR}\"}" > .badges/e2e-tests.json
echo "E2E: ${E2E_MSG} ($(grep -c '^=== E2E SUITE' e2e-output.txt) suite runs)" | tee -a "$GITHUB_STEP_SUMMARY"
- name: Set up Node.js 22
if: github.event_name == 'push'
continue-on-error: true
uses: actions/setup-node@v5
with:
node-version: '22'
# Coverage from every suite on every shard (#2203), merged into one
# number. Only master pushes record it, and it never fails the gate.
- name: Generate frontend coverage badge
if: github.event_name == 'push'
continue-on-error: true
run: |
mkdir -p .nyc_output
for f in shards/e2e-shard-*/coverage-shard.json; do
[ -f "$f" ] && cp "$f" ".nyc_output/$(basename "$(dirname "$f")").json"
done
if ! ls .nyc_output/*.json >/dev/null 2>&1; then
echo "::warning::no shard recorded frontend coverage; the badge keeps its previous value"
exit 0
fi
echo "Frontend coverage from: $(ls .nyc_output | paste -sd ' ' -)"
npx --yes nyc@18.0.0 report --reporter=text-summary --reporter=text 2>&1 | tee fe-report.txt
FE_COVERAGE=$(grep 'Statements' fe-report.txt | head -1 | grep -oP '[\d.]+(?=%)' || true)
if [ -z "$FE_COVERAGE" ]; then
echo "::warning::nyc report gave no statement coverage; the badge keeps its previous value"
exit 0
fi
FE_COLOR="red"
[ "$(echo "$FE_COVERAGE > 50" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="yellow"
[ "$(echo "$FE_COVERAGE > 80" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="brightgreen"
echo "{\"schemaVersion\":1,\"label\":\"frontend coverage\",\"message\":\"${FE_COVERAGE}%\",\"color\":\"${FE_COLOR}\"}" > .badges/frontend-coverage.json
echo "## Frontend: ${FE_COVERAGE}% coverage" >> "$GITHUB_STEP_SUMMARY"
- name: Upload E2E badges
uses: actions/upload-artifact@v6
with: