Files
meshcore-analyzer/.github/workflows/deploy.yml
T
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00

1423 lines
71 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: CI/CD Pipeline
# Documentation-only changes still trigger this workflow, on purpose. Skipping
# it at the trigger level leaves every required check in a pending state that
# can never arrive, and the pull request can then never merge. GitHub documents
# exactly this: a workflow skipped by path filtering keeps its checks pending
# and blocks the merge, while a JOB skipped by an if: conditional reports
# Success and does not. So the filtering lives on the jobs below, not here.
#
# The scope check only applies to pull requests. A push to master always runs
# the full pipeline, which keeps :edge built for every master commit. That
# matters: release-fast-path.yml re-tags :edge to :vX.Y.Z only when the :edge
# revision label matches the tagged commit, so a master commit without an image
# breaks tagging.
on:
push:
branches: [master]
pull_request:
branches: [master]
workflow_dispatch:
inputs:
images_published:
description: 'Release fast path already published the tag images'
type: boolean
default: false
permissions:
contents: read
packages: write
concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
STAGING_COMPOSE_FILE: docker-compose.staging.yml
STAGING_SERVICE: staging-go
STAGING_CONTAINER: corescope-staging-go
# Pipeline. The test jobs and the image build run side by side; only the GHCR
# push waits for all of them:
# changes ─┬─ go-test ─────────────────────────┐
# ├─ race-test (ingestor changes only) ─┤
# ├─ e2e-shard ×3 ── e2e-test (gate) ───┼─ build-and-publish → deploy, publish
# └─ image-check (two-arch build) ──────┘
# PRs stop after build-and-publish (no GHCR push). Master continues to deploy + badges.
jobs:
# ───────────────────────────────────────────────────────────────
# 0. Change scope — decides whether the expensive jobs below run.
# ───────────────────────────────────────────────────────────────
changes:
name: "🔎 Change scope"
runs-on: ubuntu-latest
outputs:
code: ${{ steps.scope.outputs.code }}
ingestor: ${{ steps.scope.outputs.ingestor }}
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Decide whether anything but documentation changed
id: scope
run: |
set -euo pipefail
# Only pull requests are scoped. Pushes and dispatches always count as
# code, so master keeps producing an :edge image and tag builds are
# never skipped.
if [ "${{ github.event_name }}" != "pull_request" ]; then
echo "code=true" >> "$GITHUB_OUTPUT"
echo "ingestor=true" >> "$GITHUB_OUTPUT"
echo "event=${{ github.event_name }} -> full pipeline"
exit 0
fi
CHANGED=$(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.event.pull_request.head.sha }}")
# Empty diff means something is off; run everything rather than guess.
if [ -z "$CHANGED" ]; then
echo "code=true" >> "$GITHUB_OUTPUT"
echo "ingestor=true" >> "$GITHUB_OUTPUT"
echo "empty diff -> full pipeline"
exit 0
fi
echo "Changed files:"
echo "$CHANGED" | sed 's/^/ /'
if echo "$CHANGED" | grep -qvE '(^docs/|[.]md$|^LICENSE$)'; then
echo "code=true" >> "$GITHUB_OUTPUT"
echo "-> non-documentation files changed, full pipeline"
else
echo "code=false" >> "$GITHUB_OUTPUT"
echo "-> documentation only, heavy jobs skip (and report Success)"
fi
# The race detector is its own job, and it only earns its ten minutes
# when the code it inspects changed. A frontend or docs PR cannot
# introduce a data race in the ingestor.
if echo "$CHANGED" | grep -qE '^cmd/ingestor/.*[.]go$'; then
echo "ingestor=true" >> "$GITHUB_OUTPUT"
echo "-> ingestor Go files changed, race detector runs"
else
echo "ingestor=false" >> "$GITHUB_OUTPUT"
echo "-> no ingestor Go changes, race detector skips"
fi
# ───────────────────────────────────────────────────────────────
# 1a. Race detector (ingestor) — its own job on purpose
# ───────────────────────────────────────────────────────────────
# Runs beside "Go Build & Test" rather than inside it, so its ten-odd
# minutes overlap the E2E job instead of extending the critical path, and
# only when ingestor Go files changed. A frontend or documentation PR cannot
# introduce a data race here, and paying for the check on every PR is what
# gets a check switched off again.
#
# The server has -race in its own step already; this closes the same gap for
# the ingestor, which carries an atomic.Pointer snapshot (the region key
# set) whose safety is an argument until something checks it.
race-test:
name: "🏁 Race detector (ingestor)"
runs-on: ubuntu-latest
needs: [changes]
if: needs.changes.outputs.ingestor == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Set up Go 1.27
uses: actions/setup-go@v6
with:
go-version: '1.27'
# All modules: a -race build of a cgo package is the slowest thing in
# this pipeline, and setup-go caches ~/.cache/go-build as well as the
# module cache, so a key that misses on an unrelated module hurts most
# here. Same reasoning as go-test below.
cache-dependency-path: "**/go.sum"
- name: go test -race
run: |
set -e -o pipefail
cd cmd/ingestor
go test -timeout 30m -race ./...
# ───────────────────────────────────────────────────────────────
# 1. Go Build & Test
# ───────────────────────────────────────────────────────────────
go-test:
name: "✅ Go Build & Test"
runs-on: ubuntu-latest
needs: [changes]
if: needs.changes.outputs.code == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Clean Go module cache
run: rm -rf ~/go/pkg/mod 2>/dev/null || true
- name: Set up Go 1.27
uses: actions/setup-go@v6
with:
go-version: '1.27'
# All modules, not just two: the cgo SQLite build is expensive to
# redo, and setup-go caches ~/.cache/go-build as well as the module
# cache, so a key that misses on an unrelated module hurts.
cache-dependency-path: "**/go.sum"
- name: Run JS unit tests
run: sh test-all.sh
- name: Enforce gofmt + go vet (issue #1859)
run: |
set -e
# gofmt: every tracked *.go file.
unformatted=$(gofmt -l $(git ls-files '*.go'))
if [ -n "$unformatted" ]; then
echo "::error::gofmt required on these files:"
echo "$unformatted"
exit 1
fi
echo "gofmt: clean"
# go vet: multi-module repo, so vet each module in turn.
for mod in $(git ls-files '*go.mod' | sed 's#/go.mod##'); do
echo "== go vet $mod =="
( cd "$mod" && go vet ./... )
done
- name: Build and test Go server (with coverage)
run: |
set -e -o pipefail
cd cmd/server
go build .
# -race gates PR #1208's atomic.Pointer migration: the race-detector
# is what makes path_inspect_atomic_race_test.go actually assert.
go test -timeout 20m -race -coverprofile=server-coverage.out ./... 2>&1 | tee server-test.log
# The e2etest build (fake mailer + /__e2e/last-mail) has its own tests.
go test -tags e2etest -count=1 -run 'E2E' .
echo "--- Go Server Coverage ---"
go tool cover -func=server-coverage.out | tail -1
- name: Build and test Go ingestor (with coverage)
run: |
set -e -o pipefail
cd cmd/ingestor
go build .
go test -timeout 20m -coverprofile=ingestor-coverage.out ./... 2>&1 | tee ingestor-test.log
echo "--- Go Ingestor Coverage ---"
go tool cover -func=ingestor-coverage.out | tail -1
- name: Build and test channel library + decrypt CLI
run: |
set -e -o pipefail
cd internal/channel
go test ./...
echo "--- Channel library tests passed ---"
cd ../../cmd/decrypt
go build -ldflags="-s -w" -o corescope-decrypt .
go test ./...
echo "--- Decrypt CLI tests passed ---"
- name: Test user-management modules (users, mailer)
run: |
set -e -o pipefail
cd internal/users && go test -race ./...
cd ../mailer && go test -race ./...
echo "--- users + mailer tests passed ---"
- name: Test migrate CLI + dbschema
run: |
set -e -o pipefail
# Neither module had CI test execution before the mattn/go-sqlite3
# migration; both open the database, and dbschema owns the migrations.
cd internal/dbschema
go test ./...
cd ../../cmd/migrate
go build -o /dev/null .
go test ./...
echo "--- migrate + dbschema tests passed ---"
- name: Verify Dockerfile COPY invariants (issue #1316)
run: bash scripts/check-dockerfile-internal-pkgs.sh
- name: Staging disk-monitor unit tests (issue #1684)
run: bash scripts/staging/test-disk-monitor.sh
- name: Container service-user selection unit tests
run: bash docker/test-select-service-user.sh
- name: QA SQL parameter-binding unit tests (issue #1977)
run: bash qa/scripts/test-blacklist-sql.sh
- name: Lint CSS variables (issue #1128)
run: |
set -e
node scripts/check-css-vars.js
node scripts/test-check-css-vars.js
- name: 🛡️ Preflight XSS gate — actual --diff check (PR only)
# The fixture self-test above (test-preflight-xss-gate.js) only
# asserts the script's behavior against fixtures. It does NOT scan
# the PR's own changes. This step closes that gap by running the
# gate against added lines in public/**/*.{js,html} on the PR.
# Gate is PR-scoped only (per djb finding: merge commits would
# slip an opt-out otherwise). Master pushes skip this step.
if: github.event_name == 'pull_request'
env:
PR_BODY: ${{ github.event.pull_request.body }}
PREFLIGHT_PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ' ') }}
run: |
set -e
git fetch origin master --depth=50 2>&1 | tail -3 || true
# Materialize PR body to a file for the opt-out parser.
printf '%s' "$PR_BODY" > /tmp/pr-body.md
PREFLIGHT_PR_BODY=/tmp/pr-body.md bash scripts/check-xss-sinks.sh --diff origin/master
- name: 🧹 Frontend lint (eslint no-undef) — issue #1342
run: |
set -e
# Use eslint@8 (legacy .eslintrc.json). Don't migrate to flat-config / eslint@9.
# --no-save: avoid touching package.json / no committed node_modules.
npm install --no-save --no-audit --no-fund eslint@8
npx eslint public/*.js
- name: Verify proto syntax
run: |
set -e
sudo apt-get update -qq
sudo apt-get install -y protobuf-compiler
for proto in proto/*.proto; do
echo " ✓ $(basename "$proto")"
protoc --proto_path=proto --descriptor_set_out=/dev/null "$proto"
done
echo "✅ All .proto files are syntactically valid"
- name: Generate Go coverage badges
if: success()
run: |
mkdir -p .badges
SERVER_COV="0"
if [ -f cmd/server/server-coverage.out ]; then
SERVER_COV=$(cd cmd/server && go tool cover -func=server-coverage.out | tail -1 | grep -oP '[\d.]+(?=%)')
fi
SERVER_COLOR="red"
if [ "$(echo "$SERVER_COV >= 80" | bc -l 2>/dev/null)" = "1" ]; then SERVER_COLOR="green"
elif [ "$(echo "$SERVER_COV >= 60" | bc -l 2>/dev/null)" = "1" ]; then SERVER_COLOR="yellow"; fi
echo "{\"schemaVersion\":1,\"label\":\"go server coverage\",\"message\":\"${SERVER_COV}%\",\"color\":\"${SERVER_COLOR}\"}" > .badges/go-server-coverage.json
INGESTOR_COV="0"
if [ -f cmd/ingestor/ingestor-coverage.out ]; then
INGESTOR_COV=$(cd cmd/ingestor && go tool cover -func=ingestor-coverage.out | tail -1 | grep -oP '[\d.]+(?=%)')
fi
INGESTOR_COLOR="red"
if [ "$(echo "$INGESTOR_COV >= 80" | bc -l 2>/dev/null)" = "1" ]; then INGESTOR_COLOR="green"
elif [ "$(echo "$INGESTOR_COV >= 60" | bc -l 2>/dev/null)" = "1" ]; then INGESTOR_COLOR="yellow"; fi
echo "{\"schemaVersion\":1,\"label\":\"go ingestor coverage\",\"message\":\"${INGESTOR_COV}%\",\"color\":\"${INGESTOR_COLOR}\"}" > .badges/go-ingestor-coverage.json
echo "## Go Coverage" >> $GITHUB_STEP_SUMMARY
echo "| Module | Coverage |" >> $GITHUB_STEP_SUMMARY
echo "|--------|----------|" >> $GITHUB_STEP_SUMMARY
echo "| Server | ${SERVER_COV}% |" >> $GITHUB_STEP_SUMMARY
echo "| Ingestor | ${INGESTOR_COV}% |" >> $GITHUB_STEP_SUMMARY
- name: Upload Go coverage badges
if: success()
uses: actions/upload-artifact@v6
with:
name: go-badges
path: .badges/go-*.json
retention-days: 1
if-no-files-found: ignore
include-hidden-files: true
# ───────────────────────────────────────────────────────────────
# 2. Playwright E2E Tests (against Go server with fixture DB)
# ───────────────────────────────────────────────────────────────
# Three shards on three runners, each with its own server and fixture copy,
# so no shard shares a browser, a CPU or server state with another. One
# runner per shard rather than parallel suites on one runner: the gesture and
# animation suites are timing-sensitive and would flake under CPU contention.
#
# It needs only `changes`, not go-test: this job builds its own binaries and
# uses nothing go-test produces. build-and-publish still needs both, so
# nothing reaches GHCR unless every test job passed.
e2e-shard:
name: "🎭 Playwright E2E shard ${{ matrix.shard }}/3"
needs: [changes]
# Pinned: on ubuntu-26.04 runners the Playwright browser was missing
# after the install step and every suite died at browserType.launch.
# Unpin once the Playwright version in package.json installs there.
runs-on: ubuntu-24.04
strategy:
matrix:
shard: [1, 2, 3]
env:
SHARD: ${{ matrix.shard }}
E2E_SHARDS: 3
defaults:
run:
shell: bash
if: needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published)
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Set up Node.js 22
uses: actions/setup-node@v5
with:
node-version: '22'
- name: Clean Go module cache
run: rm -rf ~/go/pkg/mod 2>/dev/null || true
- name: Set up Go 1.27
uses: actions/setup-go@v6
with:
go-version: '1.27'
cache-dependency-path: "**/go.sum"
- name: Build Go server
run: |
cd cmd/server
go build -o ../../corescope-server .
echo "Go server built successfully"
# Only test-user-management-e2e.js uses the e2etest build and its server,
# and it runs on shard 3. Moving that suite means moving these two too;
# it fails loudly (nothing on port 13582) rather than skipping.
- name: Build Go server (e2etest tag, user management)
if: matrix.shard == 3
run: |
cd cmd/server
go build -tags e2etest -o ../../corescope-server-e2e .
- name: Build Go migrate tool
run: |
cd cmd/migrate
go build -o ../../corescope-migrate .
echo "Go migrate tool built successfully"
- name: Install npm dependencies
run: npm ci --production=false
- name: Install Playwright browser
run: |
npx playwright install chromium
npx playwright install-deps chromium
- name: Instrument frontend JS for coverage
run: sh scripts/instrument-frontend.sh
- name: Freshen fixture timestamps
run: bash tools/freshen-fixture.sh test-fixtures/e2e-fixture.db
- name: Seed grouped-packet row for #1486 collapse test
# The committed fixture has 499 packets, each with exactly ONE
# observation, so the packets-page renders only flat
# (select-hash) rows. The #1486 repro needs at least one grouped
# (toggle-select) row. Insert a NEW transmission with 3
# observations.
#
# The server's async hash-migrate (cmd/server/hash_migrate.go)
# recomputes `transmissions.hash` from `raw_hex` via
# ComputeContentHash(), so the inserted hash MUST equal that
# function's output for the chosen raw_hex — otherwise the row
# gets relabelled and the E2E can't find it.
#
# raw_hex 15000102030405060708090a0b0c0d0e0f
# → header=0x15 (route_type=1, payload_type=5)
# → ComputeContentHash(...) = fae0c9e6d357a814
#
# Fixed old timestamps keep the row behind the freshly shifted
# fixture observations and outside the default 15-min UI window.
# The #1486 and #2104 tests reach it via an explicit hash +
# ?timeWindow=0 deep-link.
run: |
sqlite3 test-fixtures/e2e-fixture.db <<'SQL'
-- Sort the seeded row LAST in BOTH default packets views:
-- • flat view sorts by transmissions.id DESC → id=0 puts it last
-- • grouped view (#default for the packets page) sorts by
-- MAX(observations.timestamp) DESC → we must keep our obs
-- timestamps OLDER than every other fixture observation.
-- freshen-fixture.sh shifts fixture observations forward.
-- Keep this row at 2026-05-15 so it sorts LAST in the grouped
-- view too; the #1486 and #2104 tests use the explicit deep-link.
INSERT INTO transmissions(id,raw_hex,hash,first_seen,route_type,payload_type,payload_version,decoded_json,channel_hash,from_pubkey)
VALUES (0,'15000102030405060708090a0b0c0d0e0f','fae0c9e6d357a814','2026-05-15T00:00:00Z',1,5,0,'{"type":"CHAN","channel":"#test","text":"#1486 fixture"}',NULL,NULL);
INSERT INTO observations(transmission_id,observer_idx,direction,snr,rssi,score,path_json,timestamp,resolved_path) VALUES
(0,1,'rx',5.0,-95,0,'["AA"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["aa00000000000000000000000000000000000000000000000000000000000000"]'),
(0,2,'rx',5.5,-92,0,'["BB"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["bb00000000000000000000000000000000000000000000000000000000000000"]'),
(0,3,'rx',6.0,-90,0,'["CC"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["cc00000000000000000000000000000000000000000000000000000000000000"]');
-- #1791 fixture: a single GRP_DATA (payload_type=6) packet so the
-- E2E "Group Data filter" test has at least one row to filter on.
-- Use an obs timestamp within the default UI window so the row
-- appears with no time-window override.
--
-- raw_hex header byte 0x19 = bits 5-2 (payload)=0110=6 (GRP_DATA),
-- bits 1-0 (route)=01=1 (FLOOD).
-- path_len byte 0x00 = hash_size=1, hash_count=0 (zero-hop on-wire,
-- typical GRP_DATA going FLOOD). path_json/resolved_path are kept
-- EMPTY so the rendered hop-row count matches the hex-path byte
-- count (a prior fixture used path_json=["AA"] but raw_hex
-- path_len=0, which broke the "hex strip Path range matches hop
-- row count" E2E).
--
-- Note: id=-1000000 is a deliberately out-of-band sentinel id so
-- this synthetic fixture row cannot collide with real ingested
-- transmissions (real ids are positive autoincrement values).
INSERT INTO transmissions(id,raw_hex,hash,first_seen,route_type,payload_type,payload_version,decoded_json,channel_hash,from_pubkey)
VALUES (-1000000,'19000102030405060708090a0b0c0d0e0f','17910000deadbeef',strftime('%Y-%m-%dT%H:%M:%SZ','now'),1,6,0,'{"type":"GRP_DATA","channel":"#test","raw":"deadbeef"}',NULL,NULL);
INSERT INTO observations(transmission_id,observer_idx,direction,snr,rssi,score,path_json,timestamp,resolved_path) VALUES
(-1000000,1,'rx',7.0,-88,0,'[]',CAST(strftime('%s','now') AS INTEGER),'[]');
SQL
- name: Migrate fixture DB to current schema (#1287)
# Server now ASSERTs schema is migrated and refuses to start
# otherwise (cmd/server/main.go: dbschema.AssertReady). In prod
# the ingestor owns dbschema.Apply, but CI starts only the
# server against the committed e2e fixture — so we run the
# standalone migrate tool here to bring the fixture up to the
# required shape before the server boots.
run: ./corescope-migrate -db test-fixtures/e2e-fixture.db
- name: 'Seed distinct observation bytes for #2104'
# raw_hex exists only after migration. Keep the #1486 payload/hash
# while encoding each observation's existing one-hop path.
run: |
sqlite3 test-fixtures/e2e-fixture.db <<'SQL'
UPDATE observations SET raw_hex = CASE observer_idx
WHEN 1 THEN '1501aa0102030405060708090a0b0c0d0e0f'
WHEN 2 THEN '1501bb0102030405060708090a0b0c0d0e0f'
WHEN 3 THEN '1501cc0102030405060708090a0b0c0d0e0f'
END WHERE transmission_id = 0 AND observer_idx IN (1,2,3);
SQL
- name: Seed deterministic Path Inspector routes (#2060)
run: sqlite3 test-fixtures/e2e-fixture.db < test-fixtures/path-inspector.sql
- name: Start Go server with fixture DB
run: |
fuser -k 13581/tcp 2>/dev/null || true
sleep 1
./corescope-server -port 13581 -db test-fixtures/e2e-fixture.db -public public-instrumented &
echo $! > .server.pid
for i in $(seq 1 30); do
if curl -sf http://localhost:13581/api/healthz > /dev/null 2>&1; then
echo "Server ready after ${i}s"
break
fi
if [ "$i" -eq 30 ]; then
echo "Server failed to start within 30s"
exit 1
fi
sleep 1
done
- name: Start user-management E2E server (fake mailer)
if: matrix.shard == 3
run: |
fuser -k 13582/tcp 2>/dev/null || true
rm -rf /tmp/cs-um
mkdir -p /tmp/cs-um
# Own copy of the (already migrated and seeded) fixture; fresh users.db every run.
cp test-fixtures/e2e-fixture.db /tmp/cs-um/on.db
cat > /tmp/cs-um/config.json <<'JSON'
{
"port": 13582,
"userManagement": {
"enabled": true,
"dbPath": "/tmp/cs-um/users.db",
"adminEmails": ["admin@e2e.test"],
"publicBaseUrl": "http://localhost:13582",
"mail": { "provider": "fake", "fromEmail": "noreply@e2e.test" },
"channelProposals": { "enabled": true },
"notifications": { "enabled": true },
"observerLogins": {
"enabled": true,
"hookSecret": "e2e-hook-secret-0123",
"encryptionKey": "BwcHBwcHBwcHBwcHBwcHBwcHBwcHBwcHBwcHBwcHBwc=",
"broker": { "host": "mqtt.e2e.test" }
}
}
}
JSON
./corescope-server-e2e -config-dir /tmp/cs-um -port 13582 -db /tmp/cs-um/on.db -public public-instrumented &
echo $! > .server-um.pid
for i in $(seq 1 30); do
if curl -sf http://localhost:13582/api/healthz > /dev/null 2>&1; then echo "UM server ready after ${i}s"; break; fi
if [ "$i" -eq 30 ]; then echo "UM server failed to start within 30s"; exit 1; fi
sleep 1
done
- name: Start split frontend (static build + API-only server)
# The anonymous static build on :8788 against a stock server on :13591
# that lists that origin in CORS_ALLOWED_ORIGINS, the way a static
# frontend (Cloudflare Pages) calls the containers. -public on a missing
# directory is API-only mode, so nothing is served same-origin.
if: matrix.shard == 1
run: |
CHANNEL=dev COMMIT=e2e API_BASE=http://localhost:13591 AUTH=false CLASSIC_URL=http://localhost:13591 bash scripts/static/build.sh
fuser -k 13591/tcp 8788/tcp 2>/dev/null || true
CORS_ALLOWED_ORIGINS=http://localhost:8788 ./corescope-server -port 13591 -db test-fixtures/e2e-fixture.db -public /nonexistent &
echo $! > .server-split.pid
node scripts/dev-static.mjs --root dist/web --port 8788 &
echo $! > .static-split.pid
for i in $(seq 1 30); do
if curl -sf http://localhost:13591/api/healthz > /dev/null 2>&1 && curl -sf -o /dev/null http://localhost:8788/; then echo "split frontend ready after ${i}s"; break; fi
if [ "$i" -eq 30 ]; then echo "split frontend failed to start within 30s"; exit 1; fi
sleep 1
done
- name: Run Playwright E2E tests (fail-fast)
env:
# On master pushes every suite records the frontend coverage of the
# pages it opens into .nyc_output/ (#2203). Off on PRs to keep them fast.
NODE_OPTIONS: ${{ github.event_name == 'push' && '--require ./scripts/e2e-coverage-hook.js' || '' }}
run: |
# suite <shard> <file> [VAR=value ...]: runs tests/e2e/<file> with
# exactly those variables, but only on its own shard. The shard numbers
# balance the measured suite times; give a new suite the shard whose
# total is lowest. Anything but a number in 1..E2E_SHARDS (a missing
# number, 01, 4) fails the step, so a typo cannot silently drop a suite.
B=http://localhost:13581
suite() {
local shard=$1 file=$2
shift 2
if ! [[ "$shard" =~ ^[1-9][0-9]*$ ]] || [ "$shard" -gt "$E2E_SHARDS" ]; then
echo "::error::$file is assigned to shard $shard, outside 1..$E2E_SHARDS"
exit 1
fi
[ "$shard" -eq "$SHARD" ] || return 0
echo "=== E2E SUITE: $file ===" | tee -a e2e-output.txt
env "$@" node "tests/e2e/$file" 2>&1 | tee -a e2e-output.txt
}
suite 2 test-e2e-playwright.js BASE_URL=$B
# M5+M6 of #1668 — axe-core CI gate.
# M5: color-contrast on desktop dark+light.
# M6: expanded ruleset (image-alt, label, aria-required-attr,
# aria-valid-attr, aria-valid-attr-value, landmark-one-main,
# region, button-name, link-name, document-title, html-has-lang,
# duplicate-id) AND adds 375x812 mobile viewport (with
# color-contrast on mobile too).
# Allowlist: tests/a11y-allowlist.yaml (0 entries — hard pass policy).
# Per-viewport summary printed at the end; any net>0 fails the build.
suite 1 test-a11y-axe-1668.js BASE_URL=$B AXE_SCREENSHOT_DIR=/tmp/axe-1668
# The static build on another origin than the API (scripts/static/build.sh).
suite 1 test-split-anonymous-e2e.js SPLIT_URL=http://localhost:8788 API_URL=http://localhost:13591
suite 2 test-filter-ux-e2e.js BASE_URL=$B
suite 2 test-radio-setting-1869-e2e.js BASE_URL=$B
suite 2 test-observer-neighbors-2270-e2e.js BASE_URL=$B
suite 1 test-channel-issue-1087-e2e.js BASE_URL=$B
suite 1 test-channel-issue-1111-e2e.js BASE_URL=$B
suite 2 test-map-modal-fluid-e2e.js BASE_URL=$B
suite 2 test-map-nodes-pagination-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-map-estimated-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-map-foreign-advert-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-reported-nodes-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-observer-iata-1188-e2e.js BASE_URL=$B
suite 2 test-issue-1639-observers-sort-e2e.js BASE_URL=$B
suite 2 test-issue-748-node-analytics-coverage-e2e.js BASE_URL=$B
suite 2 test-issue-2218-node-analytics-stale-e2e.js BASE_URL=$B
suite 2 test-issue-661-airtime-e2e.js BASE_URL=$B
suite 2 test-health-status-e2e.js BASE_URL=$B
suite 2 test-issue-1268-rf-params-e2e.js BASE_URL=$B
suite 2 test-issue-1758-ng-filter-rerenders-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-nav-fluid-1055-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-nav-priority-1102-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-nav-priority-1311-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-nav-priority-1391-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1413-nav-overlap-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1400-nav-vertical-clip.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-nav-more-floor-1139-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-bottom-nav-1061-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-gestures-1062-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-gestures-1185-scroll-discriminator-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-gesture-hints-1065-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-touch-gestures-coverage-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-channel-fluid-e2e.js BASE_URL=$B
suite 2 test-table-fluid-e2e.js BASE_URL=$B
suite 3 test-charts-fluid-1058-e2e.js BASE_URL=$B
suite 3 test-slideover-1056-e2e.js BASE_URL=$B
suite 3 test-issue-1692-packets-init-parallel-e2e.js BASE_URL=$B
suite 2 test-slideover-1168-munger-e2e.js BASE_URL=$B
suite 3 test-logo-pulse-1173-e2e.js BASE_URL=$B
suite 2 test-issue-1122-packets-filter-ux-e2e.js BASE_URL=$B
suite 3 test-issue-1128-packets-layout-e2e.js BASE_URL=$B
suite 3 test-issue-1128-multi-viewport-e2e.js BASE_URL=$B
suite 2 test-issue-1136-live-region-e2e.js BASE_URL=$B
suite 1 test-live-multibyte-only-e2e.js BASE_URL=$B
suite 3 test-issue-1150-404-state-e2e.js BASE_URL=$B
suite 2 test-issue-1146-path-link-contrast-e2e.js BASE_URL=$B
suite 1 test-issue-1705-subpath-contrast-e2e.js CHROMIUM_REQUIRE=1
suite 2 test-issue-1147-section-order-e2e.js BASE_URL=$B
suite 3 test-issue-1151-orphan-separators-e2e.js BASE_URL=$B
suite 3 test-issue-1486-collapse-reopens-detail-e2e.js BASE_URL=$B
suite 3 test-map-deeplink-path-params-e2e.js BASE_URL=$B
suite 1 test-hop-path-inference-e2e.js BASE_URL=$B
suite 1 test-logo-rebrand-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-logo-theme-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-logo-default-sage-teal-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-theme-url-param-2236-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1109-hamburger-dropdown-visible-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-live-layout-1178-1179-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1205-live-controls-anchor-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-live-mql-leak-1180-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1204-live-panel-structure-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1234-live-chrome-pass2-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1206-vcr-overlap-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1833-legend-toggle-clickable-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1244-live-vcr-row-hints-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1510-live-nav-pin-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-live-fullscreen-1572-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1599-replay-freeze-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1648-m1-icons-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1648-m2-icons-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1648-m3-icons-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1648-m4-icons-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1657-analytics-channels-group-sprites-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1224-channels-mobile-ux-e2e.js BASE_URL=$B
suite 3 test-issue-1367-channels-chat-app-e2e.js BASE_URL=$B
suite 3 test-issue-1236-map-mobile-e2e.js BASE_URL=$B
suite 3 test-issue-1329-map-controls-accordion-e2e.js BASE_URL=$B
suite 1 test-issue-1273-qr-overlay-height-e2e.js BASE_URL=$B
suite 3 test-issue-1281-location-row-e2e.js BASE_URL=$B
suite 1 test-issue-1279-legend-p2-e2e.js BASE_URL=$B
suite 2 test-issue-1799-label-vocab-e2e.js BASE_URL=$B
suite 2 test-home-coverage-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-path-inspector-coverage-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1206-resize-observer-leak-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-nav-drawer-1064-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-audio-live-1297-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-audio-lab-1297-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-channel-decrypt-e2e.js BASE_URL=$B
suite 2 test-channel-qr-e2e.js BASE_URL=$B
suite 3 test-channel-color-picker-e2e.js BASE_URL=$B
suite 1 test-customize-theme-e2e.js BASE_URL=$B
suite 3 test-customize-branding-e2e.js BASE_URL=$B
suite 2 test-customize-display-e2e.js BASE_URL=$B
suite 3 test-customize-export-e2e.js BASE_URL=$B
suite 2 test-drag-manager-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1567-corner-clears-drag-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1306-collisions-terminology-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1374-route-map-a11y-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-channels-list-render-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-channels-selection-flow-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-channels-add-modal-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-channels-share-color-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-channels-ws-batch-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-channels-ws-race-1498-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1487-byop-modal-layout-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1630-reach-mobile-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-node-reach-coverage-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-issue-1640-compare-discovery-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-observer-detail-ownership-e2e.js BASE_URL=$B
suite 2 test-neighbor-map-btn-clip-e2e.js CHROMIUM_REQUIRE=1
suite 3 test-rx-coverage-viewport-e2e.js BASE_URL=$B
suite 1 test-rx-coverage-noise-e2e.js BASE_URL=$B
# Issue #2037: these eight sat in tests/e2e/ with no runner at all.
# Measured on upstream master in CI run 35340835178: all eight pass,
# 45 seconds together. test-rx-coverage-mobile-nav-e2e.js is the
# ninth that passed and is deliberately NOT here: it exits 0 with
# "SKIP (clientRxCoverage disabled on this deployment)", which is the
# default, so wiring it in would add a vacuous pass rather than a guard.
suite 2 test-1110-live-filter.js BASE_URL=$B
suite 3 test-analytics-fluid-charts.js BASE_URL=$B
suite 3 test-e2e-1267-mobile-vcr.js BASE_URL=$B
suite 2 test-issue-1274-legend-coverage-e2e.js BASE_URL=$B
suite 3 test-issue-1648-m5-icons-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-issue-1997-distance-building-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-channel-modal-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-issue-1522-trace-url-sync-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-marker-outline-weight.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 2 test-node-reach-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-packets-compact-columns-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-packets-node-filter-e2e.js BASE_URL=$B
suite 3 test-packets-scope-column.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-path-inspector-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-pr-1490-live-map-gpu-animations-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 1 test-region-quick-picks.js
# The lint rule that keeps network calls on CSApi (needs eslint).
suite 1 test-lint-network-calls.js
suite 3 test-table-sort.js
suite 1 test-touch-targets.js CHROMIUM_REQUIRE=1 BASE_URL=$B
suite 3 test-live-dedup.js BASE_URL=$B
suite 1 test-nodes-export-e2e.js BASE_URL=$B
suite 1 test-node-quick-actions-e2e.js BASE_URL=$B
suite 2 test-show-neighbors.js BASE_URL=$B
suite 3 test-user-management-e2e.js BASE_URL=http://localhost:13582 BASE_URL_OFF=$B
# #1616: slide-over focus-restore flake-gate. Runs the slide-over
# E2E 3 more times against the SAME backend instance (it already
# ran once in the list above) so the Chromium-headless focus race
# documented in #1172/#1616 gets repeated shots at firing. Any
# single non-zero exit aborts. This is the architectural-fix gate:
# if it ever turns red post-merge, the focused-but-hidden state has
# crept back in.
#
# PERMANENT step, on shard 3 (the shard that runs the slide-over
# suite). Adds ~70s to that shard in exchange for closing out a
# flake family that was blocking ~8 unrelated PRs at a time. If
# profiling pressures the budget later, drop repeat count first;
# do not delete.
- name: Slide-over E2E flake-gate (#1616, --repeat-each=3)
if: matrix.shard == 3
run: |
set -e
for i in $(seq 1 3); do
echo "--- slide-over E2E run $i/3 ---"
echo "=== E2E SUITE: test-slideover-1056-e2e.js ==="
BASE_URL=http://localhost:13581 node tests/e2e/test-slideover-1056-e2e.js 2>&1 | tee -a slideover-repeat-output.txt
done
echo "3 passed"
# An extra coverage source on top of the suites: a tour of the main
# page groups. Its files land in .nyc_output/ beside the suites' files.
- name: Collect frontend coverage (parallel)
if: success() && github.event_name == 'push' && matrix.shard == 2
run: |
BASE_URL=http://localhost:13581 node scripts/collect-frontend-coverage.js 2>&1 | tee fe-coverage-output.txt || true
# One file per suite is about 6 MB, mostly the same maps, so the shard
# merges them before upload. e2e-test merges the three shards and makes
# the frontend coverage badge.
# Coverage never decides the outcome: a failure here must not skip the
# shard upload below (if: success()) and so fail the E2E gate.
- name: Merge this shard's frontend coverage
if: success() && github.event_name == 'push'
continue-on-error: true
run: |
if ls .nyc_output/*.json >/dev/null 2>&1; then
npx nyc merge .nyc_output coverage-shard.json
else
echo "::warning::no frontend coverage was recorded on shard ${{ matrix.shard }}"
fi
- name: Stop test server
if: always()
run: |
if [ -f .server.pid ]; then
kill $(cat .server.pid) 2>/dev/null || true
rm -f .server.pid
fi
if [ -f .server-um.pid ]; then
kill $(cat .server-um.pid) 2>/dev/null || true
rm -f .server-um.pid
fi
for f in .server-split.pid .static-split.pid; do
if [ -f "$f" ]; then kill $(cat "$f") 2>/dev/null || true; rm -f "$f"; fi
done
- name: Upload shard results
if: success()
uses: actions/upload-artifact@v6
with:
name: e2e-shard-${{ matrix.shard }}
path: |
e2e-output.txt
coverage-shard.json
retention-days: 1
if-no-files-found: ignore
include-hidden-files: true
# The gate the rest of the pipeline (and branch protection) sees under the
# old single-job name. It folds the shard outputs into the same e2e-badges
# artifact the publish job reads.
#
# !cancelled() is what makes it a gate. Without it a failed shard would SKIP
# this job, and branch protection counts a skipped required check as
# passing. With it, the job runs and fails on its first step instead.
e2e-test:
name: "🎭 Playwright E2E Tests"
needs: [e2e-shard, changes]
runs-on: ubuntu-latest
if: ${{ !cancelled() && needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published) }}
steps:
- name: Require every shard to have passed
run: |
result="${{ needs['e2e-shard'].result }}"
echo "e2e-shard: $result"
[ "$result" = success ] || { echo "::error::E2E shards did not all pass (result: $result)"; exit 1; }
- name: Checkout code
uses: actions/checkout@v5
- name: Download shard results
uses: actions/download-artifact@v6
with:
pattern: e2e-shard-*
path: shards/
- name: Generate E2E badges
run: |
set -e
ls shards
[ "$(ls -d shards/e2e-shard-* | wc -l)" -eq 3 ] || { echo "::error::expected results from 3 shards"; exit 1; }
cat shards/e2e-shard-*/e2e-output.txt > e2e-output.txt
# Start empty: the checkout holds master's old .badges/ files,
# which would otherwise go out in e2e-badges (#2136).
rm -rf .badges && mkdir .badges
# Aggregate per-suite PASS/FAIL across every test-*-e2e.js summary.
# The previous regex (grep -oP '[0-9]+(?=/)' | tail -1) caught a
# stray digits-before-slash like the '2' in '2/3 tests passed' from
# some sub-output and stamped the badge as '2 passed'. See #1296.
eval "$(bash scripts/aggregate-e2e-pass.sh e2e-output.txt)"
E2E_PASS=${PASS:-0}
E2E_FAIL=${FAIL:-0}
if [ "${E2E_FAIL:-0}" -gt 0 ]; then
E2E_MSG="${E2E_PASS:-0} passed, ${E2E_FAIL} failed"
E2E_COLOR="red"
else
E2E_MSG="${E2E_PASS:-0} passed"
E2E_COLOR="brightgreen"
fi
echo "{\"schemaVersion\":1,\"label\":\"e2e tests\",\"message\":\"${E2E_MSG}\",\"color\":\"${E2E_COLOR}\"}" > .badges/e2e-tests.json
echo "E2E: ${E2E_MSG} ($(grep -c '^=== E2E SUITE' e2e-output.txt) suite runs)" | tee -a "$GITHUB_STEP_SUMMARY"
- name: Set up Node.js 22
if: github.event_name == 'push'
continue-on-error: true
uses: actions/setup-node@v5
with:
node-version: '22'
# Coverage from every suite on every shard (#2203), merged into one
# number. Only master pushes record it, and it never fails the gate.
- name: Generate frontend coverage badge
if: github.event_name == 'push'
continue-on-error: true
run: |
mkdir -p .nyc_output
for f in shards/e2e-shard-*/coverage-shard.json; do
[ -f "$f" ] && cp "$f" ".nyc_output/$(basename "$(dirname "$f")").json"
done
if ! ls .nyc_output/*.json >/dev/null 2>&1; then
echo "::warning::no shard recorded frontend coverage; the badge keeps its previous value"
exit 0
fi
echo "Frontend coverage from: $(ls .nyc_output | paste -sd ' ' -)"
npx --yes nyc@18.0.0 report --reporter=text-summary --reporter=text 2>&1 | tee fe-report.txt
FE_COVERAGE=$(grep 'Statements' fe-report.txt | head -1 | grep -oP '[\d.]+(?=%)' || true)
if [ -z "$FE_COVERAGE" ]; then
echo "::warning::nyc report gave no statement coverage; the badge keeps its previous value"
exit 0
fi
FE_COLOR="red"
[ "$(echo "$FE_COVERAGE > 50" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="yellow"
[ "$(echo "$FE_COVERAGE > 80" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="brightgreen"
echo "{\"schemaVersion\":1,\"label\":\"frontend coverage\",\"message\":\"${FE_COVERAGE}%\",\"color\":\"${FE_COLOR}\"}" > .badges/frontend-coverage.json
echo "## Frontend: ${FE_COVERAGE}% coverage" >> "$GITHUB_STEP_SUMMARY"
- name: Upload E2E badges
uses: actions/upload-artifact@v6
with:
name: e2e-badges
path: .badges/
retention-days: 1
if-no-files-found: ignore
include-hidden-files: true
# ───────────────────────────────────────────────────────────────
# 3a. Two-arch image build — beside the tests, never publishes
# ───────────────────────────────────────────────────────────────
# Since the SQLite driver became cgo, the container build depends on a zig
# cross-toolchain and static musl linking. Builds both architectures and
# actually runs the arm64 binaries under QEMU, on every event: for a PR it is
# the only image check (the GHCR push below is push/tag-only, so without it
# the first signal would arrive on master); for a push it fills the layer
# cache that the push then reuses.
#
# It needs only `changes`, so its minutes overlap the test jobs instead of
# following them. It cannot publish anything, so running it before the tests
# pass is safe. The build metadata is computed here once and handed to
# build-and-publish: BUILD_TIME is a build arg of the Go layers, so a second
# timestamp would miss the cache and rebuild everything.
image-check:
name: "🧱 Two-arch image build"
needs: [changes]
runs-on: ubuntu-latest
if: needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published)
outputs:
build_time: ${{ steps.meta.outputs.build_time }}
git_commit: ${{ steps.meta.outputs.git_commit }}
app_version: ${{ steps.meta.outputs.app_version }}
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Compute build metadata
id: meta
run: |
BUILD_TIME=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
GIT_COMMIT="${GITHUB_SHA::7}"
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
APP_VERSION="${GITHUB_REF#refs/tags/}"
else
APP_VERSION="edge"
fi
echo "build_time=$BUILD_TIME" >> "$GITHUB_OUTPUT"
echo "git_commit=$GIT_COMMIT" >> "$GITHUB_OUTPUT"
echo "app_version=$APP_VERSION" >> "$GITHUB_OUTPUT"
echo "Build: version=$APP_VERSION commit=$GIT_COMMIT time=$BUILD_TIME"
# The staging compose file used to be built here as well. That image was
# never used: the runner is discarded, and the deploy job pulls :edge from
# GHCR on its own runner. It rebuilt the same Dockerfile as the step
# below for two minutes. Validating the compose file is what is left.
- name: Validate staging compose file
run: docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging config --quiet
# Split containers: operators build the web image from their checkout and
# nothing publishes it, so without this a broken base image, package or
# Caddyfile would reach an operator first.
- name: Validate split compose file
run: CORESCOPE_IMAGE=corescope:check docker compose -f docker-compose.split.yml config --quiet
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and smoke the web image (Dockerfile.web)
env:
GIT_COMMIT: ${{ steps.meta.outputs.git_commit }}
run: |
set -euo pipefail
docker buildx build . -f Dockerfile.web --load -t corescope-web:check --build-arg GIT_COMMIT="$GIT_COMMIT"
# BACKEND is a closed port: /api must reach it (502), not the file server (404).
docker run -d --name web-check -p 8080:80 -e BACKEND=127.0.0.1:9 corescope-web:check
trap 'docker logs web-check 2>&1 | tail -20; docker rm -f web-check >/dev/null' EXIT
for i in $(seq 1 30); do curl -fs -o /dev/null http://localhost:8080/ && break; sleep 1; done
page=$(curl -fsS -D headers.txt http://localhost:8080/)
echo "$page" | grep -q "app.js?v=$GIT_COMMIT" || { echo "::error::index.html is not busted with the commit"; exit 1; }
if echo "$page" | grep -q __BUST__; then echo "::error::index.html still has __BUST__"; exit 1; fi
grep -qi '^cache-control: no-cache' headers.txt || { echo "::error::index.html may be cached"; exit 1; }
curl -fsS http://localhost:8080/deployment.js | grep -q "apiBase: ''" || { echo "::error::deployment.js is not same-origin"; exit 1; }
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/api/healthz)
[ "$code" = 502 ] || { echo "::error::/api/healthz answered $code, not 502 from the closed BACKEND"; exit 1; }
echo "web image serves the packaged UI and forwards /api ✅"
- name: Set up QEMU (arm64 runtime stage + arm64 smoke)
uses: docker/setup-qemu-action@v3
- name: Two-arch build (no publish)
uses: docker/build-push-action@v6
with:
context: .
push: false
platforms: linux/amd64,linux/arm64
tags: corescope:check-${{ github.run_id }}
build-args: |
APP_VERSION=${{ steps.meta.outputs.app_version }}
GIT_COMMIT=${{ steps.meta.outputs.git_commit }}
BUILD_TIME=${{ steps.meta.outputs.build_time }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Smoke the arm64 binaries under QEMU
run: |
set -e
# Export the arm64 binaries so the runner's own `file` can confirm
# they are static, then execute one under QEMU. Cheap: the build above
# already populated the layer cache.
docker buildx build . --platform linux/arm64 \
--build-arg APP_VERSION=smoke --cache-from type=gha \
--output type=local,dest=./arm64-image
for b in corescope-server corescope-ingestor corescope-decrypt; do
file "./arm64-image/app/$b" | tee /dev/stderr | grep -q 'statically linked' \
|| { echo "::error::$b is not statically linked"; exit 1; }
file "./arm64-image/app/$b" | grep -q 'ARM aarch64' \
|| { echo "::error::$b is not an arm64 binary"; exit 1; }
done
docker run --rm --platform linux/arm64 \
-v "$PWD/arm64-image/app:/bin-under-test:ro" alpine:3.24 \
/bin-under-test/corescope-decrypt --version
rm -rf ./arm64-image
echo "arm64 static binaries run under QEMU ✅"
# ───────────────────────────────────────────────────────────────
# 3b. Publish Docker Image — only after every test job passed
# ───────────────────────────────────────────────────────────────
# The GHCR steps below publish on a push to master (the :edge image) and
# on a tag ref. The tag half is not decoration: release-fast-path.yml re-tags
# :edge to :vX.Y.Z when the :edge revision label matches the tagged commit,
# and dispatches THIS workflow when it does not. That fallback previously
# produced no image at all, because a workflow_dispatch is not a push and
# every publishing step was gated on github.event_name == 'push'. It built
# locally, reported success, and pushed nothing.
#
# That went unnoticed until v3.10.0, where the tagged commit was
# documentation-only. At that time a trigger-level paths-ignore skipped this
# workflow for such commits, so no :edge image was ever built for it, the label
# comparison failed, and the fallback ran for the first time. That filter is
# gone: the `changes` job now forces code=true for anything that is not a pull
# request, so every master commit gets an image and a documentation commit is
# safe to tag. See the comment on the `on:` block.
#
# On a pull request only the first step runs; the job stays so that its
# check still means "tests and image build all passed".
#
# !cancelled() plus the first step make that true for failures too: without
# them a failed needed job would SKIP this job, and branch protection counts
# a skipped required check as passing. race-test is skipped when no ingestor
# Go file changed, so for it skipped is accepted; a master push always runs it.
build-and-publish:
name: "🏗️ Build & Publish Docker Image"
needs: [go-test, race-test, e2e-test, image-check, changes]
runs-on: ubuntu-latest
if: ${{ !cancelled() && needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published) }}
steps:
- name: Require every test job and the image check to have passed
run: |
fail=0
require() {
echo "$1: $2"
case " $3 " in
*" $2 "*) ;;
*) echo "::error::$1 result is $2"; fail=1 ;;
esac
}
require go-test "${{ needs['go-test'].result }}" "success"
require race-test "${{ needs['race-test'].result }}" "success skipped"
require e2e-test "${{ needs['e2e-test'].result }}" "success"
require image-check "${{ needs['image-check'].result }}" "success"
exit $fail
- name: Checkout code
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
uses: actions/checkout@v5
- name: Set up Docker Buildx
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
uses: docker/setup-buildx-action@v3
- name: Set up QEMU (arm64 runtime stage)
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
uses: docker/setup-qemu-action@v3
- name: Log in to GHCR
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
id: docker-meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/kpa-clawbot/corescope
tags: |
type=semver,pattern=v{{version}}
type=semver,pattern=v{{major}}.{{minor}}
type=semver,pattern=v{{major}}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=edge,branch=master
# Same build args as image-check, so every layer is a cache hit.
- name: Build and push to GHCR
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
uses: docker/build-push-action@v6
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.docker-meta.outputs.tags }}
labels: ${{ steps.docker-meta.outputs.labels }}
build-args: |
APP_VERSION=${{ needs.image-check.outputs.app_version }}
GIT_COMMIT=${{ needs.image-check.outputs.git_commit }}
BUILD_TIME=${{ needs.image-check.outputs.build_time }}
# No cache-to: image-check exported these same layers minutes ago.
cache-from: type=gha
# ───────────────────────────────────────────────────────────────
# 4. Release Artifacts (tags only)
# ───────────────────────────────────────────────────────────────
release-artifacts:
name: "📦 Release Artifacts"
needs: [go-test, changes]
runs-on: ubuntu-latest
permissions:
contents: write
if: startsWith(github.ref, 'refs/tags/v') && needs.changes.outputs.code == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Set up Go 1.27
uses: actions/setup-go@v6
with:
go-version: '1.27'
cache-dependency-path: |
cmd/decrypt/go.sum
internal/channel/go.sum
# The SQLite driver is cgo now (github.com/mattn/go-sqlite3), so the Go
# toolchain alone cannot cross-compile these. zig cc is the C compiler
# that can target both architectures; musl makes the result static.
- name: Set up Zig
uses: mlugg/setup-zig@v2
with:
version: 0.16.0
- name: Build corescope-decrypt (static, linux/amd64)
run: |
cd cmd/decrypt
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CC="zig cc -target x86_64-linux-musl" go build -trimpath -tags netgo,osusergo,sqlite_omit_load_extension -ldflags="-s -w -extldflags '-static -Wl,-s' -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-amd64 .
- name: Build corescope-decrypt (static, linux/arm64)
run: |
cd cmd/decrypt
CGO_ENABLED=1 GOOS=linux GOARCH=arm64 CC="zig cc -target aarch64-linux-musl" go build -trimpath -tags netgo,osusergo,sqlite_omit_load_extension -ldflags="-s -w -extldflags '-static -Wl,-s' -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-arm64 .
- name: Verify release binaries are static and runnable
run: |
set -e
for f in corescope-decrypt-linux-amd64 corescope-decrypt-linux-arm64; do
file "$f" | grep -q 'statically linked' || { echo "::error::$f is not statically linked"; exit 1; }
done
# amd64 runs natively on the runner; arm64 is only checked structurally
# here (the container image job exercises arm64 under QEMU).
./corescope-decrypt-linux-amd64 --version
- name: Locate the release notes for this tag
id: notes
run: |
set -euo pipefail
# docs/release-notes/<tag>.md is the convention: v3.10.1.md, v3.12.0.md.
F="docs/release-notes/${GITHUB_REF_NAME}.md"
if [ -f "$F" ]; then
echo "path=$F" >> "$GITHUB_OUTPUT"
echo "found $F ($(wc -c < "$F") bytes)"
else
# Not an error. A release without a notes file still gets the
# generated pull request list below, which is more than the empty
# body v3.10.1, v3.11.0 and v3.12.0 were published with.
echo "path=" >> "$GITHUB_OUTPUT"
echo "::warning::no $F, publishing with generated notes only"
fi
- name: Upload release assets
# Standard releases upload both assets to a draft before publishing.
# Keep one writer: published releases are immutable in this repository.
#
# body_path and generate_release_notes together: our own notes first,
# then GitHub's generated pull request list under them. Without either,
# this step publishes an empty release body, which is what happened to
# every release from v3.10.1 to v3.12.0 (v3.12.0's body was filled in by
# hand afterwards). The hand-written file is the substance; the generated
# list is the "what changed" index that a reader expects to find.
uses: softprops/action-gh-release@v2
with:
fail_on_unmatched_files: true
body_path: ${{ steps.notes.outputs.path }}
generate_release_notes: true
files: |
corescope-decrypt-linux-amd64
corescope-decrypt-linux-arm64
# ───────────────────────────────────────────────────────────────
# 4b. Deploy Staging (master only)
# ───────────────────────────────────────────────────────────────
deploy:
name: "🚀 Deploy Staging"
# DISABLED. Re-enable by setting the repository variable
# ENABLE_STAGING_DEPLOY to 'true' (Settings > Secrets and variables >
# Actions > Variables). No code change needed.
#
# Why: this job runs on [self-hosted, meshcore-runner-2] and that runner
# has not picked up a job since at least 2026-08-31. It has no
# timeout-minutes, so it sat queued for 22+ hours, held its run open, and
# through the concurrency group ci-refs/heads/master caused GitHub to
# cancel every subsequent master push. The result was that master produced
# no completed pipeline result at all: 30+ runs cancelled or stuck while
# go-test, e2e-test and build-and-publish were passing inside them.
#
# timeout-minutes alone would unblock the queue but leave master
# permanently red on a job that cannot succeed while the runner is absent,
# so the deploy is gated instead. It is added here as well, so that if the
# variable is set while the runner is still missing the job fails in ten
# minutes rather than blocking the branch again.
if: |
vars.ENABLE_STAGING_DEPLOY == 'true'
&& (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
&& github.ref == 'refs/heads/master'
needs: [build-and-publish]
runs-on: [self-hosted, meshcore-runner-2]
timeout-minutes: 10
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Pull latest image from GHCR
run: |
# Try to pull the edge image from GHCR and tag for docker-compose compatibility
if docker pull ghcr.io/kpa-clawbot/corescope:edge; then
docker tag ghcr.io/kpa-clawbot/corescope:edge corescope-go:latest
echo "Pulled and tagged GHCR edge image ✅"
else
echo "⚠️ GHCR pull failed — falling back to locally built image"
fi
- name: Deploy staging
run: |
# Force-remove the staging container regardless of how it was created
# (compose-managed OR manually created via docker run)
docker stop corescope-staging-go 2>/dev/null || true
docker rm -f corescope-staging-go 2>/dev/null || true
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging down --timeout 30 2>/dev/null || true
# Wait for container to be fully gone and OS to reclaim memory (3GB limit)
for i in $(seq 1 15); do
if ! docker ps -a --format '{{.Names}}' | grep -q 'corescope-staging-go'; then
break
fi
sleep 1
done
sleep 5 # extra pause for OS memory reclaim
# Ensure staging data dir exists (config.json lives here, no separate file mount)
STAGING_DATA="${STAGING_DATA_DIR:-$HOME/meshcore-staging-data}"
mkdir -p "$STAGING_DATA"
# If no config exists, copy the example (CI doesn't have a real prod config)
if [ ! -f "$STAGING_DATA/config.json" ]; then
echo "Staging config missing — copying config.example.json"
cp config.example.json "$STAGING_DATA/config.json" 2>/dev/null || true
fi
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging up -d staging-go
- name: Healthcheck staging container
run: |
for i in $(seq 1 120); do
HEALTH=$(docker inspect corescope-staging-go --format '{{.State.Health.Status}}' 2>/dev/null || echo "starting")
if [ "$HEALTH" = "healthy" ]; then
echo "Staging healthy after ${i}s"
break
fi
if [ "$i" -eq 120 ]; then
echo "Staging failed health check after 120s"
docker logs corescope-staging-go --tail 50
exit 1
fi
sleep 1
done
- name: Smoke test staging API
run: |
PORT="${STAGING_GO_HTTP_PORT:-80}"
if curl -sf "http://localhost:${PORT}/api/stats" | grep -q engine; then
echo "Staging verified — engine field present ✅"
else
echo "Staging /api/stats did not return engine field (port ${PORT})"
exit 1
fi
- name: Clean up old Docker images
if: always()
run: |
# Remove dangling images and images older than 24h (keeps current build)
echo "--- Docker disk usage before cleanup ---"
docker system df
docker image prune -af --filter "until=24h" 2>/dev/null || true
docker builder prune -f --keep-storage=1GB 2>/dev/null || true
echo "--- Docker disk usage after cleanup ---"
docker system df
# ───────────────────────────────────────────────────────────────
# 5. Publish Badges & Summary (master only)
# ───────────────────────────────────────────────────────────────
publish:
name: "📝 Publish Badges & Summary"
if: github.event_name == 'push'
# Was needs: [deploy]. The badges are built from the go-test and e2e-test
# artifacts and never needed the deploy step; depending on it meant the
# coverage badges stopped updating whenever the self-hosted runner was
# unavailable. build-and-publish already transitively requires both test
# jobs, so ordering is unchanged.
needs: [build-and-publish]
runs-on: ubuntu-latest
# The badges go to their own `badges` branch (#2136). master is protected,
# so writing there took a PAT (BADGE_PUSH_TOKEN), which stopped working.
# The badges branch is unprotected, so the job's own GITHUB_TOKEN can
# write it.
permissions:
contents: write
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
# The push below passes its own token; later steps need none.
persist-credentials: false
# go-badges second, so its files win if e2e-badges ever carries go
# badges again.
- name: Download E2E badges
continue-on-error: true
uses: actions/download-artifact@v6
with:
name: e2e-badges
path: badges-new/
- name: Download Go coverage badges
continue-on-error: true
uses: actions/download-artifact@v6
with:
name: go-badges
path: badges-new/
- name: Publish coverage badges to the badges branch
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
# Start from what the badges branch holds now (master's .badges/ the
# first time) and overwrite only the badges this run produced, so a
# missing artifact never removes a badge. The branch is one orphan
# commit, force-pushed, so it never grows.
# Every step checks its own result: bash ignores set -e inside a
# function whose status is tested, as the `if` below does.
publish() {
ls badges-new/*.json >/dev/null 2>&1 || { echo "no badge artifacts were downloaded"; return 1; }
out=$(mktemp -d) || return 1
# Exit 2 means the branch does not exist yet; any other failure
# must not be taken for that, or the badges this run did not
# produce would be reset.
git ls-remote --exit-code --heads origin badges >/dev/null
case $? in
0)
git fetch -q --depth=1 origin badges || return 1
git archive FETCH_HEAD | tar -x -C "$out" || return 1
;;
2)
if ls .badges/*.json >/dev/null 2>&1; then
cp .badges/*.json "$out"/ || return 1
fi
;;
*) return 1 ;;
esac
cp badges-new/*.json "$out"/ || return 1
cd "$out" || return 1
git init -q -b badges || return 1
git add -- *.json || return 1
git -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -q -m "ci: badges for ${GITHUB_SHA::8}" || return 1
git push -q --force "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" HEAD:badges || return 1
ls -- *.json
}
if ! (publish); then
echo "::warning title=Badges not published::Publishing to the badges branch failed (see the step log); the README badges keep their previous values."
fi
- name: Post deployment summary
run: |
echo "## Staging Deployed ✓" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Commit:** \`$(git rev-parse --short HEAD)\` — $(git log -1 --format=%s)" >> $GITHUB_STEP_SUMMARY