mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-26 05:43:54 +00:00
Anupam MedirattaandClaude Opus 5
de237fc29c
Closes #1977. Supersedes #1952. Follow-up filed as #1983. ## What §10.2 did ```bash q="SELECT COUNT(*) FROM transmissions WHERE from_node = '$TEST_PUBKEY';" qq=$(printf %q "$q") if ! count=$(ssh_t "docker exec … sqlite3 … $qq" 2>/dev/null); then count=$(ssh_t "sqlite3 … $qq" 2>/dev/null || echo "") fi ``` The injection is not reachable today — `TEST_PUBKEY` is hex-gated and the script `exit 2`s before the SQL is built. The problem is that the SQL layer's safety rests entirely on that outer gate rather than on the SQL layer itself. #1952 proposed doubling embedded quotes; that is string escaping, not parameterisation, which is why it was withdrawn in favour of this. ## What this does Per the four points in the sign-off on #1977: **1. Bind the value.** A constant `SELECT` and a bound `:pubkey`, fed to sqlite3 on stdin. The SQL no longer crosses the remote shell as a command word, so there is no `printf %q` on the query at all any more. **Why hex rather than `.parameter set :pk '<value>'`.** Dot-command arguments are split on whitespace, so a payload containing a space produces too many arguments — and sqlite3 responds by printing the `.parameter` help to **stdout**, exiting **0**, and leaving `:pk` **unbound**. `COUNT(*)` then returns 0, which reads exactly like a passing security fix. `-bail` does not catch it. Verified on 3.51.0: ``` $ printf ".parameter set :pk '' OR 1=1 --'\nSELECT COUNT(*) FROM transmissions WHERE from_node = :pk;\n" \ | sqlite3 -bail ptest.db .parameter CMD ... Manage SQL parameter bindings # <- help, on stdout … 0 # <- :pk never bound $ echo $? 0 ``` `.parameter set :pk 1+1` also binds the integer `2` — the value is evaluated as an SQL expression and only falls back to a text literal when evaluation fails. So interpolating into the `.parameter set` line trades one hazard for another. Hex-encoding removes the quoting layer instead of adding one: the value is bound as `cast(x'<hex>' as text)`, so its contribution to the SQL text is drawn from the alphabet `[0-9a-f]` only. Nothing to quote, no tokenizer arity hazard, and it holds for **arbitrary** input rather than only for hex-gated input — which is the point. Verified against a fixture table holding two rows, one of them `deadbeef`: | value | result | exit | |---|---|---| | `deadbeef`, bound as `cast(x'6465616462656566' as text)` | `1` | 0 | | `' OR 1=1 --`, bound the same way | `0` | 0 | | `' OR 1=1 --`, interpolated the current way | `2` (whole table) | 0 | | query against a DB with no `transmissions` table, `-bail` | `Parse error … no such table` on **stderr** | **1** | **2. Probe the capability, not a version.** `resolve_sqlite_runner` binds `corescope-probe-ok` and asserts it comes back — a round trip, not a bare `.parameter init`, so the positive control runs against the operator's actual binary rather than one we pin. If neither the container nor the host qualifies, it fails loudly and names what is needed: ``` ❌ retain-failed: no sqlite3 able to bind a parameter on the target tried: docker exec -i corescope-prod sqlite3, then sqlite3 on runner@example need: the sqlite3 CLI reachable over ssh, supporting '.parameter set' OCI runtime exec failed: exec: "sqlite3": executable file not found in $PATH bash: line 1: sqlite3: command not found ``` There is deliberately **no** interpolating fallback. That would leave the vulnerable path in place under a nicer name. **3. The hex gate is kept**, with its comment updated to say why: for the SQL layer it is now defence in depth rather than the only guard. Redundant is not the same as wrong. **4. The exit status and stderr survive.** `-batch -bail -init /dev/null -noheader -list` (stop at the first SQL error; ignore the operator's `~/.sqliterc`, where a stray `.mode` would make the count unparseable; stdout is exactly the number). Query stderr is captured and printed on failure rather than sent to `/dev/null`, so a broken query is distinguishable from a legitimately empty result. Probe stderr is collected too, and printed only if *both* probes fail — the container miss is the known-normal case, so surfacing it on every run would be noise. ## Also fixed An existing double-count in §10.2: the `TARGET_DB_PATH unset` branch incremented `$fails` and then left `count=""`, so the generic branch incremented it a **second** time for the same failure. `read_retain_count` now gives §10.2 exactly one increment point. Opportunistic cleanup in a file already being touched (AGENTS.md line 318). ## Tests New `qa/scripts/test-blacklist-sql.sh`, wired into the `go-test` job. 24 assertions, modelled on `scripts/staging/test-disk-monitor.sh`. Both directions are asserted, because a zero from a command that failed proves nothing: - **Positive control** — `deadbeef` still returns its row (`1`, exit 0), and so does `cafebabe`; an absent pubkey returns `0`. - **Negative** — `' OR 1=1 --` returns `0` while the table demonstrably holds 2 rows, and the old interpolated form is asserted to leak all `2`. That last assertion is what makes the `0` above worth something. - **Error surfacing** — the same SQL against a DB with no `transmissions` table exits non-zero with a message on stderr and nothing on stdout. - **Alphabet** — `sql_hex_literal` output matches `^x'[0-9a-f]*'$` for the SQL payloads, a backslash, `$(id)` / backticks, an embedded newline, `héllo`, and a 4096-byte repetitive string. That last one is a regression guard for `od -v`: without the flag `od` collapses repeated identical lines to `*`. - `run_sqlite` with no resolved runner refuses rather than guessing. Group 2 skips loudly (rather than silently) if `sqlite3` is not on PATH; group 1 needs no sqlite3 and always runs. **Mutation-tested** — each of these breaks the suite, so the assertions have teeth: | mutation | caught by | |---|---| | restore full interpolation | `injection payload → 0 rows — expected '0' got '2'` | | naive `.parameter set '%s'` | `expected '0' got '.parameter CMD ...'` | | drop `od -v` | alphabet failure on `*`, plus `expected '8192' got '33'` | Commit 1 is a behaviour-neutral refactor that moves the imperative body into `main()` behind a `BASH_SOURCE` guard, so the test can source the script and exercise individual helpers. Same idiom as `scripts/staging/disk-monitor.sh:99`. ## Verification - `bash qa/scripts/test-blacklist-sql.sh` → 24 passed, 0 failed - `bash -n` on both scripts - All three runtime paths exercised end to end with PATH shims for `ssh`/`docker`/`sqlite3` against a real fixture DB: success (`sqlite3 runner: host`, count 2), query failure (classified message + `Parse error … no such table`, `fails=1`), and no-capability (the loud block above, both probe stderrs, `fails=1` — not 2) - The new step lands inside `go-test`, which runs when `changes.outputs.code == 'true'`; `qa/scripts/*.sh` does not match that job's `^docs/|[.]md$|^LICENSE$` documentation filter, so it is not skipped ## Deliberately out of scope - **The `docker exec` branch is dead on current images** → filed as #1983. The app container has no `sqlite3` at all: `Dockerfile:15` is pure-Go SQLite with no CGO, and the `apk add` installs only `mosquitto mosquitto-clients supervisor caddy wget`. So the host "fallback" is the only path that has ever executed, silently, because both branches discarded stderr. This change keeps both branches and merely makes the outcome visible (`sqlite3 runner: …` on every run). - **`-readonly` on the target DB.** Tempting, and verified compatible with `.parameter` (the binding table lives in the TEMP database), but a WAL database needing journal recovery can refuse a read-only open. Adding it here risks exactly the "trades an unreachable injection for a script that does not run" outcome flagged in the #1952 thread. Worth its own issue. - **The other `2>/dev/null` sites** in this file, which also sit awkwardly with `qa/README.md`'s "Don't silence stderr". Only the §10.2 lines named in the sign-off are touched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
965 lines
51 KiB
YAML
965 lines
51 KiB
YAML
name: CI/CD Pipeline
|
||
|
||
# Documentation-only changes still trigger this workflow, on purpose. Skipping
|
||
# it at the trigger level leaves every required check in a pending state that
|
||
# can never arrive, and the pull request can then never merge. GitHub documents
|
||
# exactly this: a workflow skipped by path filtering keeps its checks pending
|
||
# and blocks the merge, while a JOB skipped by an if: conditional reports
|
||
# Success and does not. So the filtering lives on the jobs below, not here.
|
||
#
|
||
# The scope check only applies to pull requests. A push to master always runs
|
||
# the full pipeline, which keeps :edge built for every master commit. That
|
||
# matters: release-fast-path.yml re-tags :edge to :vX.Y.Z only when the :edge
|
||
# revision label matches the tagged commit, so a master commit without an image
|
||
# breaks tagging.
|
||
on:
|
||
push:
|
||
branches: [master]
|
||
pull_request:
|
||
branches: [master]
|
||
workflow_dispatch:
|
||
inputs:
|
||
images_published:
|
||
description: 'Release fast path already published the tag images'
|
||
type: boolean
|
||
default: false
|
||
|
||
permissions:
|
||
contents: read
|
||
packages: write
|
||
|
||
concurrency:
|
||
group: ci-${{ github.event.pull_request.number || github.ref }}
|
||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||
|
||
env:
|
||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
|
||
STAGING_COMPOSE_FILE: docker-compose.staging.yml
|
||
STAGING_SERVICE: staging-go
|
||
STAGING_CONTAINER: corescope-staging-go
|
||
|
||
# Pipeline (sequential, fail-fast):
|
||
# go-test → e2e-test → build-and-publish → deploy → publish-badges
|
||
# PRs stop after build-and-publish (no GHCR push). Master continues to deploy + badges.
|
||
|
||
jobs:
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 0. Change scope — decides whether the expensive jobs below run.
|
||
# ───────────────────────────────────────────────────────────────
|
||
changes:
|
||
name: "🔎 Change scope"
|
||
runs-on: ubuntu-latest
|
||
outputs:
|
||
code: ${{ steps.scope.outputs.code }}
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
with:
|
||
fetch-depth: 0
|
||
|
||
- name: Decide whether anything but documentation changed
|
||
id: scope
|
||
run: |
|
||
set -euo pipefail
|
||
# Only pull requests are scoped. Pushes and dispatches always count as
|
||
# code, so master keeps producing an :edge image and tag builds are
|
||
# never skipped.
|
||
if [ "${{ github.event_name }}" != "pull_request" ]; then
|
||
echo "code=true" >> "$GITHUB_OUTPUT"
|
||
echo "event=${{ github.event_name }} -> full pipeline"
|
||
exit 0
|
||
fi
|
||
CHANGED=$(git diff --name-only "${{ github.event.pull_request.base.sha }}" "${{ github.event.pull_request.head.sha }}")
|
||
# Empty diff means something is off; run everything rather than guess.
|
||
if [ -z "$CHANGED" ]; then
|
||
echo "code=true" >> "$GITHUB_OUTPUT"
|
||
echo "empty diff -> full pipeline"
|
||
exit 0
|
||
fi
|
||
echo "Changed files:"
|
||
echo "$CHANGED" | sed 's/^/ /'
|
||
if echo "$CHANGED" | grep -qvE '(^docs/|[.]md$|^LICENSE$)'; then
|
||
echo "code=true" >> "$GITHUB_OUTPUT"
|
||
echo "-> non-documentation files changed, full pipeline"
|
||
else
|
||
echo "code=false" >> "$GITHUB_OUTPUT"
|
||
echo "-> documentation only, heavy jobs skip (and report Success)"
|
||
fi
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 1. Go Build & Test
|
||
# ───────────────────────────────────────────────────────────────
|
||
go-test:
|
||
name: "✅ Go Build & Test"
|
||
runs-on: ubuntu-latest
|
||
needs: [changes]
|
||
if: needs.changes.outputs.code == 'true'
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
with:
|
||
fetch-depth: 0
|
||
|
||
- name: Clean Go module cache
|
||
run: rm -rf ~/go/pkg/mod 2>/dev/null || true
|
||
|
||
- name: Set up Go 1.27
|
||
uses: actions/setup-go@v6
|
||
with:
|
||
go-version: '1.27'
|
||
cache-dependency-path: |
|
||
cmd/server/go.sum
|
||
cmd/ingestor/go.sum
|
||
|
||
- name: Enforce gofmt + go vet (issue #1859)
|
||
run: |
|
||
set -e
|
||
# gofmt: every tracked *.go file except the misnamed Dockerfile.go,
|
||
# which is a Dockerfile (FROM ...) that gofmt cannot parse.
|
||
unformatted=$(gofmt -l $(git ls-files '*.go' | grep -vi 'Dockerfile.go'))
|
||
if [ -n "$unformatted" ]; then
|
||
echo "::error::gofmt required on these files:"
|
||
echo "$unformatted"
|
||
exit 1
|
||
fi
|
||
echo "gofmt: clean"
|
||
# go vet: multi-module repo, so vet each module in turn.
|
||
for mod in $(git ls-files '*go.mod' | sed 's#/go.mod##'); do
|
||
echo "== go vet $mod =="
|
||
( cd "$mod" && go vet ./... )
|
||
done
|
||
|
||
- name: Build and test Go server (with coverage)
|
||
run: |
|
||
set -e -o pipefail
|
||
cd cmd/server
|
||
go build .
|
||
# -race gates PR #1208's atomic.Pointer migration: the race-detector
|
||
# is what makes path_inspect_atomic_race_test.go actually assert.
|
||
go test -timeout 20m -race -coverprofile=server-coverage.out ./... 2>&1 | tee server-test.log
|
||
echo "--- Go Server Coverage ---"
|
||
go tool cover -func=server-coverage.out | tail -1
|
||
|
||
- name: Build and test Go ingestor (with coverage)
|
||
run: |
|
||
set -e -o pipefail
|
||
cd cmd/ingestor
|
||
go build .
|
||
go test -timeout 20m -coverprofile=ingestor-coverage.out ./... 2>&1 | tee ingestor-test.log
|
||
echo "--- Go Ingestor Coverage ---"
|
||
go tool cover -func=ingestor-coverage.out | tail -1
|
||
|
||
- name: Build and test channel library + decrypt CLI
|
||
run: |
|
||
set -e -o pipefail
|
||
cd internal/channel
|
||
go test ./...
|
||
echo "--- Channel library tests passed ---"
|
||
cd ../../cmd/decrypt
|
||
CGO_ENABLED=0 go build -ldflags="-s -w" -o corescope-decrypt .
|
||
go test ./...
|
||
echo "--- Decrypt CLI tests passed ---"
|
||
|
||
- name: Verify Dockerfile COPY invariants (issue #1316)
|
||
run: bash scripts/check-dockerfile-internal-pkgs.sh
|
||
|
||
- name: Staging disk-monitor unit tests (issue #1684)
|
||
run: bash scripts/staging/test-disk-monitor.sh
|
||
|
||
- name: QA SQL parameter-binding unit tests (issue #1977)
|
||
run: bash qa/scripts/test-blacklist-sql.sh
|
||
|
||
- name: Lint CSS variables (issue #1128)
|
||
run: |
|
||
set -e
|
||
node scripts/check-css-vars.js
|
||
node scripts/test-check-css-vars.js
|
||
|
||
- name: Run JS unit tests (packet-filter)
|
||
run: |
|
||
set -e
|
||
node test-packet-filter.js
|
||
node test-packet-filter-time.js
|
||
node test-confidence-indicator.js
|
||
node test-1659-analytics-warmup.js
|
||
node test-channels-merge-1498-unit.js
|
||
node test-issue-1518-home-url.js
|
||
node test-channel-decrypt-insecure-context.js
|
||
node test-live-region-filter.js
|
||
node test-issue-1136-observer-iata-map.js
|
||
node test-channel-qr.js
|
||
node test-channel-qr-wiring.js
|
||
node test-channel-modal-ux.js
|
||
node test-channel-issue-1087.js
|
||
node test-issue-1409-no-encrypted-flood.js
|
||
node test-channel-issue-1101.js
|
||
node test-observer-iata-1188.js
|
||
node test-pull-to-reconnect-1091.js
|
||
node test-channel-fluid-layout.js
|
||
node test-issue-1279-p2-code-filter.js
|
||
node test-area-filter.js
|
||
node test-issue-1293-marker-shapes.js
|
||
node test-issue-1356-map-a11y.js
|
||
node test-issue-1360-pill-letter-count.js
|
||
node test-issue-1364-pill-no-clamp.js
|
||
node test-issue-1375-scope-stats-fetch.js
|
||
node test-issue-1361-cb-presets.js
|
||
node test-issue-1380-cb-sim-overlay.js
|
||
node test-issue-1380-cb-reset-button.js
|
||
node test-issue-1407-cb-preset-propagation.js
|
||
node test-issue-1412-customizer-no-override.js
|
||
node test-issue-1418-raw-hex-extraction.js
|
||
node test-issue-1418-edge-weights.js
|
||
node test-issue-1418-cb-preset-ramp.js
|
||
node test-issue-1418-spider-fan.js
|
||
node test-issue-1418-deeplink-hops-channels.js
|
||
node test-issue-1418-polish-review.js
|
||
node test-issue-1420-tile-providers.js
|
||
node test-issue-1614-tile-url-function.js
|
||
node test-issue-1438-marker-css-vars.js
|
||
node test-issue-1846-observers-width.js
|
||
node test-issue-1562-observers-summary.js
|
||
node test-issue-1509-nav-active-bg.js
|
||
node test-issue-1509-detect-preset.js
|
||
node test-live.js
|
||
node test-coverage-gate.js
|
||
node test-node-reach-coverage.js
|
||
node test-issue-1107-live-layout.js
|
||
node test-issue-1532-live-fullscreen.js
|
||
node test-issue-1619-feed-detail-card-draggable.js
|
||
node test-xss-escape-sinks.js
|
||
node test-preflight-xss-gate.js
|
||
node test-traces.js
|
||
node test-issue-1648-m4-emoji-scan.js
|
||
node test-issue-1753-copy-url-slash.js
|
||
node test-issue-1668-m3-typography.js
|
||
node test-mqtt-status-panel.js
|
||
node test-issue-1697-mqtt-mobile-e2e.js
|
||
node test-warmup-banner.js
|
||
node test-issue-1633-hide-1byte-hops.js
|
||
node test-issue-1668-m4-per-route.js
|
||
node test-a11y-axe-1668-selftest.js
|
||
node test-a11y-1716-rf-range-btn-active.js
|
||
node test-issue-1705-subpath-contrast.js
|
||
node test-issue-1770-mobile-row-clamp.js
|
||
node test-a11y-axe-routes-coverage.js
|
||
node test-issue-1956-release-routing.js
|
||
|
||
- name: 🛡️ Preflight XSS gate — actual --diff check (PR only)
|
||
# The fixture self-test above (test-preflight-xss-gate.js) only
|
||
# asserts the script's behavior against fixtures. It does NOT scan
|
||
# the PR's own changes. This step closes that gap by running the
|
||
# gate against added lines in public/**/*.{js,html} on the PR.
|
||
# Gate is PR-scoped only (per djb finding: merge commits would
|
||
# slip an opt-out otherwise). Master pushes skip this step.
|
||
if: github.event_name == 'pull_request'
|
||
env:
|
||
PR_BODY: ${{ github.event.pull_request.body }}
|
||
PREFLIGHT_PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ' ') }}
|
||
run: |
|
||
set -e
|
||
git fetch origin master --depth=50 2>&1 | tail -3 || true
|
||
# Materialize PR body to a file for the opt-out parser.
|
||
printf '%s' "$PR_BODY" > /tmp/pr-body.md
|
||
PREFLIGHT_PR_BODY=/tmp/pr-body.md bash scripts/check-xss-sinks.sh --diff origin/master
|
||
|
||
- name: 🧹 Frontend lint (eslint no-undef) — issue #1342
|
||
run: |
|
||
set -e
|
||
# Use eslint@8 (legacy .eslintrc.json). Don't migrate to flat-config / eslint@9.
|
||
# --no-save: avoid touching package.json / no committed node_modules.
|
||
npm install --no-save --no-audit --no-fund eslint@8
|
||
npx eslint public/*.js
|
||
|
||
- name: Verify proto syntax
|
||
run: |
|
||
set -e
|
||
sudo apt-get update -qq
|
||
sudo apt-get install -y protobuf-compiler
|
||
for proto in proto/*.proto; do
|
||
echo " ✓ $(basename "$proto")"
|
||
protoc --proto_path=proto --descriptor_set_out=/dev/null "$proto"
|
||
done
|
||
echo "✅ All .proto files are syntactically valid"
|
||
|
||
- name: Generate Go coverage badges
|
||
if: success()
|
||
run: |
|
||
mkdir -p .badges
|
||
|
||
SERVER_COV="0"
|
||
if [ -f cmd/server/server-coverage.out ]; then
|
||
SERVER_COV=$(cd cmd/server && go tool cover -func=server-coverage.out | tail -1 | grep -oP '[\d.]+(?=%)')
|
||
fi
|
||
SERVER_COLOR="red"
|
||
if [ "$(echo "$SERVER_COV >= 80" | bc -l 2>/dev/null)" = "1" ]; then SERVER_COLOR="green"
|
||
elif [ "$(echo "$SERVER_COV >= 60" | bc -l 2>/dev/null)" = "1" ]; then SERVER_COLOR="yellow"; fi
|
||
echo "{\"schemaVersion\":1,\"label\":\"go server coverage\",\"message\":\"${SERVER_COV}%\",\"color\":\"${SERVER_COLOR}\"}" > .badges/go-server-coverage.json
|
||
|
||
INGESTOR_COV="0"
|
||
if [ -f cmd/ingestor/ingestor-coverage.out ]; then
|
||
INGESTOR_COV=$(cd cmd/ingestor && go tool cover -func=ingestor-coverage.out | tail -1 | grep -oP '[\d.]+(?=%)')
|
||
fi
|
||
INGESTOR_COLOR="red"
|
||
if [ "$(echo "$INGESTOR_COV >= 80" | bc -l 2>/dev/null)" = "1" ]; then INGESTOR_COLOR="green"
|
||
elif [ "$(echo "$INGESTOR_COV >= 60" | bc -l 2>/dev/null)" = "1" ]; then INGESTOR_COLOR="yellow"; fi
|
||
echo "{\"schemaVersion\":1,\"label\":\"go ingestor coverage\",\"message\":\"${INGESTOR_COV}%\",\"color\":\"${INGESTOR_COLOR}\"}" > .badges/go-ingestor-coverage.json
|
||
|
||
echo "## Go Coverage" >> $GITHUB_STEP_SUMMARY
|
||
echo "| Module | Coverage |" >> $GITHUB_STEP_SUMMARY
|
||
echo "|--------|----------|" >> $GITHUB_STEP_SUMMARY
|
||
echo "| Server | ${SERVER_COV}% |" >> $GITHUB_STEP_SUMMARY
|
||
echo "| Ingestor | ${INGESTOR_COV}% |" >> $GITHUB_STEP_SUMMARY
|
||
|
||
- name: Upload Go coverage badges
|
||
if: success()
|
||
uses: actions/upload-artifact@v6
|
||
with:
|
||
name: go-badges
|
||
path: .badges/go-*.json
|
||
retention-days: 1
|
||
if-no-files-found: ignore
|
||
include-hidden-files: true
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 2. Playwright E2E Tests (against Go server with fixture DB)
|
||
# ───────────────────────────────────────────────────────────────
|
||
e2e-test:
|
||
name: "🎭 Playwright E2E Tests"
|
||
needs: [go-test, changes]
|
||
runs-on: ubuntu-latest
|
||
defaults:
|
||
run:
|
||
shell: bash
|
||
if: needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published)
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
with:
|
||
fetch-depth: 0
|
||
|
||
- name: Set up Node.js 22
|
||
uses: actions/setup-node@v5
|
||
with:
|
||
node-version: '22'
|
||
|
||
- name: Clean Go module cache
|
||
run: rm -rf ~/go/pkg/mod 2>/dev/null || true
|
||
|
||
- name: Set up Go 1.27
|
||
uses: actions/setup-go@v6
|
||
with:
|
||
go-version: '1.27'
|
||
cache-dependency-path: cmd/server/go.sum
|
||
|
||
- name: Build Go server
|
||
run: |
|
||
cd cmd/server
|
||
go build -o ../../corescope-server .
|
||
echo "Go server built successfully"
|
||
|
||
- name: Build Go migrate tool
|
||
run: |
|
||
cd cmd/migrate
|
||
go build -o ../../corescope-migrate .
|
||
echo "Go migrate tool built successfully"
|
||
|
||
- name: Install npm dependencies
|
||
run: npm ci --production=false
|
||
|
||
- name: Install Playwright browser
|
||
run: |
|
||
npx playwright install chromium 2>/dev/null || true
|
||
npx playwright install-deps chromium 2>/dev/null || true
|
||
|
||
- name: Instrument frontend JS for coverage
|
||
run: sh scripts/instrument-frontend.sh
|
||
|
||
- name: Freshen fixture timestamps
|
||
run: bash tools/freshen-fixture.sh test-fixtures/e2e-fixture.db
|
||
|
||
- name: Seed grouped-packet row for #1486 collapse test
|
||
# The committed fixture has 499 packets, each with exactly ONE
|
||
# observation, so the packets-page renders only flat
|
||
# (select-hash) rows. The #1486 repro needs at least one grouped
|
||
# (toggle-select) row. Insert a NEW transmission with 3
|
||
# observations.
|
||
#
|
||
# The server's async hash-migrate (cmd/server/hash_migrate.go)
|
||
# recomputes `transmissions.hash` from `raw_hex` via
|
||
# ComputeContentHash(), so the inserted hash MUST equal that
|
||
# function's output for the chosen raw_hex — otherwise the row
|
||
# gets relabelled and the E2E can't find it.
|
||
#
|
||
# raw_hex 15000102030405060708090a0b0c0d0e0f
|
||
# → header=0x15 (route_type=1, payload_type=5)
|
||
# → ComputeContentHash(...) = fae0c9e6d357a814
|
||
#
|
||
# The first_seen / observation timestamps are pinned to a date
|
||
# within retentionHours but outside the default 15-min UI
|
||
# window so the row is hidden in the default view (keeping
|
||
# test-e2e-playwright's first-10-rows hex-pane test
|
||
# unaffected) and reachable via the explicit ?timeWindow=0
|
||
# deep-link the #1486 test uses.
|
||
run: |
|
||
sqlite3 test-fixtures/e2e-fixture.db <<'SQL'
|
||
-- Sort the seeded row LAST in BOTH default packets views:
|
||
-- • flat view sorts by transmissions.id DESC → id=0 puts it last
|
||
-- • grouped view (#default for the packets page) sorts by
|
||
-- MAX(observations.timestamp) DESC → we must keep our obs
|
||
-- timestamps OLDER than every other fixture observation.
|
||
-- Fixture (after freshen) has obs timestamps spanning
|
||
-- 2026-05-17 16:01:39Z .. 2026-05-28 00:00:00Z (max).
|
||
-- Note: freshen only shifts transmissions.first_seen forward
|
||
-- to ~now; observation.timestamp is left alone except for
|
||
-- the timestamp=0 case.
|
||
-- Use 2026-05-15 (~2 days older than the oldest fixture obs)
|
||
-- so our row sorts LAST in the grouped view too, keeping
|
||
-- test-e2e-playwright's first-10-rows hex-pane test
|
||
-- unaffected. The #1486 test still reaches the row via the
|
||
-- explicit hash + ?timeWindow=0 deep-link.
|
||
INSERT INTO transmissions(id,raw_hex,hash,first_seen,route_type,payload_type,payload_version,decoded_json,channel_hash,from_pubkey)
|
||
VALUES (0,'15000102030405060708090a0b0c0d0e0f','fae0c9e6d357a814','2026-05-15T00:00:00Z',1,5,0,'{"type":"CHAN","channel":"#test","text":"#1486 fixture"}',NULL,NULL);
|
||
INSERT INTO observations(transmission_id,observer_idx,direction,snr,rssi,score,path_json,timestamp,resolved_path) VALUES
|
||
(0,1,'rx',5.0,-95,0,'["AA"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["aa00000000000000000000000000000000000000000000000000000000000000"]'),
|
||
(0,2,'rx',5.5,-92,0,'["BB"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["bb00000000000000000000000000000000000000000000000000000000000000"]'),
|
||
(0,3,'rx',6.0,-90,0,'["CC"]',CAST(strftime('%s','2026-05-15T00:00:00Z') AS INTEGER),'["cc00000000000000000000000000000000000000000000000000000000000000"]');
|
||
-- #1791 fixture: a single GRP_DATA (payload_type=6) packet so the
|
||
-- E2E "Group Data filter" test has at least one row to filter on.
|
||
-- Use an obs timestamp within the default UI window so the row
|
||
-- appears with no time-window override.
|
||
--
|
||
-- raw_hex header byte 0x19 = bits 5-2 (payload)=0110=6 (GRP_DATA),
|
||
-- bits 1-0 (route)=01=1 (FLOOD).
|
||
-- path_len byte 0x00 = hash_size=1, hash_count=0 (zero-hop on-wire,
|
||
-- typical GRP_DATA going FLOOD). path_json/resolved_path are kept
|
||
-- EMPTY so the rendered hop-row count matches the hex-path byte
|
||
-- count (a prior fixture used path_json=["AA"] but raw_hex
|
||
-- path_len=0, which broke the "hex strip Path range matches hop
|
||
-- row count" E2E).
|
||
--
|
||
-- Note: id=-1000000 is a deliberately out-of-band sentinel id so
|
||
-- this synthetic fixture row cannot collide with real ingested
|
||
-- transmissions (real ids are positive autoincrement values).
|
||
INSERT INTO transmissions(id,raw_hex,hash,first_seen,route_type,payload_type,payload_version,decoded_json,channel_hash,from_pubkey)
|
||
VALUES (-1000000,'19000102030405060708090a0b0c0d0e0f','17910000deadbeef',strftime('%Y-%m-%dT%H:%M:%SZ','now'),1,6,0,'{"type":"GRP_DATA","channel":"#test","raw":"deadbeef"}',NULL,NULL);
|
||
INSERT INTO observations(transmission_id,observer_idx,direction,snr,rssi,score,path_json,timestamp,resolved_path) VALUES
|
||
(-1000000,1,'rx',7.0,-88,0,'[]',CAST(strftime('%s','now') AS INTEGER),'[]');
|
||
SQL
|
||
|
||
- name: Migrate fixture DB to current schema (#1287)
|
||
# Server now ASSERTs schema is migrated and refuses to start
|
||
# otherwise (cmd/server/main.go: dbschema.AssertReady). In prod
|
||
# the ingestor owns dbschema.Apply, but CI starts only the
|
||
# server against the committed e2e fixture — so we run the
|
||
# standalone migrate tool here to bring the fixture up to the
|
||
# required shape before the server boots.
|
||
run: ./corescope-migrate -db test-fixtures/e2e-fixture.db
|
||
|
||
- name: Start Go server with fixture DB
|
||
run: |
|
||
fuser -k 13581/tcp 2>/dev/null || true
|
||
sleep 1
|
||
./corescope-server -port 13581 -db test-fixtures/e2e-fixture.db -public public-instrumented &
|
||
echo $! > .server.pid
|
||
for i in $(seq 1 30); do
|
||
if curl -sf http://localhost:13581/api/healthz > /dev/null 2>&1; then
|
||
echo "Server ready after ${i}s"
|
||
break
|
||
fi
|
||
if [ "$i" -eq 30 ]; then
|
||
echo "Server failed to start within 30s"
|
||
exit 1
|
||
fi
|
||
sleep 1
|
||
done
|
||
|
||
- name: Run Playwright E2E tests (fail-fast)
|
||
run: |
|
||
BASE_URL=http://localhost:13581 node test-e2e-playwright.js 2>&1 | tee e2e-output.txt
|
||
# M5+M6 of #1668 — axe-core CI gate.
|
||
# M5: color-contrast on desktop dark+light.
|
||
# M6: expanded ruleset (image-alt, label, aria-required-attr,
|
||
# aria-valid-attr, aria-valid-attr-value, landmark-one-main,
|
||
# region, button-name, link-name, document-title, html-has-lang,
|
||
# duplicate-id) AND adds 375x812 mobile viewport (with
|
||
# color-contrast on mobile too).
|
||
# Allowlist: tests/a11y-allowlist.yaml (0 entries — hard pass policy).
|
||
# Per-viewport summary printed at the end; any net>0 fails the build.
|
||
BASE_URL=http://localhost:13581 AXE_SCREENSHOT_DIR=/tmp/axe-1668 \
|
||
node test-a11y-axe-1668.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-filter-ux-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-issue-1087-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-issue-1111-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-map-modal-fluid-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-map-nodes-pagination-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-observer-iata-1188-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1639-observers-sort-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1758-ng-filter-rerenders-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-fluid-1055-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-priority-1102-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-priority-1311-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-priority-1391-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1413-nav-overlap-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1400-nav-vertical-clip.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-more-floor-1139-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-bottom-nav-1061-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-gestures-1062-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-gestures-1185-scroll-discriminator-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-gesture-hints-1065-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-touch-gestures-coverage-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-fluid-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-table-fluid-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-charts-fluid-1058-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-slideover-1056-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1692-packets-init-parallel-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-slideover-1168-munger-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-logo-pulse-1173-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1122-packets-filter-ux-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1128-packets-layout-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1128-multi-viewport-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1136-live-region-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-live-multibyte-only-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1150-404-state-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1146-path-link-contrast-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 node test-issue-1705-subpath-contrast-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1147-section-order-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1151-orphan-separators-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1486-collapse-reopens-detail-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-logo-rebrand-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-logo-theme-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-logo-default-sage-teal-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1109-hamburger-dropdown-visible-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-live-layout-1178-1179-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1205-live-controls-anchor-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-live-mql-leak-1180-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1204-live-panel-structure-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1234-live-chrome-pass2-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1206-vcr-overlap-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1244-live-vcr-row-hints-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1510-live-nav-pin-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-live-fullscreen-1572-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1599-replay-freeze-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1648-m1-icons-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1648-m2-icons-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1648-m3-icons-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1648-m4-icons-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1657-analytics-channels-group-sprites-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1224-channels-mobile-ux-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1367-channels-chat-app-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1236-map-mobile-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1329-map-controls-accordion-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1273-qr-overlay-height-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1281-location-row-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1279-legend-p2-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-issue-1799-label-vocab-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-home-coverage-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-path-inspector-coverage-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1206-resize-observer-leak-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-nav-drawer-1064-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-audio-live-1297-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-audio-lab-1297-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-decrypt-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-qr-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-channel-color-picker-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-customize-theme-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-customize-branding-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-customize-display-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
BASE_URL=http://localhost:13581 node test-customize-export-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-drag-manager-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1567-corner-clears-drag-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1306-collisions-terminology-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1374-route-map-a11y-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-list-render-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-selection-flow-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-add-modal-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-share-color-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-ws-batch-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-ws-race-1498-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1487-byop-modal-layout-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1630-reach-mobile-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-node-reach-coverage-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1640-compare-discovery-e2e.js 2>&1 | tee -a e2e-output.txt
|
||
|
||
# #1616: slide-over focus-restore flake-gate. Runs the slide-over
|
||
# E2E 20 consecutive times against the SAME backend instance so
|
||
# the Chromium-headless focus race documented in #1172/#1616 has
|
||
# a 20× shot at firing. Any single non-zero exit aborts. This is
|
||
# the architectural-fix gate — if it ever turns red post-merge,
|
||
# the focused-but-hidden state has crept back in.
|
||
#
|
||
# PERMANENT step. Adds ~3-4 min to the e2e-test job in exchange
|
||
# for closing out a flake family that was blocking ~8 unrelated
|
||
# PRs at a time. If profiling pressures the budget later, drop
|
||
# repeat count first; do not delete.
|
||
- name: Slide-over E2E flake-gate (#1616, --repeat-each=3)
|
||
run: |
|
||
set -e
|
||
for i in $(seq 1 3); do
|
||
echo "--- slide-over E2E run $i/20 ---"
|
||
BASE_URL=http://localhost:13581 node test-slideover-1056-e2e.js 2>&1 | tee -a slideover-repeat-output.txt
|
||
done
|
||
echo "3 passed"
|
||
|
||
- name: Collect frontend coverage (parallel)
|
||
if: success() && github.event_name == 'push'
|
||
run: |
|
||
BASE_URL=http://localhost:13581 node scripts/collect-frontend-coverage.js 2>&1 | tee fe-coverage-output.txt || true
|
||
|
||
- name: Generate frontend coverage badges
|
||
if: success()
|
||
run: |
|
||
# Aggregate per-suite PASS/FAIL across every test-*-e2e.js summary.
|
||
# The previous regex (grep -oP '[0-9]+(?=/)' | tail -1) caught a
|
||
# stray digits-before-slash like the '2' in '2/3 tests passed' from
|
||
# some sub-output and stamped the badge as '2 passed'. See #1296.
|
||
eval "$(bash scripts/aggregate-e2e-pass.sh e2e-output.txt)"
|
||
E2E_PASS=${PASS:-0}
|
||
E2E_FAIL=${FAIL:-0}
|
||
|
||
mkdir -p .badges
|
||
if [ -f .nyc_output/frontend-coverage.json ] || [ -f .nyc_output/e2e-coverage.json ]; then
|
||
npx nyc report --reporter=text-summary --reporter=text 2>&1 | tee fe-report.txt
|
||
FE_COVERAGE=$(grep 'Statements' fe-report.txt | head -1 | grep -oP '[\d.]+(?=%)' || echo "0")
|
||
FE_COVERAGE=${FE_COVERAGE:-0}
|
||
FE_COLOR="red"
|
||
[ "$(echo "$FE_COVERAGE > 50" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="yellow"
|
||
[ "$(echo "$FE_COVERAGE > 80" | bc -l 2>/dev/null)" = "1" ] && FE_COLOR="brightgreen"
|
||
echo "{\"schemaVersion\":1,\"label\":\"frontend coverage\",\"message\":\"${FE_COVERAGE}%\",\"color\":\"${FE_COLOR}\"}" > .badges/frontend-coverage.json
|
||
echo "## Frontend: ${FE_COVERAGE}% coverage" >> $GITHUB_STEP_SUMMARY
|
||
fi
|
||
if [ "${E2E_FAIL:-0}" -gt 0 ]; then
|
||
E2E_MSG="${E2E_PASS:-0} passed, ${E2E_FAIL} failed"
|
||
E2E_COLOR="red"
|
||
else
|
||
E2E_MSG="${E2E_PASS:-0} passed"
|
||
E2E_COLOR="brightgreen"
|
||
fi
|
||
echo "{\"schemaVersion\":1,\"label\":\"e2e tests\",\"message\":\"${E2E_MSG}\",\"color\":\"${E2E_COLOR}\"}" > .badges/e2e-tests.json
|
||
|
||
- name: Stop test server
|
||
if: always()
|
||
run: |
|
||
if [ -f .server.pid ]; then
|
||
kill $(cat .server.pid) 2>/dev/null || true
|
||
rm -f .server.pid
|
||
fi
|
||
|
||
- name: Upload E2E badges
|
||
if: success()
|
||
uses: actions/upload-artifact@v6
|
||
with:
|
||
name: e2e-badges
|
||
path: .badges/
|
||
retention-days: 1
|
||
if-no-files-found: ignore
|
||
include-hidden-files: true
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 3. Build & Publish Docker Image
|
||
# ───────────────────────────────────────────────────────────────
|
||
# The five GHCR steps below publish on a push to master (the :edge image) and
|
||
# on a tag ref. The tag half is not decoration: release-fast-path.yml re-tags
|
||
# :edge to :vX.Y.Z when the :edge revision label matches the tagged commit,
|
||
# and dispatches THIS workflow when it does not. That fallback previously
|
||
# produced no image at all, because a workflow_dispatch is not a push and
|
||
# every publishing step was gated on github.event_name == 'push'. It built
|
||
# locally, reported success, and pushed nothing.
|
||
#
|
||
# That went unnoticed until v3.10.0, where the tagged commit was
|
||
# documentation-only. Documentation-only commits skip this workflow (see the
|
||
# paths-ignore above), so no :edge image was ever built for it, the label
|
||
# comparison failed, and the fallback ran for the first time.
|
||
build-and-publish:
|
||
name: "🏗️ Build & Publish Docker Image"
|
||
needs: [e2e-test, changes]
|
||
runs-on: ubuntu-latest
|
||
if: needs.changes.outputs.code == 'true' && !(startsWith(github.ref, 'refs/tags/v') && inputs.images_published)
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
|
||
- name: Compute build metadata
|
||
id: meta
|
||
run: |
|
||
BUILD_TIME=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
|
||
GIT_COMMIT="${GITHUB_SHA::7}"
|
||
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
|
||
APP_VERSION="${GITHUB_REF#refs/tags/}"
|
||
else
|
||
APP_VERSION="edge"
|
||
fi
|
||
echo "build_time=$BUILD_TIME" >> "$GITHUB_OUTPUT"
|
||
echo "git_commit=$GIT_COMMIT" >> "$GITHUB_OUTPUT"
|
||
echo "app_version=$APP_VERSION" >> "$GITHUB_OUTPUT"
|
||
echo "Build: version=$APP_VERSION commit=$GIT_COMMIT time=$BUILD_TIME"
|
||
|
||
- name: Build Go Docker image (local staging)
|
||
run: |
|
||
GIT_COMMIT="${{ steps.meta.outputs.git_commit }}" \
|
||
APP_VERSION="${{ steps.meta.outputs.app_version }}" \
|
||
BUILD_TIME="${{ steps.meta.outputs.build_time }}" \
|
||
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging build "$STAGING_SERVICE"
|
||
echo "Built Go staging image ✅"
|
||
|
||
- name: Set up Docker Buildx
|
||
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
|
||
uses: docker/setup-buildx-action@v3
|
||
|
||
- name: Set up QEMU (arm64 runtime stage)
|
||
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
|
||
uses: docker/setup-qemu-action@v3
|
||
|
||
- name: Log in to GHCR
|
||
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
|
||
uses: docker/login-action@v3
|
||
with:
|
||
registry: ghcr.io
|
||
username: ${{ github.actor }}
|
||
password: ${{ secrets.GITHUB_TOKEN }}
|
||
|
||
- name: Extract Docker metadata
|
||
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
|
||
id: docker-meta
|
||
uses: docker/metadata-action@v5
|
||
with:
|
||
images: ghcr.io/kpa-clawbot/corescope
|
||
tags: |
|
||
type=semver,pattern=v{{version}}
|
||
type=semver,pattern=v{{major}}.{{minor}}
|
||
type=semver,pattern=v{{major}}
|
||
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
||
type=edge,branch=master
|
||
|
||
- name: Build and push to GHCR
|
||
if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }}
|
||
uses: docker/build-push-action@v6
|
||
with:
|
||
context: .
|
||
push: true
|
||
platforms: linux/amd64,linux/arm64
|
||
tags: ${{ steps.docker-meta.outputs.tags }}
|
||
labels: ${{ steps.docker-meta.outputs.labels }}
|
||
build-args: |
|
||
APP_VERSION=${{ steps.meta.outputs.app_version }}
|
||
GIT_COMMIT=${{ steps.meta.outputs.git_commit }}
|
||
BUILD_TIME=${{ steps.meta.outputs.build_time }}
|
||
cache-from: type=gha
|
||
cache-to: type=gha,mode=max
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 4. Release Artifacts (tags only)
|
||
# ───────────────────────────────────────────────────────────────
|
||
release-artifacts:
|
||
name: "📦 Release Artifacts"
|
||
needs: [go-test, changes]
|
||
runs-on: ubuntu-latest
|
||
permissions:
|
||
contents: write
|
||
if: startsWith(github.ref, 'refs/tags/v') && needs.changes.outputs.code == 'true'
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
|
||
- name: Set up Go 1.27
|
||
uses: actions/setup-go@v6
|
||
with:
|
||
go-version: '1.27'
|
||
|
||
- name: Build corescope-decrypt (static, linux/amd64)
|
||
run: |
|
||
cd cmd/decrypt
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-amd64 .
|
||
|
||
- name: Build corescope-decrypt (static, linux/arm64)
|
||
run: |
|
||
cd cmd/decrypt
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-arm64 .
|
||
|
||
- name: Upload release assets
|
||
# Standard releases upload both assets to a draft before publishing.
|
||
# Keep one writer: published releases are immutable in this repository.
|
||
uses: softprops/action-gh-release@v2
|
||
with:
|
||
fail_on_unmatched_files: true
|
||
files: |
|
||
corescope-decrypt-linux-amd64
|
||
corescope-decrypt-linux-arm64
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 4b. Deploy Staging (master only)
|
||
# ───────────────────────────────────────────────────────────────
|
||
deploy:
|
||
name: "🚀 Deploy Staging"
|
||
# DISABLED. Re-enable by setting the repository variable
|
||
# ENABLE_STAGING_DEPLOY to 'true' (Settings > Secrets and variables >
|
||
# Actions > Variables). No code change needed.
|
||
#
|
||
# Why: this job runs on [self-hosted, meshcore-runner-2] and that runner
|
||
# has not picked up a job since at least 2026-08-31. It has no
|
||
# timeout-minutes, so it sat queued for 22+ hours, held its run open, and
|
||
# through the concurrency group ci-refs/heads/master caused GitHub to
|
||
# cancel every subsequent master push. The result was that master produced
|
||
# no completed pipeline result at all: 30+ runs cancelled or stuck while
|
||
# go-test, e2e-test and build-and-publish were passing inside them.
|
||
#
|
||
# timeout-minutes alone would unblock the queue but leave master
|
||
# permanently red on a job that cannot succeed while the runner is absent,
|
||
# so the deploy is gated instead. It is added here as well, so that if the
|
||
# variable is set while the runner is still missing the job fails in ten
|
||
# minutes rather than blocking the branch again.
|
||
if: |
|
||
vars.ENABLE_STAGING_DEPLOY == 'true'
|
||
&& (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||
&& github.ref == 'refs/heads/master'
|
||
needs: [build-and-publish]
|
||
runs-on: [self-hosted, meshcore-runner-2]
|
||
timeout-minutes: 10
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
|
||
- name: Pull latest image from GHCR
|
||
run: |
|
||
# Try to pull the edge image from GHCR and tag for docker-compose compatibility
|
||
if docker pull ghcr.io/kpa-clawbot/corescope:edge; then
|
||
docker tag ghcr.io/kpa-clawbot/corescope:edge corescope-go:latest
|
||
echo "Pulled and tagged GHCR edge image ✅"
|
||
else
|
||
echo "⚠️ GHCR pull failed — falling back to locally built image"
|
||
fi
|
||
|
||
- name: Deploy staging
|
||
run: |
|
||
# Force-remove the staging container regardless of how it was created
|
||
# (compose-managed OR manually created via docker run)
|
||
docker stop corescope-staging-go 2>/dev/null || true
|
||
docker rm -f corescope-staging-go 2>/dev/null || true
|
||
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging down --timeout 30 2>/dev/null || true
|
||
|
||
# Wait for container to be fully gone and OS to reclaim memory (3GB limit)
|
||
for i in $(seq 1 15); do
|
||
if ! docker ps -a --format '{{.Names}}' | grep -q 'corescope-staging-go'; then
|
||
break
|
||
fi
|
||
sleep 1
|
||
done
|
||
sleep 5 # extra pause for OS memory reclaim
|
||
|
||
# Ensure staging data dir exists (config.json lives here, no separate file mount)
|
||
STAGING_DATA="${STAGING_DATA_DIR:-$HOME/meshcore-staging-data}"
|
||
mkdir -p "$STAGING_DATA"
|
||
|
||
# If no config exists, copy the example (CI doesn't have a real prod config)
|
||
if [ ! -f "$STAGING_DATA/config.json" ]; then
|
||
echo "Staging config missing — copying config.example.json"
|
||
cp config.example.json "$STAGING_DATA/config.json" 2>/dev/null || true
|
||
fi
|
||
|
||
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging up -d staging-go
|
||
|
||
- name: Healthcheck staging container
|
||
run: |
|
||
for i in $(seq 1 120); do
|
||
HEALTH=$(docker inspect corescope-staging-go --format '{{.State.Health.Status}}' 2>/dev/null || echo "starting")
|
||
if [ "$HEALTH" = "healthy" ]; then
|
||
echo "Staging healthy after ${i}s"
|
||
break
|
||
fi
|
||
if [ "$i" -eq 120 ]; then
|
||
echo "Staging failed health check after 120s"
|
||
docker logs corescope-staging-go --tail 50
|
||
exit 1
|
||
fi
|
||
sleep 1
|
||
done
|
||
|
||
- name: Smoke test staging API
|
||
run: |
|
||
PORT="${STAGING_GO_HTTP_PORT:-80}"
|
||
if curl -sf "http://localhost:${PORT}/api/stats" | grep -q engine; then
|
||
echo "Staging verified — engine field present ✅"
|
||
else
|
||
echo "Staging /api/stats did not return engine field (port ${PORT})"
|
||
exit 1
|
||
fi
|
||
|
||
- name: Clean up old Docker images
|
||
if: always()
|
||
run: |
|
||
# Remove dangling images and images older than 24h (keeps current build)
|
||
echo "--- Docker disk usage before cleanup ---"
|
||
docker system df
|
||
docker image prune -af --filter "until=24h" 2>/dev/null || true
|
||
docker builder prune -f --keep-storage=1GB 2>/dev/null || true
|
||
echo "--- Docker disk usage after cleanup ---"
|
||
docker system df
|
||
|
||
# ───────────────────────────────────────────────────────────────
|
||
# 5. Publish Badges & Summary (master only)
|
||
# ───────────────────────────────────────────────────────────────
|
||
publish:
|
||
name: "📝 Publish Badges & Summary"
|
||
if: github.event_name == 'push'
|
||
# Was needs: [deploy]. The badges are built from the go-test and e2e-test
|
||
# artifacts and never needed the deploy step; depending on it meant the
|
||
# coverage badges stopped updating whenever the self-hosted runner was
|
||
# unavailable. build-and-publish already transitively requires both test
|
||
# jobs, so ordering is unchanged.
|
||
needs: [build-and-publish]
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- name: Checkout code
|
||
uses: actions/checkout@v5
|
||
|
||
- name: Download Go coverage badges
|
||
continue-on-error: true
|
||
uses: actions/download-artifact@v6
|
||
with:
|
||
name: go-badges
|
||
path: .badges/
|
||
|
||
- name: Download E2E badges
|
||
continue-on-error: true
|
||
uses: actions/download-artifact@v6
|
||
with:
|
||
name: e2e-badges
|
||
path: .badges/
|
||
|
||
- name: Publish coverage badges to repo
|
||
continue-on-error: true
|
||
env:
|
||
GH_TOKEN: ${{ secrets.BADGE_PUSH_TOKEN }}
|
||
run: |
|
||
# GITHUB_TOKEN cannot push to protected branches (required status checks).
|
||
# Use admin PAT (BADGE_PUSH_TOKEN) via GitHub Contents API instead.
|
||
for badge in .badges/*.json; do
|
||
FILENAME=$(basename "$badge")
|
||
FILEPATH=".badges/$FILENAME"
|
||
CONTENT=$(base64 -w0 "$badge")
|
||
CURRENT_SHA=$(gh api "repos/${{ github.repository }}/contents/$FILEPATH" --jq '.sha' 2>/dev/null || echo "")
|
||
if [ -n "$CURRENT_SHA" ]; then
|
||
gh api "repos/${{ github.repository }}/contents/$FILEPATH" \
|
||
-X PUT \
|
||
-f message="ci: update $FILENAME [skip ci]" \
|
||
-f content="$CONTENT" \
|
||
-f sha="$CURRENT_SHA" \
|
||
-f branch="master" \
|
||
--silent 2>&1 || echo "Failed to update $FILENAME"
|
||
else
|
||
gh api "repos/${{ github.repository }}/contents/$FILEPATH" \
|
||
-X PUT \
|
||
-f message="ci: update $FILENAME [skip ci]" \
|
||
-f content="$CONTENT" \
|
||
-f branch="master" \
|
||
--silent 2>&1 || echo "Failed to create $FILENAME"
|
||
fi
|
||
done
|
||
echo "Badge publish complete"
|
||
|
||
- name: Post deployment summary
|
||
run: |
|
||
echo "## Staging Deployed ✓" >> $GITHUB_STEP_SUMMARY
|
||
echo "" >> $GITHUB_STEP_SUMMARY
|
||
echo "**Commit:** \`$(git rev-parse --short HEAD)\` — $(git log -1 --format=%s)" >> $GITHUB_STEP_SUMMARY
|