From b8c8d98e618529883ee631233f71802f619a06b1 Mon Sep 17 00:00:00 2001 From: efiten Date: Wed, 16 Sep 2026 11:07:18 +0200 Subject: [PATCH] fix(release): keep every platform when re-tagging :edge as a release (#2031) ## Problem `crane mutate` works on one image, not on an index. Pointed at the multi-arch `:edge` tag it silently resolves the default platform, so the fast path published v3.11.0 as a single amd64 OCI manifest, and `crane tag` then pointed `v3.11`, `v3` and `latest` at that same manifest. `docker pull` on arm64 against any of those four tags fails. Verified in the registry: | tag | shape | arch | |---|---|---| | `v3.9.2`, `v3.10`, `v3.10.1`, `edge` | index, 4 children | multi-arch | | `v3.11.0`, `v3.11`, `v3`, `latest` | `oci.image.manifest.v1`, 17 layers | amd64 only, revision `a2ea18f7` | Earlier releases are indexes, so this only hit v3.11.0. The GitHub release and both `corescope-decrypt` binaries are unaffected. ## Change - The fast path now reads the `:edge` manifest, mutates each runnable platform child by digest (`/app/.image-version` plus the version label, as #1807 intended) and reassembles an index with `crane index append`. A single-platform `:edge` still takes the old single mutate path. - Attestation manifests (`platform.architecture == "unknown"`) are not carried over: they reference the pre-mutation digests, so copying them would attest the wrong images. - A new verification step compares the platform set of `vX.Y.Z`, `vX.Y`, `vX` and `latest` against `:edge` and fails the run if any of them differs. A release tag that resolves to one platform is worse than a slow release, so this should break the build rather than ship. - Scratch tags (`tmp-vX.Y.Z-linux-amd64`, ...) are deleted best-effort afterwards; the index references the manifests by digest, so leaving them behind is only untidy. - Added `workflow_dispatch` with a `tag` input to republish the images for an existing release. A dispatched run resolves the tagged commit itself, because `github.sha` is then the ref the workflow file came from, and it skips the `deploy.yml` dispatch: that release already exists and releases here are immutable (the trap from #1955/#1956). ## Tests None: this repository has no harness that executes workflow files, and the CI jobs cannot reach a step that pushes to GHCR. What the change is verified against instead: - `crane index append` accepts `-m/--manifest` repeated plus `-t/--tag`, with the base index optional, so building an index from scratch is supported (crane docs for `index append`). - The YAML parses and every `run:` block passes `bash -n`. - The platform comparison was run by hand against the live registry: `:edge` reports `linux/amd64,linux/arm64` and `v3.11.0` reports `single`, which is exactly the case the new step must fail on. - The real test is the dispatch on `v3.11.0` right after merge, which is also the repair. If the verification step fails there, nothing is published and the tags stay as they are. ## Not verified - The scratch-tag delete needs `delete:packages`; `GITHUB_TOKEN` may not have it. It cannot fail the run. - Whether GHCR keeps the attestation manifests attached to `:edge` reachable after the index is rebuilt for a release tag (they stay on `:edge` itself, which is untouched). --------- Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/release-fast-path.yml | 156 +++++++++++++++++++++--- test-issue-1956-release-routing.js | 92 +++++++++++++- 2 files changed, 230 insertions(+), 18 deletions(-) diff --git a/.github/workflows/release-fast-path.yml b/.github/workflows/release-fast-path.yml index e71b9f6a..02980411 100644 --- a/.github/workflows/release-fast-path.yml +++ b/.github/workflows/release-fast-path.yml @@ -8,10 +8,21 @@ name: Release Fast-Path # # This workflow is the SOLE consumer of push.tags. deploy.yml's tag # trigger has been removed to prevent double-fire. +# +# It can also be dispatched by hand with an existing tag, to republish the +# images for a release whose tags are wrong (v3.11.0 shipped an amd64-only +# manifest, see below). A manual run never dispatches deploy.yml: the release +# for that tag already exists and this repository has immutable releases. on: push: tags: ['v[0-9]+.[0-9]+.[0-9]+'] + workflow_dispatch: + inputs: + tag: + description: 'Existing vX.Y.Z tag to republish container tags for' + required: true + type: string permissions: contents: read @@ -19,7 +30,7 @@ permissions: actions: write # issue #1702: required for the `gh workflow run deploy.yml` dispatch concurrency: - group: release-fast-path-${{ github.ref }} + group: release-fast-path-${{ inputs.tag || github.ref }} cancel-in-progress: false jobs: @@ -39,22 +50,41 @@ jobs: - name: Parse semver from tag id: semver + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail - TAG="${GITHUB_REF#refs/tags/}" - # Expect vMAJOR.MINOR.PATCH (workflow trigger already enforces this). + if [[ -n "${{ inputs.tag || '' }}" ]]; then + TAG="${{ inputs.tag || '' }}" + else + TAG="${GITHUB_REF#refs/tags/}" + fi + # Expect vMAJOR.MINOR.PATCH (the push trigger already enforces this; + # a dispatched tag has to be checked here). if [[ ! "$TAG" =~ ^v([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then echo "Tag $TAG does not match vMAJOR.MINOR.PATCH" >&2 exit 1 fi MAJOR="${BASH_REMATCH[1]}" MINOR="${BASH_REMATCH[2]}" + # On a dispatched run github.sha is the ref the workflow file came + # from, not the tagged commit, so resolve the tag itself. + if [[ -n "${{ inputs.tag || '' }}" ]]; then + TARGET_SHA="$(gh api "repos/${{ github.repository }}/git/refs/tags/$TAG" --jq '.object.sha')" + OBJ_TYPE="$(gh api "repos/${{ github.repository }}/git/refs/tags/$TAG" --jq '.object.type')" + if [[ "$OBJ_TYPE" == "tag" ]]; then + TARGET_SHA="$(gh api "repos/${{ github.repository }}/git/tags/$TARGET_SHA" --jq '.object.sha')" + fi + else + TARGET_SHA="${{ github.sha }}" + fi { echo "tag=$TAG" echo "vMajor=v$MAJOR" echo "vMajorMinor=v$MAJOR.$MINOR" + echo "targetSha=$TARGET_SHA" } >> "$GITHUB_OUTPUT" - echo "Parsed: $TAG → v$MAJOR / v$MAJOR.$MINOR / $TAG" + echo "Parsed: $TAG → v$MAJOR / v$MAJOR.$MINOR / $TAG at $TARGET_SHA" - name: Inspect :edge revision label id: edge @@ -76,17 +106,22 @@ jobs: echo "edge_revision=$REV" >> "$GITHUB_OUTPUT" echo "no_edge=false" >> "$GITHUB_OUTPUT" echo ":edge org.opencontainers.image.revision = $REV" - echo "tag SHA (github.sha) = ${{ github.sha }}" + echo "tagged commit = ${{ steps.semver.outputs.targetSha }}" # ─────────── FAST PATH: SHAs match, metadata-only retag ─────────── # Issue #1807: a plain `crane tag` keeps the ldflags-baked # Version="edge" inside the binary, so /api/stats reports "edge" on # tagged releases. Instead of a full rebuild we append a single-file # layer (/app/.image-version containing the tag) and set the OCI - # version label, then point the remaining tags at that mutated image. - # The server reads .image-version at startup (see resolveVersion()). + # version label. The server reads .image-version at startup (see + # resolveVersion()). + # + # `crane mutate` works on ONE image, not on an index: pointed at a + # multi-arch tag it silently returns the default platform. v3.11.0 + # shipped amd64-only that way. So mutate each platform child of the + # :edge index and reassemble an index from the results. - name: Re-tag :edge → :vX.Y.Z + :vX.Y + :vX + :latest (fast path) - if: steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == github.sha + if: steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == steps.semver.outputs.targetSha run: | set -euo pipefail IMAGE="ghcr.io/kpa-clawbot/corescope" @@ -100,11 +135,47 @@ jobs: printf '%s' "$TAG" > "${LAYER_DIR}/app/.image-version" tar --owner=0 --group=0 --mtime='UTC 2020-01-01' \ -C "$LAYER_DIR" -cf /tmp/image-version-layer.tar app - echo " crane mutate $SRC (+.image-version layer, +version label) → ${IMAGE}:${TAG}" - crane mutate "$SRC" \ - --append /tmp/image-version-layer.tar \ - --label "org.opencontainers.image.version=${TAG}" \ - --tag "${IMAGE}:${TAG}" + + crane manifest "$SRC" > /tmp/edge-manifest.json + MEDIA="$(jq -r '.mediaType // ""' /tmp/edge-manifest.json)" + + if [[ "$MEDIA" == *".index."* || "$MEDIA" == *".manifest.list."* ]]; then + # Attestation manifests carry platform architecture "unknown" and + # are rebuilt by buildx, not by us; carrying them over would point + # them at the pre-mutation digests, so the release index holds the + # runnable platforms only. + mapfile -t PLATFORMS < <(jq -r ' + .manifests[] + | select((.platform.architecture // "unknown") != "unknown") + | "\(.platform.os)/\(.platform.architecture)\(if .platform.variant then "/" + .platform.variant else "" end)\t\(.digest)" + ' /tmp/edge-manifest.json) + if [[ "${#PLATFORMS[@]}" -eq 0 ]]; then + echo "No runnable platforms in $SRC" >&2 + exit 1 + fi + MANIFEST_ARGS=() + for entry in "${PLATFORMS[@]}"; do + PLATFORM="${entry%%$'\t'*}" + DIGEST="${entry##*$'\t'}" + SAFE="${PLATFORM//\//-}" + TMP_TAG="${IMAGE}:tmp-${TAG}-${SAFE}" + echo " crane mutate ${SRC}@${DIGEST} ($PLATFORM) → $TMP_TAG" + crane mutate "${IMAGE}@${DIGEST}" \ + --append /tmp/image-version-layer.tar \ + --label "org.opencontainers.image.version=${TAG}" \ + --tag "$TMP_TAG" + MANIFEST_ARGS+=(-m "$TMP_TAG") + done + echo " crane index append ${MANIFEST_ARGS[*]} → ${IMAGE}:${TAG}" + crane index append "${MANIFEST_ARGS[@]}" -t "${IMAGE}:${TAG}" + else + # Single-platform :edge (nothing to reassemble). + echo " crane mutate $SRC (single platform) → ${IMAGE}:${TAG}" + crane mutate "$SRC" \ + --append /tmp/image-version-layer.tar \ + --label "org.opencontainers.image.version=${TAG}" \ + --tag "${IMAGE}:${TAG}" + fi for NEW_TAG in \ "${{ steps.semver.outputs.vMajorMinor }}" \ @@ -115,14 +186,69 @@ jobs: done echo "Fast-path complete — release tags point at :edge plus a one-file version layer." - # Both image routes need the single release-artifacts writer in deploy.yml. + # A release tag that resolves to one platform is worse than a slow + # release: `docker pull` on the other architecture fails outright. + - name: Verify the published tags carry every :edge platform + if: steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == steps.semver.outputs.targetSha + run: | + set -euo pipefail + IMAGE="ghcr.io/kpa-clawbot/corescope" + TAG="${{ steps.semver.outputs.tag }}" + platforms_of() { + crane manifest "$1" | jq -r ' + if (.mediaType // "") | test("index|manifest.list") then + [ .manifests[] + | select((.platform.architecture // "unknown") != "unknown") + | "\(.platform.os)/\(.platform.architecture)\(if .platform.variant then "/" + .platform.variant else "" end)" ] + | sort | join(",") + else + "single" + end' + } + WANT="$(platforms_of "${IMAGE}:edge")" + echo " :edge platforms = $WANT" + for CHECK in \ + "$TAG" \ + "${{ steps.semver.outputs.vMajorMinor }}" \ + "${{ steps.semver.outputs.vMajor }}" \ + "latest"; do + GOT="$(platforms_of "${IMAGE}:${CHECK}")" + echo " :${CHECK} platforms = $GOT" + if [[ "$GOT" != "$WANT" ]]; then + echo "::error::${IMAGE}:${CHECK} has platforms [$GOT], expected [$WANT]" + exit 1 + fi + done + echo "All release tags carry the same platforms as :edge." + + - name: Remove the per-platform scratch tags + if: steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == steps.semver.outputs.targetSha + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -uo pipefail + IMAGE="ghcr.io/kpa-clawbot/corescope" + TAG="${{ steps.semver.outputs.tag }}" + # Best effort: the index already references the manifests by digest, + # so deleting the scratch tags does not affect it. Leaving them is + # only untidy, hence no failure here. This step is skipped when an + # earlier one failed, which keeps the scratch tags around to inspect. + for T in $(crane ls "$IMAGE" | grep "^tmp-${TAG}-" || true); do + echo " crane delete ${IMAGE}:${T}" + crane delete "${IMAGE}:${T}" || echo " (could not delete ${T}, leaving it)" + done + + # Both image routes need the single release-artifacts writer in + # deploy.yml. A manual republish skips it: the release exists already and + # releases here are immutable, so action-gh-release would fail. - name: Dispatch release artifacts or full deploy.yml pipeline + if: ${{ !inputs.tag }} env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | set -euo pipefail DISPATCH_ARGS=() - if [[ "${{ steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == github.sha }}" == 'true' ]]; then + if [[ "${{ steps.edge.outputs.no_edge == 'false' && steps.edge.outputs.edge_revision == steps.semver.outputs.targetSha }}" == 'true' ]]; then echo "Images published — dispatching Go validation and release artifacts" DISPATCH_ARGS+=(--field images_published=true) else diff --git a/test-issue-1956-release-routing.js b/test-issue-1956-release-routing.js index 97b20fa4..fc725883 100644 --- a/test-issue-1956-release-routing.js +++ b/test-issue-1956-release-routing.js @@ -68,17 +68,57 @@ function runSteps(source, context, edge, mutateFails = false) { if [ "$1" = config ]; then [ "$EDGE_CONFIG" != missing ] || return 1 printf '%s' "$EDGE_CONFIG" + elif [ "$1" = manifest ]; then + log crane "$@" + printf '%s' "$EDGE_MANIFEST" + elif [ "$1" = ls ]; then + log crane "$@" + printf 'edge\\ntmp-v9.8.7-linux-amd64\\ntmp-v9.8.7-linux-arm64\\n' else log crane "$@" [ "$1" != mutate ] || [ "$MUTATE_FAILS" != true ] fi } - gh() { log gh "$@"; } + # gh api answers the tag lookup a dispatched republish makes; everything + # else (workflow run) only gets logged. + gh() { + log gh "$@" + if [ "$1" = api ]; then + case "$*" in + *.object.type*) printf 'commit\\n' ;; + *.object.sha*) printf '%s\\n' "$GH_API_SHA" ;; + esac + fi + } tar() { log tar "$@"; } go() { log go "$GOOS" "$GOARCH" "$CGO_ENABLED" "$CC" "$@"; } file() { log file "$@"; echo "$1: ELF 64-bit LSB executable, statically linked"; } + # Stand-in for the jq filters the release steps use. Each branch mirrors one + # filter, so a filter that changes shape without the test knowing fails here + # instead of silently returning nothing. jq() { - node -e 'const fs=require("fs"); const assert=require("assert/strict"); assert.equal(process.argv[1], ".config.Labels[\\"org.opencontainers.image.revision\\"] // \\\"\\\""); console.log(JSON.parse(fs.readFileSync(0,"utf8")).config.Labels["org.opencontainers.image.revision"] || "")' "$2" + node -e ' + const fs = require("fs"); + const filter = process.argv[1]; + const source = process.argv[2]; + const text = source && fs.existsSync(source) ? fs.readFileSync(source, "utf8") : fs.readFileSync(0, "utf8"); + const data = text.trim() ? JSON.parse(text) : {}; + const platform = m => [m.platform.os, m.platform.architecture].join("/") + (m.platform.variant ? "/" + m.platform.variant : ""); + const runnable = () => (data.manifests || []).filter(m => (m.platform && m.platform.architecture || "unknown") !== "unknown"); + if (filter.includes("org.opencontainers.image.revision")) { + console.log((((data.config || {}).Labels || {})["org.opencontainers.image.revision"]) || ""); + } else if (filter.trim() === ".mediaType // \\"\\"") { + console.log(data.mediaType || ""); + } else if (filter.includes("\\\\t\\\\(.digest)")) { + for (const m of runnable()) console.log(platform(m) + "\\t" + m.digest); + } else if (filter.includes("index|manifest.list")) { + const isIndex = /index|manifest.list/.test(data.mediaType || ""); + console.log(isIndex ? runnable().map(platform).sort().join(",") : "single"); + } else { + console.error("unstubbed jq filter: " + filter); + process.exit(3); + } + ' "$2" "\${3:-}" } `; try { @@ -92,6 +132,18 @@ function runSteps(source, context, edge, mutateFails = false) { ...process.env, GITHUB_REF: context.github.ref, GITHUB_SHA: context.github.sha, GITHUB_OUTPUT: bashPath(output), COMMAND_LOG: bashPath(log), TMPDIR: bashPath(dir), EDGE_CONFIG: edge === null ? 'missing' : JSON.stringify({ config: { Labels: { 'org.opencontainers.image.revision': edge } } }), + // :edge is a two-platform index plus the two buildx attestation + // manifests, which is what the registry actually holds. + EDGE_MANIFEST: JSON.stringify({ + mediaType: 'application/vnd.oci.image.index.v1+json', + manifests: [ + { digest: 'sha256:' + '1'.repeat(64), platform: { os: 'linux', architecture: 'amd64' } }, + { digest: 'sha256:' + '2'.repeat(64), platform: { os: 'linux', architecture: 'arm64' } }, + { digest: 'sha256:' + '3'.repeat(64), platform: { os: 'unknown', architecture: 'unknown' } }, + { digest: 'sha256:' + '4'.repeat(64), platform: { os: 'unknown', architecture: 'unknown' } } + ] + }), + GH_API_SHA: edge === null ? 'a'.repeat(40) : edge, MUTATE_FAILS: String(mutateFails) } }); @@ -139,7 +191,22 @@ for (const [name, edge] of [['matching', 'a'.repeat(40)], ['missing', null], ['m const matching = name === 'matching'; assert.equal(dispatch.includes('images_published=true'), matching); if (!matching) assert.ok(!dispatch.includes('--field') && !dispatch.includes('-f'), 'old-tag fallback must not require new workflow inputs'); - assert.equal(commands.filter(command => command[0] === 'crane' && command[1] === 'mutate').length, matching ? 1 : 0); + // One mutate per runnable platform of the :edge index, never one for the + // index itself: `crane mutate` on an index silently drops to one platform, + // which is how v3.11.0 shipped amd64-only. + const mutates = commands.filter(command => command[0] === 'crane' && command[1] === 'mutate'); + assert.equal(mutates.length, matching ? 2 : 0, `${name}: one mutate per platform`); + if (matching) { + assert.deepEqual(mutates.map(command => command.at(-1)).sort(), + ['ghcr.io/kpa-clawbot/corescope:tmp-v9.8.7-linux-amd64', 'ghcr.io/kpa-clawbot/corescope:tmp-v9.8.7-linux-arm64'], + 'each platform is mutated into its own scratch tag'); + assert.ok(mutates.every(command => command[2].includes('@sha256:')), 'mutate must address a platform by digest, not the index tag'); + const indexes = commands.filter(command => command[0] === 'crane' && command[1] === 'index'); + assert.equal(indexes.length, 1, 'the release tag is assembled as one index'); + assert.deepEqual(indexes[0].slice(1, 3), ['index', 'append']); + assert.equal(indexes[0].filter(argument => argument === '-m').length, 2, 'the index carries both platforms'); + assert.equal(indexes[0].at(-1), 'ghcr.io/kpa-clawbot/corescope:v9.8.7'); + } assert.deepEqual(commands.filter(command => command[0] === 'crane' && command[1] === 'tag').map(command => command.at(-1)), matching ? ['v9.8', 'v9', 'latest'] : []); const jobs = route(context(undefined, undefined, { images_published: matching })); assert.equal(jobs['release-artifacts'].result, 'success', `${name}: release artifacts must run`); @@ -200,3 +267,22 @@ const checkout = steps(release).find(step => step.includes('uses: actions/checko assert.equal(value(checkout, 'ref', 10), '', 'checkout must retain the dispatched tag/SHA'); assert.ok(!value(block(deploy, 'push', 2), 'tags', 4), 'fast path must remain the sole tag-triggered image writer'); console.log('PASS failed retag/Go gates, branch/PR routes, and complete tagged release assets'); + +// Republishing the images for a tag that already has a release: the tag comes +// from the dispatch input, the commit is resolved from the tag itself (the +// workflow file's own ref is master there), and deploy.yml must NOT be +// dispatched again — the release exists and releases here are immutable. +{ + const sha = 'c'.repeat(40); + const ctx = context('refs/heads/master', 'workflow_dispatch', { tag: 'v9.8.7' }); + const { commands } = runSteps(block(fast, 'retag-or-fallback', 2), ctx, sha); + assert.equal(ctx.steps.semver.outputs.tag, 'v9.8.7', 'the dispatched tag drives the release tags'); + assert.equal(ctx.steps.semver.outputs.targetSha, sha, 'the tagged commit is resolved from the tag, not from github.sha'); + assert.ok(commands.some(command => command[0] === 'gh' && command[1] === 'api'), 'the tag has to be looked up'); + assert.equal(commands.filter(command => command[0] === 'gh' && command[1] === 'workflow').length, 0, + 'a republish must not dispatch deploy.yml into an immutable release'); + assert.equal(commands.filter(command => command[0] === 'crane' && command[1] === 'mutate').length, 2, 'still one mutate per platform'); + assert.equal(commands.filter(command => command[0] === 'crane' && command[1] === 'index').length, 1, 'still one index'); + assert.deepEqual(commands.filter(command => command[0] === 'crane' && command[1] === 'tag').map(command => command.at(-1)), ['v9.8', 'v9', 'latest']); + console.log('PASS dispatched republish: tags rebuilt, no second release dispatch'); +}