ukmesh/docs/runbook-release-rollback.md

Release and rollback runbook

Production application containers are released by immutable registry digest. Do not deploy a mutable tag and do not substitute docker compose build for the signed release path.

Preconditions

• The release workflow has built the exact Git revision, scanned it for critical vulnerabilities, emitted an SPDX SBOM and provenance, and signed every image digest.
• Keyless Cosign trust identifies the exact GitHub release workflow and GitHub Actions OIDC issuer. A separately managed COSIGN_PUBLIC_KEY is also supported when releases intentionally use key-pair signing.
• A complete encrypted backup has a signed receipt.
• scripts/restore-drill.sh has produced a signed, verified restore receipt less than seven days old.
• The tracked checkout is clean and is exactly SOURCE_REVISION.
• Current service health is green and the prior signed digest is recorded.

Verify the evidence before changing a service:

git rev-parse HEAD
git status --short
openssl dgst -sha256 -verify "$RESTORE_RECEIPT_VERIFY_KEY" \
  -signature "${RESTORE_RECEIPT_PATH}.sig" "$RESTORE_RECEIPT_PATH"
jq '{status,source_revision,restore_verified_at,checks}' "$RESTORE_RECEIPT_PATH"
cosign verify "$BACKEND_IMAGE" \
  --certificate-identity-regexp "$COSIGN_CERTIFICATE_IDENTITY_REGEXP" \
  --certificate-oidc-issuer "$COSIGN_CERTIFICATE_OIDC_ISSUER"
cosign verify "$TARGET_IMAGE" \
  --certificate-identity-regexp "$COSIGN_CERTIFICATE_IDENTITY_REGEXP" \
  --certificate-oidc-issuer "$COSIGN_CERTIFICATE_OIDC_ISSUER"
docker compose config -q

Release one service

Set immutable values and invoke the controlled replacement:

export COSIGN_CERTIFICATE_IDENTITY_REGEXP='^https://github.com/OWNER/REPOSITORY/.github/workflows/release.yml@'
export COSIGN_CERTIFICATE_OIDC_ISSUER='https://token.actions.githubusercontent.com'
export RESTORE_RECEIPT_PATH=/secure/receipts/latest.json
export RESTORE_RECEIPT_VERIFY_KEY=/secure/receipts/verify.pem
export RELEASE_RECEIPT_SIGNING_KEY=/secure/release/receipt-signing.pem

scripts/replace-container.sh backend \
  --image="$BACKEND_IMAGE" \
  --backend-image="$BACKEND_IMAGE" \
  --source-revision="$SOURCE_REVISION"

Repeat for each application service, passing its digest as --image and the same signed backend digest as --backend-image. The script verifies signatures and revision labels, runs migrations, starts the prior backend against the post-migration schema, replaces only the selected service, checks health, readiness and metrics, and writes a release status receipt.

An existing mutable local deployment may enter this regime once by supplying the exact prompt printed by the script:

--bootstrap-immutable=bootstrap-SERVICE-SOURCE_REVISION

This is not a general signature bypass. Record the approval and do not use it again after the first signed digest is live.

Stop and rollback conditions

Stop immediately for a privacy regression, cross-network response, migration compatibility failure, data loss, failed readiness, missing metrics, or a service health failure. replace-container.sh automatically rolls back only when the prior image is itself a verified signed digest.

For a manual rollback, use the prior digests from the release receipt:

jq '{service,prior_image,prior_backend_image,schema_version,status}' \
  /home/ben/meshcore-releases/RELEASE_RECEIPT.json

export BACKEND_IMAGE='REGISTRY/backend@sha256:PRIOR_DIGEST'
docker compose pull backend
docker compose up -d --no-build --no-deps backend
curl --fail http://127.0.0.1:3000/readyz

Additive migrations are deliberately compatible with the prior backend. Never attempt a destructive down-migration. If database contents must roll back, stop MQTT ingest and all writers, restore the signed pre-release backup in an isolated environment first, then perform a separately approved disaster recovery cutover.

Post-release verification

docker compose ps
curl --fail http://127.0.0.1:3000/readyz | jq .
curl --fail 'http://127.0.0.1:3000/api/stats?network=ukmesh' | jq .
docker compose exec -T backend wget -qO- http://127.0.0.1:9091/metrics \
  | grep '^meshcore_process_'
docker compose logs --since=10m backend

Confirm the public app, a WebSocket initial state, the owner login page, the local operator page, Prometheus targets, and Alertmanager are healthy. Keep the prior digest until the observation window ends.