ci: render iOS previews in the simulator, compare with references (#7905)

* ci: render iOS previews in the simulator, compare with references

* docs: assets license covers app screenshots and test snapshot images

* ci: references of the iOS preview snapshots

scripts/ci/ios-sim/README.md

@@ -6,14 +6,14 @@
 
 The workflow runs only when started by hand, on any branch, from the Actions tab or with `gh workflow run`: [CI on GitHub's runners](../README.md) says how and lists the inputs.
 
-Results: the job logs, the Claude job's summary (Claude's usage) and the artifacts `ios-sim-smoke-test` (a screenshot, a 1000-px view and an accessibility tree per step, the app log), `ios-sim-backend-logs` and `ios-sim-claude-test` (the Claude job's smoke test in `smoke-test/`, and per session: `prompt.md`, `report.md`, `shots/`, `transcript.jsonl`, `usage.json`). Read them with `gh run view <id> --log` and `gh run download <id> -n <artifact>`.
+Results: the job logs, the Claude job's summary (Claude's usage) and the artifacts `ios-sim-smoke-test` (a screenshot, a 1000-px view and an accessibility tree per step, the app log, the snapshots in `snapshots/`), `ios-sim-backend-logs` and `ios-sim-claude-test` (the Claude job's smoke test in `smoke-test/`, and per session: `prompt.md`, `report.md`, `shots/`, `transcript.jsonl`, `usage.json`). Read them with `gh run view <id> --log` and `gh run download <id> -n <artifact>`.
 
 ## How long it takes, what it costs
 
 - `plan` (ubuntu-24.04, 10 s) reads what the run asks for, names the core library and the backend after their sources, `hashFiles` over the Haskell sources, `cabal.project`, `simplex-chat.cabal` and the build script, and reuses an artifact of the same name that an earlier run of the same branch uploaded (30 days). So every run builds the core library, and the backend when asked, from the branch's own Haskell code, and rebuilds only when those sources change.
 - `core` (macOS, 30–45 min when not reused): `build-core.sh` builds `lib:simplex-chat` with cabal and the iOS flags, as the Nix iOS build does, into the package `scripts/ios/prepare.sh` expects, and `prepare-libs.sh` marks it for the simulator (`mac2ios`).
 - `backend` (macOS, 20–45 min when not reused, only when asked): `build-backend.sh` builds `simplex-chat`, `simplex-directory-service` and `smp-server`.
-- `app` (macOS): the Xcode build 8–22 min, the smoke test about 10 min, the backend start about 10 min when asked.
+- `app` (macOS): the Xcode build 8–22 min, the smoke test about 10 min with the snapshots a few minutes more, the backend start about 10 min when asked.
 - `claude` (macOS, only when asked, after `app` passed): a new simulator with the app job's build and the smoke test, about 7 min, then the sessions (the smoke plan's took 2.6 min).
 
 A run with reused builds and no extras takes about 25 minutes plus the wait for a macOS runner. Hosted minutes are free for this public repository; the limit is the repository's five concurrent macOS jobs, shared with PR builds (each `build.yml` run takes two), so queue waits can be long. The workflow saves no Actions caches, which this repository evicts within hours and shares with `build.yml`: builds are reused only as the artifacts above, and a core or backend rebuild compiles its dependencies too.
@@ -38,6 +38,20 @@ Haiku 5.5 drives the app well: the smoke plan takes a few dozen turns and cents;
 
 `ui` wraps AXe (`install-axe.sh`) for scripts and for Claude: tap by label, value or point, swipe, type, paste, keys, wait for text, find, screenshots with trees, switch the layout (one-hand UI or top bar), relaunch, text size, and the backend's directory (`ui` with no arguments lists them). It needs `SIM_UDID` and `UI_OUT`, which the smoke test exports to the job. Lessons kept in its comments and in the smoke test: text fields have no label (tap them by value), the keyboard capitalizes and autocorrects (compare typed text case-insensitively), the closed user picker is always in the tree, "Cancel" clears the search field and its "Close" image doesn't, the keyboard's keys are below the screen.
 
+## Snapshots: `snapshots.sh`
+
+At the end of the app job's smoke test, `smoke-test.sh` runs `snapshots.sh <derived data> <smoke test dir>/snapshots` when the app it tested lies in the derived data of a build (`…/Build/Products/Debug-iphonesimulator`); the Claude job, which installs the app from a tarball, skips it.
+
+`snapshots.sh` builds the unit-test target `SimpleX Snapshots` (`apps/ios/SimpleX Snapshots/SnapshotTests.swift`, hosted by the app) with `xcodebuild build-for-testing`, then runs it in the smoke test's simulator three times, after `xcrun simctl ui <udid> content_size` `large`, `extra-extra-extra-large` and `accessibility-medium`, and sets `large` back. The test hosts each preview listed in `previewScenes()` (the `previews` of a `PreviewProvider`) in a `UIHostingController` in a window of the screen's width, in the light and dark themes without a wallpaper, and draws it with `drawHierarchy` at scale 1, one pixel per point.
+
+The test runs in UTC (`NSTimeZone.default`) and en_US (`-testLanguage en -testRegion US`), so the renders do not depend on the runner's time zone or locale; a preview that shows the current date or time differs between runs.
+
+The images go to `snapshots/` of the artifact `ios-sim-smoke-test` as 8-bit sRGB PNG files without alpha (`preferredRange = .standard`, `opaque`), named `<preview>-<light|dark>-<size>.png`, with `index-<size>.json` listing each file's scene, theme, size, pixel width and height, scale and the content size category the app saw.
+
+When `apps/ios/SimpleX Snapshots/References` exists, `snapshots.sh` then runs `scripts/ci/snapshots-compare.py <references> <renders> <diff dir>`. It compares each render listed in `index-*.json` with the reference of the same name: a render passes when its file is identical to the reference, or when no channel of any pixel differs from it by more than 8 of 255. A missing or extra render, a changed size or a pixel beyond the tolerance fails: the script writes `compare.json` (each file's result and the failed files) next to the renders and, per failed file, `snapshots-diff/<file>` with the reference, the render and the changed pixels in red side by side, lists the failed files in the job summary and exits with 1, which fails the smoke test step.
+
+`snapshots-remote.py record|check [branch]` renders the snapshots of a pushed branch (the current one by default) on GitHub's runner: it starts `ios-sim.yml` on the branch, waits for the run and downloads `snapshots/` and `snapshots-diff/` of its `ios-sim-smoke-test` to a temporary folder. `record` replaces the PNG files in `apps/ios/SimpleX Snapshots/References` with the run's renders and prints the folder's `git status`, for review and commit; `check` prints the files that failed the run's comparison and the folder of their diff images, and exits with 1 when any failed. It takes the token from `GITHUB_TOKEN` or `GH_TOKEN`, otherwise from `gh auth token`; the token must be allowed to start workflows.
+
 ## The test backend
 
 `backend.sh start <backend zip> <work dir>` (with the `backend` input) starts, on the runner, against the preset SMP and XFTP servers:

scripts/ci/ios-sim/smoke-test.sh

@@ -2,6 +2,8 @@
 
 # Installs the app in a new iPhone simulator and runs the scripted UI cases with the ui helper: onboarding to the
 # chat list with the one-hand UI (the default), then the chat list again with the top bar. It leaves the app in the one-hand UI.
+# For an app in the derived data of a build, it then renders the previews of the SimpleX Snapshots tests to the snapshots
+# folder of the output dir (snapshots.sh).
 # Usage: scripts/ci/ios-sim/smoke-test.sh <path to SimpleX.app> <output dir>
 # Each step saves a screenshot and an accessibility tree to the output dir and prints the tree to the log.
 # Later steps of the job get SIM_UDID and UI_OUT, with the simulator still running.
@@ -122,3 +124,8 @@ expect_chat_list reachable
 expect_chat_list top
 "$ui" layout reachable
 echo "smoke test passed: onboarding reaches the chat list, which shows its onboarding pages and toolbar with the one-hand UI and with the top bar"
+
+products=$(dirname "$app")
+if [[ "$products" == */Build/Products/Debug-iphonesimulator ]]; then
+  "$(dirname "$0")/snapshots.sh" "${products%/Build/Products/Debug-iphonesimulator}" "$UI_OUT/snapshots"
+fi

scripts/ci/ios-sim/snapshots-remote.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+
+import io
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+import zipfile
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+WORKFLOW = "ios-sim.yml"
+ARTIFACT = "ios-sim-smoke-test"
+REFERENCES = "apps/ios/SimpleX Snapshots/References"
+
+
+def git(*args):
+    return subprocess.run(["git", *args], check=True, capture_output=True, text=True).stdout.strip()
+
+
+def github_token():
+    token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN")
+    if token:
+        return token
+    try:
+        return subprocess.run(["gh", "auth", "token"], check=True, capture_output=True, text=True).stdout.strip()
+    except (OSError, subprocess.CalledProcessError):
+        sys.exit("set GITHUB_TOKEN, or log in with gh")
+
+
+class NoRedirect(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, *args):
+        return None
+
+
+class GitHub:
+    def __init__(self):
+        self.repo = git("remote", "get-url", "origin").split("github.com", 1)[1].lstrip(":/").removesuffix(".git")
+        self.token = github_token()
+
+    def request(self, url, body=None):
+        return urllib.request.Request(
+            url if url.startswith("https://") else f"https://api.github.com/repos/{self.repo}{url}",
+            data=None if body is None else json.dumps(body).encode(),
+            headers={"Authorization": f"Bearer {self.token}", "Accept": "application/vnd.github+json"},
+        )
+
+    def get(self, path):
+        with urllib.request.urlopen(self.request(path)) as r:
+            return json.loads(r.read())
+
+    def post(self, path, body):
+        urllib.request.urlopen(self.request(path, body)).close()
+
+    def download(self, url):
+        try:
+            urllib.request.build_opener(NoRedirect).open(self.request(url)).close()
+            sys.exit(f"no redirect from {url}")
+        except urllib.error.HTTPError as e:
+            if e.code not in (301, 302, 303, 307, 308):
+                raise
+            location = e.headers["Location"]
+        with urllib.request.urlopen(location) as r:
+            return r.read()
+
+
+def run_workflow(github, branch):
+    head = git("rev-parse", "HEAD")
+    quoted = urllib.parse.quote(branch, safe="")
+    remote_head = github.get(f"/branches/{quoted}")["commit"]["sha"]
+    if remote_head != head:
+        sys.exit(f"{branch} is at {remote_head[:9]} on GitHub and at {head[:9]} here: push it first")
+    dispatched = datetime.now(timezone.utc) - timedelta(minutes=1)
+    github.post(f"/actions/workflows/{WORKFLOW}/dispatches", {"ref": branch})
+    run = None
+    while run is None:
+        time.sleep(15)
+        runs = github.get(f"/actions/workflows/{WORKFLOW}/runs?branch={quoted}&event=workflow_dispatch&per_page=10")["workflow_runs"]
+        run = next((r for r in runs if r["head_sha"] == head and datetime.fromisoformat(r["created_at"].replace("Z", "+00:00")) >= dispatched), None)
+    print(f"run {run['html_url']}", flush=True)
+    while run["status"] != "completed":
+        time.sleep(60)
+        run = github.get(f"/actions/runs/{run['id']}")
+    print(f"run {run['id']}: {run['conclusion']}")
+    return run
+
+
+def download_snapshots(github, run):
+    artifacts = github.get(f"/actions/runs/{run['id']}/artifacts?name={ARTIFACT}")["artifacts"]
+    if not artifacts:
+        sys.exit(f"run {run['id']} has no artifact {ARTIFACT}")
+    target = Path(tempfile.mkdtemp(prefix="ios-snapshots-"))
+    with zipfile.ZipFile(io.BytesIO(github.download(artifacts[0]["archive_download_url"]))) as z:
+        for name in z.namelist():
+            if name.startswith(("snapshots/", "snapshots-diff/")):
+                z.extract(name, target)
+    return target
+
+
+def record(downloaded):
+    renders = downloaded / "snapshots"
+    files = sorted({e["file"] for index in renders.glob("index-*.json") for e in json.loads(index.read_text())})
+    if not files:
+        sys.exit(f"no snapshots in {renders}")
+    references = Path(git("rev-parse", "--show-toplevel")) / REFERENCES
+    references.mkdir(parents=True, exist_ok=True)
+    for old in references.glob("*.png"):
+        if old.name not in files:
+            old.unlink()
+    for name in files:
+        shutil.copyfile(renders / name, references / name)
+    size = sum((references / name).stat().st_size for name in files)
+    print(f"{len(files)} references, {size / 1e6:.1f} MB, in {references}")
+    print(git("status", "--short", "--", str(references)))
+
+
+def check(downloaded):
+    result = downloaded / "snapshots" / "compare.json"
+    if not result.exists():
+        sys.exit(f"the run compared no snapshots; its renders are in {downloaded / 'snapshots'}")
+    compared = json.loads(result.read_text())
+    for name in compared["failed"]:
+        print(f"{name}: {compared['results'][name]}")
+    print(f"{len(compared['results'])} snapshots, {len(compared['failed'])} differ; diff images in {downloaded / 'snapshots-diff'}")
+    sys.exit(1 if compared["failed"] else 0)
+
+
+def main():
+    if len(sys.argv) not in (2, 3) or sys.argv[1] not in ("record", "check"):
+        sys.exit("usage: snapshots-remote.py record|check [branch]")
+    branch = sys.argv[2] if len(sys.argv) == 3 else git("rev-parse", "--abbrev-ref", "HEAD")
+    github = GitHub()
+    downloaded = download_snapshots(github, run_workflow(github, branch))
+    if sys.argv[1] == "record":
+        record(downloaded)
+    else:
+        check(downloaded)
+
+
+main()

scripts/ci/ios-sim/snapshots.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+set -euo pipefail
+
+derived=$1
+out=$2
+mkdir -p "$out"
+
+xcode=(
+  -project apps/ios/SimpleX.xcodeproj
+  -scheme "SimpleX (iOS)"
+  -configuration Debug
+  -destination "id=$SIM_UDID"
+  -derivedDataPath "$derived"
+  -onlyUsePackageVersionsFromResolvedFile
+  -skipPackagePluginValidation
+  -skipMacroValidation
+  -only-testing:"SimpleX Snapshots"
+  ARCHS=arm64
+  CODE_SIGN_STYLE=Manual
+  CODE_SIGN_IDENTITY=-
+  DEVELOPMENT_TEAM=
+  PROVISIONING_PROFILE_SPECIFIER=
+)
+
+xcodebuild build-for-testing -quiet "${xcode[@]}"
+
+for size in large extra-extra-extra-large accessibility-medium; do
+  xcrun simctl ui "$SIM_UDID" content_size "$size"
+  TEST_RUNNER_SNAPSHOT_DIR="$out" TEST_RUNNER_SNAPSHOT_SIZE="$size" xcodebuild test-without-building -testLanguage en -testRegion US "${xcode[@]}"
+done
+xcrun simctl ui "$SIM_UDID" content_size large
+
+ls -l "$out"
+
+references="apps/ios/SimpleX Snapshots/References"
+if [ -d "$references" ]; then
+  python3 "$(dirname "$0")/../snapshots-compare.py" "$references" "$out" "$(dirname "$out")/snapshots-diff"
+fi

scripts/ci/snapshots-compare.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+
+import json
+import os
+import struct
+import sys
+import zlib
+from pathlib import Path
+
+CHANNEL_TOLERANCE = 8
+
+
+def unfilter(kind, line, prev, bpp):
+    if kind == 1:
+        for i in range(bpp, len(line)):
+            line[i] = (line[i] + line[i - bpp]) & 0xFF
+    elif kind == 2:
+        for i in range(len(line)):
+            line[i] = (line[i] + prev[i]) & 0xFF
+    elif kind == 3:
+        for i in range(len(line)):
+            left = line[i - bpp] if i >= bpp else 0
+            line[i] = (line[i] + ((left + prev[i]) >> 1)) & 0xFF
+    elif kind == 4:
+        for i in range(len(line)):
+            a = line[i - bpp] if i >= bpp else 0
+            b = prev[i]
+            c = prev[i - bpp] if i >= bpp else 0
+            p = a + b - c
+            pa, pb, pc = abs(p - a), abs(p - b), abs(p - c)
+            line[i] = (line[i] + (a if pa <= pb and pa <= pc else b if pb <= pc else c)) & 0xFF
+
+
+def read_rgb(path):
+    data = path.read_bytes()
+    pos, idat = 8, []
+    while pos < len(data):
+        length, kind = struct.unpack(">I4s", data[pos:pos + 8])
+        body = data[pos + 8:pos + 8 + length]
+        pos += 12 + length
+        if kind == b"IHDR":
+            width, height, depth, color, _, _, interlace = struct.unpack(">IIBBBBB", body)
+        elif kind == b"IDAT":
+            idat.append(body)
+    if depth not in (8, 16) or color not in (2, 6) or interlace:
+        raise ValueError(f"{path.name}: bit depth {depth}, colour type {color}, interlace {interlace}")
+    channels = 3 if color == 2 else 4
+    bpp = channels * depth // 8
+    stride = width * bpp
+    raw = zlib.decompress(b"".join(idat))
+    rows, prev = [], bytearray(stride)
+    for y in range(height):
+        start = y * (stride + 1)
+        line = bytearray(raw[start + 1:start + 1 + stride])
+        unfilter(raw[start], line, prev, bpp)
+        prev = line
+        high = line[0::2] if depth == 16 else line
+        rgb = bytearray(width * 3)
+        for c in range(3):
+            rgb[c::3] = high[c::channels]
+        rows.append(bytes(rgb))
+    return width, height, rows
+
+
+def write_rgb(path, width, rows):
+    def chunk(kind, body):
+        return struct.pack(">I", len(body)) + kind + body + struct.pack(">I", zlib.crc32(kind + body))
+    raw = b"".join(b"\0" + row for row in rows)
+    path.write_bytes(
+        b"\x89PNG\r\n\x1a\n"
+        + chunk(b"IHDR", struct.pack(">IIBBBBB", width, len(rows), 8, 2, 0, 0, 0))
+        + chunk(b"IDAT", zlib.compress(raw, 6))
+        + chunk(b"IEND", b"")
+    )
+
+
+def changed_pixels(ref_rows, out_rows, width):
+    changed, max_delta = {}, 0
+    for y, (a, b) in enumerate(zip(ref_rows, out_rows)):
+        if a == b:
+            continue
+        for x in range(width):
+            i = x * 3
+            delta = max(abs(a[i] - b[i]), abs(a[i + 1] - b[i + 1]), abs(a[i + 2] - b[i + 2]))
+            max_delta = max(max_delta, delta)
+            if delta > CHANNEL_TOLERANCE:
+                changed.setdefault(y, []).append(x)
+    return changed, max_delta
+
+
+def diff_image(path, ref, out, changed):
+    (rw, rh, ref_rows), (ow, oh, out_rows) = ref, out
+    height = max(rh, oh)
+    grey = bytes([128]) * 3
+    rows = []
+    for y in range(height):
+        a = ref_rows[y] if y < rh else grey * rw
+        b = out_rows[y] if y < oh else grey * ow
+        row = a + grey * 8 + b
+        if (rw, rh) == (ow, oh):
+            mask = bytearray(255 - (255 - v) // 4 for v in a)
+            for x in changed.get(y, []):
+                mask[x * 3:x * 3 + 3] = b"\xff\0\0"
+            row += grey * 8 + mask
+        rows.append(bytes(row))
+    write_rgb(path, len(rows[0]) // 3, rows)
+
+
+def compare(ref_path, out_path, diff_dir):
+    if ref_path.read_bytes() == out_path.read_bytes():
+        return {"status": "equal"}
+    ref, out = read_rgb(ref_path), read_rgb(out_path)
+    if ref[:2] != out[:2]:
+        diff_image(diff_dir / out_path.name, ref, out, {})
+        return {"status": "size", "reference": list(ref[:2]), "render": list(out[:2])}
+    changed, max_delta = changed_pixels(ref[2], out[2], ref[0])
+    if not changed:
+        return {"status": "within tolerance", "maxDelta": max_delta}
+    diff_image(diff_dir / out_path.name, ref, out, changed)
+    return {"status": "changed", "changedPixels": sum(len(xs) for xs in changed.values()), "maxDelta": max_delta}
+
+
+def main():
+    if len(sys.argv) != 4:
+        sys.exit("usage: snapshots-compare.py <references dir> <renders dir> <diff dir>")
+    references, renders, diff_dir = map(Path, sys.argv[1:])
+    diff_dir.mkdir(parents=True, exist_ok=True)
+    rendered = {e["file"] for index in sorted(renders.glob("index-*.json")) for e in json.loads(index.read_text())}
+    expected = {p.name for p in references.glob("*.png")}
+    results = {}
+    for name in sorted(expected | rendered):
+        if name not in rendered:
+            results[name] = {"status": "missing"}
+        elif name not in expected:
+            results[name] = {"status": "extra"}
+        else:
+            results[name] = compare(references / name, renders / name, diff_dir)
+    failed = {n: r for n, r in results.items() if r["status"] not in ("equal", "within tolerance")}
+    (renders / "compare.json").write_text(json.dumps({"results": results, "failed": sorted(failed)}, indent=2, sort_keys=True))
+    lines = [f"{len(results)} snapshots, {len(failed)} differ from the references in {references}"]
+    lines += [f"- {n}: {', '.join(f'{k} {v}' for k, v in sorted(r.items()))}" for n, r in failed.items()]
+    print("\n".join(lines))
+    summary = os.environ.get("GITHUB_STEP_SUMMARY")
+    if summary:
+        with open(summary, "a") as f:
+            f.write("\n".join(["## Snapshots", ""] + lines) + "\n")
+    sys.exit(1 if failed else 0)
+
+
+main()