diff --git a/scripts/ci/desktop-linux/README.md b/scripts/ci/desktop-linux/README.md index 8bb40683d3..ad47c7cb5f 100644 --- a/scripts/ci/desktop-linux/README.md +++ b/scripts/ci/desktop-linux/README.md @@ -1,6 +1,6 @@ # Desktop app CI on Linux -`.github/workflows/desktop-linux.yml` builds the desktop app from the branch on a GitHub runner (Ubuntu 24.04), runs its JVM tests (`:common:desktopTest`), and drives the app on a virtual display: onboarding to the chat list, then the chat list with the toolbar at the bottom (the default) and at the top. On request it also runs Claude Code driving the same path, a test backend, and the latest release as a baseline. With the test backend, both the scripted test and Claude also search the test directory and a SimpleX name in both layouts. +`.github/workflows/desktop-linux.yml` builds the desktop app from the branch on a GitHub runner (Ubuntu 24.04), runs its JVM tests (`:common:desktopTest`), and drives the app on a virtual display: onboarding to the chat list, then the chat list with the toolbar at the bottom (the default) and at the top. On request it also runs Claude Code driving the same path, a test backend, and the latest release as a baseline. With the test backend, both the scripted test and Claude also search the test directory in both layouts, including with a SimpleX name typed, which only "Connect to …" looks up. ## Running it @@ -24,7 +24,7 @@ Queueing for a runner adds from under a minute to 20 minutes. Results are artifa ## Scripts - `ui`: the UI driver. Compose for Desktop has no accessibility tree on Linux, so it finds text on screenshots by OCR (tesseract) and acts with xdotool on an Xvfb display. `ui` alone prints its commands: start (with app settings), shot, text, find, wait, click, right-click, click-text, scroll, type, key, avatar (a profile image's colour), ink (a text's height), do (several in a row), stop. `screen-text.py`, `screen-colour.py` and `screen-ink.py` are its helpers. -- `smoke-test.sh [start|onboarding|setup|backend|search]`: the scripted cases; the default is onboarding and both layouts. `setup` also creates a local group and makes the test SMP server the app's only names server; `backend` does that, connects to the seeded name, and checks both layouts; `search`, which the workflow runs when the backend started, also searches in both layouts: "Search in Directory" for "ci seed", and the full search for the name `seeded` with Enter (the local chat, the Names part, the Directory part, in order), and with the toolbar at the bottom the full search in the New chat sheet, checking headers, rows, avatars, counts and the cards hidden while results show. It then joins `ci_channel` from its Directory row and checks, in both layouts, the note "All results on this page are already in your chats" when the typed `ci_channel` finds only the listed chat, and, with "seeded notes" open and `ci seed` typed, `ci_channel` as its chat's row in the Directory part (OCR reads no `#` before its name and no "Join channel" next to it; `#ci_group` keeps "Join group") that opens the channel without an alert; and the same row in the New chat sheet. The search case makes 129 checks (106 before the known-chat ones). +- `smoke-test.sh [start|onboarding|setup|backend|search]`: the scripted cases; the default is onboarding and both layouts. `setup` also creates a local group and makes the test SMP server the app's only names server; `backend` does that, connects to the seeded name, and checks both layouts; `search`, which the workflow runs when the backend started, also searches in both layouts: "Search in Directory" for "ci seed", which with the toolbar at the bottom first asks "Enable Directory search?" (Cancel searches nothing and keeps "Search in Directory", then Enable searches; no later search asks), and the search for the name `seeded` with Enter, which shows the local chat and the directory results with no "Names" header and no `ci_contact` row, keeps "Connect to seeded.testing", and orders "Connect to …", the local chat, then the results from the field; and with the toolbar at the bottom the same search in the New chat sheet. It checks one list without a "Directory" header, "Open" in each result's row, avatars, counts, the load row ("Load more" and "1 left" after the first page, read apart because OCR can misread the middle dot), no "Search in Directory" while results show, the cards hidden while results show, and, with the toolbar at the bottom, a right click on `#ci_group` opening a menu with "Open". It then opens `ci_channel` with its row's Open (the prepared channel with "Tap Join channel", no "Open channel" alert on the way, the subscriber count in the toolbar and the banner), joins it, and checks, in both layouts, the note "All results on this page are already in your chats" when the typed `ci_channel` finds only the listed chat, and, with "seeded notes" open and `ci seed` typed, `ci_channel` as its chat's row among the results (OCR reads no `#` before its name and no "Open" next to it; `#ci_group` keeps "Open") that opens the channel without an alert; and the same row in the New chat sheet. A passing search run logs 143 checks, the backend's two included, 117 of them before Open on `ci_channel`. - `backend.sh start ` / `resume ` / `stop `: the test backend, below; `resume` starts the same backend again from a stopped work dir, for the Claude job. - `claude-test.sh`, `claude-test.md` (the prompt), `claude-search.md` (the prompt when the backend runs), `claude-scope` (whether the prompt uses the test backend), `claude-guard.py`, `claude-usage.py`: the Claude job. - `build-core.sh`, `install-core.sh`, `build-backend.sh`, `earlier-run.sh`, `release-app.sh`, `test-summary.py`: builds, reuse and reports. @@ -41,7 +41,7 @@ It writes the directory's address to `/directory-link` and the SMP server's ## The Claude job -It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. Which branches may use it is decided only by that environment's branch rule in the repository settings: on a branch the rule doesn't allow, a requested Claude run fails at the environment check. It runs once the app job has passed, on that job's distributable, one per branch at a time. The token goes only to the step that runs `claude`; `claude-test.sh` passes it to the `claude` process alone, and every `ui` call unsets it too. The job starts the test backend when `claude-scope` says `backend: yes`, as it does on this branch; a run that asks for Claude must then ask for the backend too, or the plan job fails. The app job built the distributable pointing at its own backend's directory, which a new backend would not have, so the app job saves that backend's stopped work dir (the artifact `desktop-backend-state`, without the binaries) and the Claude job starts it again with `backend.sh resume`: the same servers' keys and databases, so the same addresses. That artifact holds the test backend's databases and server keys, unencrypted, and like every artifact of this public repository anyone signed in to GitHub can download it while it is kept (a day); with it, someone could act as the test directory or relay while the Claude job runs and show the session crafted results. This is an accepted risk: these are throwaway test identities, the session has no shell and no token in reach of its tools, so what is at stake is the test's integrity, and encrypting the state would need a key only the two jobs hold, a second secret, which this repository's rules for the Claude job don't allow. With the backend it sets the app up at the chat list (`smoke-test.sh … setup`) and gives Claude `claude-search.md`, the online search test, in place of the onboarding path in `claude-test.md`: the directory and full searches in both layouts (parts A and B), then part C, which joins `ci_channel` from its Directory row and checks that later searches show it as a chat (the note for a page of chats already listed, its chat row in the Directory part of the chat list and of New chat, opening without an alert). The job installs Claude Code at the version the iOS workflow pins too, and runs `claude-haiku-5-5` with `--max-turns 120`, `--max-budget-usd 3` and the iOS sessions' other flags (`--restricted`, `--permission-prompts none`, `--effort high`). Its Bash runs only single `ui` commands (`claude-guard.py`, a PreToolUse hook: Claude Code runs read-only commands without asking in every permission mode). Before anything the session wrote is printed or uploaded, `claude-test.sh` checks it for the token; if it's there, the output, transcript, Claude Code's stderr and the guard log are deleted and the job fails. It passes when Claude's final message ends with `CLAUDE-TEST: PASS`; `report.md`, the transcript, the guard's log and `claude-usage.json` are in `desktop-claude-test`. +It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. Which branches may use it is decided only by that environment's branch rule in the repository settings: on a branch the rule doesn't allow, a requested Claude run fails at the environment check. It runs once the app job has passed, on that job's distributable, one per branch at a time. The token goes only to the step that runs `claude`; `claude-test.sh` passes it to the `claude` process alone, and every `ui` call unsets it too. The job starts the test backend when `claude-scope` says `backend: yes`, as it does on this branch; a run that asks for Claude must then ask for the backend too, or the plan job fails. The app job built the distributable pointing at its own backend's directory, which a new backend would not have, so the app job saves that backend's stopped work dir (the artifact `desktop-backend-state`, without the binaries) and the Claude job starts it again with `backend.sh resume`: the same servers' keys and databases, so the same addresses. That artifact holds the test backend's databases and server keys, unencrypted, and like every artifact of this public repository anyone signed in to GitHub can download it while it is kept (a day); with it, someone could act as the test directory or relay while the Claude job runs and show the session crafted results. This is an accepted risk: these are throwaway test identities, the session has no shell and no token in reach of its tools, so what is at stake is the test's integrity, and encrypting the state would need a key only the two jobs hold, a second secret, which this repository's rules for the Claude job don't allow. With the backend it sets the app up at the chat list (`smoke-test.sh … setup`) and gives Claude `claude-search.md`, the online search test, in place of the onboarding path in `claude-test.md`: the directory search with its consent, Open, the load row and the right-click menu (part A) and the Search key, which searches the directory only and keeps "Connect to …" (part B), in both layouts, then part C, which opens `ci_channel` with its row's Open, without an alert, joins it, and checks that later searches show it as a chat (the note for a page of chats already listed, its chat row among the results of the chat list and of New chat, opening without an alert). The job installs Claude Code at the version the iOS workflow pins too, and runs `claude-haiku-5-5` with `--max-turns 120`, `--max-budget-usd 3` and the iOS sessions' other flags (`--restricted`, `--permission-prompts none`, `--effort high`). Its Bash runs only single `ui` commands (`claude-guard.py`, a PreToolUse hook: Claude Code runs read-only commands without asking in every permission mode). Before anything the session wrote is printed or uploaded, `claude-test.sh` checks it for the token; if it's there, the output, transcript, Claude Code's stderr and the guard log are deleted and the job fails. It passes when Claude's final message ends with `CLAUDE-TEST: PASS`; `report.md`, the transcript, the guard's log and `claude-usage.json` are in `desktop-claude-test`. ## Cost diff --git a/scripts/ci/ios-sim/README.md b/scripts/ci/ios-sim/README.md index 092d68ca77..911210d0b4 100644 --- a/scripts/ci/ios-sim/README.md +++ b/scripts/ci/ios-sim/README.md @@ -27,7 +27,7 @@ It runs in its own job, `claude`, once the app job has passed: the job installs Each session's final message ends with `CLAUDE-TEST: PASS`, or `CLAUDE-TEST: FAIL` and the failed cases; the step fails unless every session passes. - `claude-scope`: whether the cases use the test backend, the model, the layouts, the cases per layout, optional entries per layout, `max-turns` and `max-budget-usd` per session. -- `device-test-plan.md`: the plan (`| ID | Do | Expect |` tables): here, online search (directory and SimpleX names, chat list and New chat), about 50 cases. A feature branch replaces it with its own and names its cases in `claude-scope`. +- `device-test-plan.md`: the plan (`| ID | Do | Expect |` tables): here, online search (SimpleX Directory, the Directory search switch, "Connect to …" next to the results, chat list and New chat), about 50 cases. A feature branch replaces it with its own and names its cases in `claude-scope`. - `claude-prompt.md`: what any session needs to know about the app, the `ui` helper and the test backend; `claude-test.sh` adds the `ui` help, the session's cases, layout, start state and limits, the seeded backend and the plan. - `claude-guard.py`: a PreToolUse hook that lets through only one plain `ui` command per Bash call, since read-only commands run without asking in every permission mode and Bash inherits the token. Read, Write and Edit are allowed only in the session's own directory, so a session can't change `ui` or the scripts it runs. After each session `claude-test.sh` deletes the session's files, and fails, if the token appears in them or the check for it fails. - `claude-usage.sh`: each session's turns, duration, tokens and cost "estimated at API prices" (Claude Code's own estimate, not a bill) to the job summary and `claude-usage.json`, or why they are unknown (a session cut short before its token check is not uploaded). @@ -47,7 +47,7 @@ Haiku 5.5 drives the app well: the smoke plan takes a few dozen turns and cents; - a local SMP server with a `[NAMES]` section and `names-resolver.py`, the names `cinews.testing`, `ciboth.testing`, `cialice.testing` and `cibobby.testing` claimed by their owners, and the contacts cialice and Robert, online and accepting connections; its address goes to `/names-server`; - checks before the app starts: a search sent to the directory as a message must list `ci_group` and `ci_channel`, `/_connect plan` must resolve each name from a fresh peer, and a search request as the app sends it (`/_service_request`) must find `ci_group` and `ci_channel`; then the searches "ci seed" and "ci avatar" are followed page by page, recording each page's entries and size against the response limit in `seed.md`. -With the backend, the workflow points the app at the test directory before building it (`DIRECTORY_SERVICE_LINK` in `apps/ios/Shared/Model/DirectorySearch.swift`, the only change it makes to app sources), and the smoke test goes on to search it: the Directory header above the results and their avatars, in both layouts; once the profile has joined `ci_cards`, the note "All results on this page are already in your chats" when that channel, listed as a chat, is the page's only result; the chat list's cards shown while typing and hidden under the results, in both layouts; `ci_cards` found by `ci cards` (which the local filter doesn't match) shown as its chat row, with no `#ci_cards` or "Join channel", and opening the channel, on the chat list in both layouts and in New chat; and "Show more" disabled while its page loads. +With the backend, the workflow points the app at the test directory before building it (`DIRECTORY_SERVICE_LINK` in `apps/ios/Shared/Model/DirectorySearch.swift`, the only change it makes to app sources), and the smoke test goes on to search it: the first search's "Enable Directory search?" alert, where Cancel shows no results and keeps "Search in Directory", and Enable searches, with no alert on any later search; `#ci_group` and `#ci_channel` with Open and their avatars, with no "Directory" header and no "Search in Directory" next to them; the Search key with `cinews` typed, which shows directory results and keeps "Connect to cinews.testing", with no "Names"; the same results in the top layout; Open on `#ci_cards`, which reaches the prepared channel's "Tap Join channel" with no "Open channel" alert on the way and its subscriber count in the toolbar and the banner, then joins it; once joined, the note "All results on this page are already in your chats" when that channel, listed as a chat, is the page's only result; the chat list's cards shown while typing and hidden under the results, in both layouts; `ci_cards` found by `ci cards` (which the local filter doesn't match) shown as its chat row, with no `#ci_cards` or "Open", and opening the channel, on the chat list in both layouts and in New chat, where `ci seed` also shows `#ci_group` with Open; the load row for `ci avatar`, which reads "Load more · N left", is disabled while its page loads with the directory stopped, turns into "Search failed, tap to retry", and loads more entries once the directory is back; and last, as a check that warns rather than fails, a long press on `#ci_group` that shows a menu with Open (screenshot `long-press-menu`). It writes what it seeded to `/seed.md` (the Claude sessions get it) and logs to `/*.log`; `resume` (the same backend again from a stopped work dir, for the Claude job), `stop`, `directory-stop`, `directory-start` and `listings` are its other commands. The app reaches it through the simulator, which shares the runner's network; pointing the app at the names server is a few steps in Settings, described in `claude-prompt.md`.