fixup! ci: check that a page of chats already listed adds nothing to the search results

This commit is contained in:
spacedandy-agents-simplex committed 2026-10-21 15:27:45 +00:00
1 parent a429a5c6f1
commit 181c92e9a4
4 files changed
+15 -11

No files matched your search

+2 -2
View File
@@ -24,7 +24,7 @@ Queueing for a runner adds from under a minute to 20 minutes. Results are artifa
## Scripts
- `ui`: the UI driver. Compose for Desktop has no accessibility tree on Linux, so it finds text on screenshots by OCR (tesseract) and acts with xdotool on an Xvfb display. `ui` alone prints its commands: start (with app settings), shot, text, find, wait, click, right-click, click-text, scroll, type, key, avatar (a profile image's colour), ink (a text's height), do (several in a row), stop. `screen-text.py`, `screen-colour.py` and `screen-ink.py` are its helpers.
- `smoke-test.sh <app dir> <out dir> [start|onboarding|setup|backend|search]`: the scripted cases; the default is onboarding and both layouts. `setup` also creates a local group and makes the test SMP server the app's only names server; `backend` does that, connects to the seeded name, and checks both layouts; `search`, which the workflow runs when the backend started, also searches in both layouts: "Search in Directory" for "ci seed", which with the toolbar at the bottom first asks "Enable Directory search?" (Cancel searches nothing and keeps "Search in Directory", then Enable searches; no later search asks), and the search for the name `seeded` with Enter, which shows the local chat and the directory results with no "Names" header and no `ci_contact` row, keeps "Connect to seeded.testing", and orders "Connect to …", the local chat, then the results from the field; and with the toolbar at the bottom the same search in the New chat sheet. It checks one list without a "Directory" header, "Open" in each result's row, avatars, counts, the load row ("Load more" and "1 left" after the first page, read apart because OCR can misread the middle dot), no "Search in Directory" while results show, the cards hidden while results show, and, with the toolbar at the bottom, a right click on `#ci_group` opening a menu with "Open". It then opens `ci_channel` with its row's Open (the prepared channel with "Tap Join channel", no "Open channel" alert on the way, the subscriber count in the toolbar and the banner), joins it, and checks, in both layouts, that the typed `ci_channel`, which finds only the listed chat, adds no row and no note (it waits out the request, as nothing marks its end), and, with "seeded notes" open and `ci seed` typed, `ci_channel` as its chat's row among the results (OCR reads no `#` before its name and no "Open" next to it; `#ci_group` keeps "Open") that opens the channel without an alert; and the same row in the New chat sheet. A passing search run logs 145 checks, the backend's two included, 117 of them before Open on `ci_channel`.
- `smoke-test.sh <app dir> <out dir> [start|onboarding|setup|backend|search]`: the scripted cases; the default is onboarding and both layouts. `setup` also creates a local group and makes the test SMP server the app's only names server; `backend` does that, connects to the seeded name, and checks both layouts; `search`, which the workflow runs when the backend started, also searches in both layouts: "Search in Directory" for "ci seed", which with the toolbar at the bottom first asks "Enable Directory search?" (Cancel searches nothing and keeps "Search in Directory", then Enable searches; no later search asks), and the search for the name `seeded` with Enter, which shows the local chat and the directory results with no "Names" header and no `ci_contact` row, keeps "Connect to seeded.testing", and orders "Connect to …", the local chat, then the results from the field; and with the toolbar at the bottom the same search in the New chat sheet. It checks one list without a "Directory" header, "Open" in each result's row, avatars, counts, the load row ("Load more" and "1 left" after the first page, read apart because OCR can misread the middle dot), no "Search in Directory" while results show, the cards hidden while results show, and, with the toolbar at the bottom, a right click on `#ci_group` opening a menu with "Open". It then opens `ci_channel` with its row's Open (the prepared channel with "Tap Join channel", no "Open channel" alert on the way, the subscriber count in the toolbar and the banner), joins it, and checks, in both layouts, that the typed `ci_channel`, which finds only the listed chat, adds no row and no note, after a search that ran ("Search in Directory" gone; it waits out the request, as nothing marks its end), and, with "seeded notes" open and `ci seed` typed, `ci_channel` as its chat's row among the results (OCR reads no `#` before its name and no "Open" next to it; `#ci_group` keeps "Open") that opens the channel without an alert; and the same row in the New chat sheet. A passing search run logs 147 checks, the backend's two included, 117 of them before Open on `ci_channel`.
- `backend.sh start <tar.gz> <dir>` / `resume <tar.gz> <dir>` / `stop <dir>`: the test backend, below; `resume` starts the same backend again from a stopped work dir, for the Claude job.
- `claude-test.sh`, `claude-test.md` (the prompt), `claude-search.md` (the prompt when the backend runs), `claude-scope` (whether the prompt uses the test backend), `claude-guard.py`, `claude-usage.py`: the Claude job.
- `build-core.sh`, `install-core.sh`, `build-backend.sh`, `earlier-run.sh`, `release-app.sh`, `test-summary.py`: builds, reuse and reports.
@@ -41,7 +41,7 @@ It writes the directory's address to `<dir>/directory-link` and the SMP server's
## The Claude job
It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. Which branches may use it is decided only by that environment's branch rule in the repository settings: on a branch the rule doesn't allow, a requested Claude run fails at the environment check. It runs once the app job has passed, on that job's distributable, one per branch at a time. The token goes only to the step that runs `claude`; `claude-test.sh` passes it to the `claude` process alone, and every `ui` call unsets it too. The job starts the test backend when `claude-scope` says `backend: yes`, as it does on this branch; a run that asks for Claude must then ask for the backend too, or the plan job fails. The app job built the distributable pointing at its own backend's directory, which a new backend would not have, so the app job saves that backend's stopped work dir (the artifact `desktop-backend-state`, without the binaries) and the Claude job starts it again with `backend.sh resume`: the same servers' keys and databases, so the same addresses. That artifact holds the test backend's databases and server keys, unencrypted, and like every artifact of this public repository anyone signed in to GitHub can download it while it is kept (a day); with it, someone could act as the test directory or relay while the Claude job runs and show the session crafted results. This is an accepted risk: these are throwaway test identities, the session has no shell and no token in reach of its tools, so what is at stake is the test's integrity, and encrypting the state would need a key only the two jobs hold, a second secret, which this repository's rules for the Claude job don't allow. With the backend it sets the app up at the chat list (`smoke-test.sh … setup`) and gives Claude `claude-search.md`, the online search test, in place of the onboarding path in `claude-test.md`: the directory search with its consent, Open, the load row and the right-click menu (part A) and the Search key, which searches the directory only and keeps "Connect to …" (part B), in both layouts, then part C, which opens `ci_channel` with its row's Open, without an alert, joins it, and checks that later searches show it as a chat (the note for a page of chats already listed, its chat row among the results of the chat list and of New chat, opening without an alert). The job installs Claude Code at the version the iOS workflow pins too, and runs `claude-haiku-5-5` with `--max-turns 120`, `--max-budget-usd 3` and the iOS sessions' other flags (`--restricted`, `--permission-prompts none`, `--effort high`). Its Bash runs only single `ui` commands (`claude-guard.py`, a PreToolUse hook: Claude Code runs read-only commands without asking in every permission mode). Before anything the session wrote is printed or uploaded, `claude-test.sh` checks it for the token; if it's there, the output, transcript, Claude Code's stderr and the guard log are deleted and the job fails. It passes when Claude's final message ends with `CLAUDE-TEST: PASS`; `report.md`, the transcript, the guard's log and `claude-usage.json` are in `desktop-claude-test`.
It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. Which branches may use it is decided only by that environment's branch rule in the repository settings: on a branch the rule doesn't allow, a requested Claude run fails at the environment check. It runs once the app job has passed, on that job's distributable, one per branch at a time. The token goes only to the step that runs `claude`; `claude-test.sh` passes it to the `claude` process alone, and every `ui` call unsets it too. The job starts the test backend when `claude-scope` says `backend: yes`, as it does on this branch; a run that asks for Claude must then ask for the backend too, or the plan job fails. The app job built the distributable pointing at its own backend's directory, which a new backend would not have, so the app job saves that backend's stopped work dir (the artifact `desktop-backend-state`, without the binaries) and the Claude job starts it again with `backend.sh resume`: the same servers' keys and databases, so the same addresses. That artifact holds the test backend's databases and server keys, unencrypted, and like every artifact of this public repository anyone signed in to GitHub can download it while it is kept (a day); with it, someone could act as the test directory or relay while the Claude job runs and show the session crafted results. This is an accepted risk: these are throwaway test identities, the session has no shell and no token in reach of its tools, so what is at stake is the test's integrity, and encrypting the state would need a key only the two jobs hold, a second secret, which this repository's rules for the Claude job don't allow. With the backend it sets the app up at the chat list (`smoke-test.sh … setup`) and gives Claude `claude-search.md`, the online search test, in place of the onboarding path in `claude-test.md`: the directory search with its consent, Open, the load row and the right-click menu (part A) and the Search key, which searches the directory only and keeps "Connect to …" (part B), in both layouts, then part C, which opens `ci_channel` with its row's Open, without an alert, joins it, and checks that later searches show it as a chat (no row and no note for a page of chats already listed, its chat row among the results of the chat list and of New chat, opening without an alert). The job installs Claude Code at the version the iOS workflow pins too, and runs `claude-haiku-5-5` with `--max-turns 120`, `--max-budget-usd 3` and the iOS sessions' other flags (`--restricted`, `--permission-prompts none`, `--effort high`). Its Bash runs only single `ui` commands (`claude-guard.py`, a PreToolUse hook: Claude Code runs read-only commands without asking in every permission mode). Before anything the session wrote is printed or uploaded, `claude-test.sh` checks it for the token; if it's there, the output, transcript, Claude Code's stderr and the guard log are deleted and the job fails. It passes when Claude's final message ends with `CLAUDE-TEST: PASS`; `report.md`, the transcript, the guard's log and `claude-usage.json` are in `desktop-claude-test`.
## Cost
+5 -3
View File
@@ -569,13 +569,15 @@ known_channel_searches() {
"$ui" type ci_channel
"$ui" wait "Search in Directory" 15
"$ui" click-text "Search in Directory"
# nothing marks the end of a search that adds no row: wait out the request, which times out after 10 s
sleep 12
# nothing marks the end of a search that adds no row: wait it out (once the directory's link is fetched, the request
# fails after 10 s); "Search in Directory" hides once a search for the text runs
sleep 20
if on_screen "Search failed, tap to retry"; then
"$ui" click-text "Search failed, tap to retry"
sleep 12
sleep 20
fi
"$ui" shot "$layout-known-listed"
check "$layout: the search for the listed chat ran" not_on_screen "Search in Directory"
check "$layout: the search for the listed chat didn't fail" not_on_screen "Search failed, tap to retry"
check "$layout: a page of chats already listed adds no note" not_on_screen "All results on this page"
check "$layout: no #ci_channel result next to the listed chat" no_ci_channel_directory_row
+1 -1
View File
@@ -47,7 +47,7 @@ Haiku 5.5 drives the app well: the smoke plan takes a few dozen turns and cents;
- a local SMP server with a `[NAMES]` section and `names-resolver.py`, the names `cinews.testing`, `ciboth.testing`, `cialice.testing` and `cibobby.testing` claimed by their owners, and the contacts cialice and Robert, online and accepting connections; its address goes to `<work dir>/names-server`;
- checks before the app starts: a search sent to the directory as a message must list `ci_group` and `ci_channel`, `/_connect plan` must resolve each name from a fresh peer, and a search request as the app sends it (`/_service_request`) must find `ci_group` and `ci_channel`; then the searches "ci seed" and "ci avatar" are followed page by page, recording each page's entries and size against the response limit in `seed.md`.
With the backend, the workflow points the app at the test directory before building it (`DIRECTORY_SERVICE_LINK` in `apps/ios/Shared/Model/DirectorySearch.swift`, the only change it makes to app sources), and the smoke test goes on to search it: the first search's "Enable Directory search?" alert, where Cancel shows no results and keeps "Search in Directory", and Enable searches, with no alert on any later search; `#ci_group` and `#ci_channel` with Open and their avatars, with no "Directory" header and no "Search in Directory" next to them; the Search key with `cinews` typed, which shows directory results and keeps "Connect to cinews.testing", with no "Names"; the same results in the top layout; Open on `#ci_cards`, which reaches the prepared channel's "Tap Join channel" with no "Open channel" alert on the way and its subscriber count in the toolbar and the banner, then joins it; once joined, no row and no note when that channel, listed as a chat, is the page's only result (the step waits out the request, as nothing marks its end); the chat list's cards shown while typing and hidden under the results, in both layouts; `ci_cards` found by `ci cards` (which the local filter doesn't match) shown as its chat row, with no `#ci_cards` or "Open", and opening the channel, on the chat list in both layouts and in New chat, where `ci seed` also shows `#ci_group` with Open; the load row for `ci avatar`, which reads "Load more · N left", is disabled while its page loads with the directory stopped, turns into "Search failed, tap to retry", and loads more entries once the directory is back; and last, as a check that warns rather than fails, a long press on `#ci_group` that shows a menu with Open (screenshot `long-press-menu`).
With the backend, the workflow points the app at the test directory before building it (`DIRECTORY_SERVICE_LINK` in `apps/ios/Shared/Model/DirectorySearch.swift`, the only change it makes to app sources), and the smoke test goes on to search it: the first search's "Enable Directory search?" alert, where Cancel shows no results and keeps "Search in Directory", and Enable searches, with no alert on any later search; `#ci_group` and `#ci_channel` with Open and their avatars, with no "Directory" header and no "Search in Directory" next to them; the Search key with `cinews` typed, which shows directory results and keeps "Connect to cinews.testing", with no "Names"; the same results in the top layout; Open on `#ci_cards`, which reaches the prepared channel's "Tap Join channel" with no "Open channel" alert on the way and its subscriber count in the toolbar and the banner, then joins it; once joined, no row and no note when that channel, listed as a chat, is the page's only result, after a search that ran ("Search in Directory" gone; the step waits out the request, as nothing marks its end); the chat list's cards shown while typing and hidden under the results, in both layouts; `ci_cards` found by `ci cards` (which the local filter doesn't match) shown as its chat row, with no `#ci_cards` or "Open", and opening the channel, on the chat list in both layouts and in New chat, where `ci seed` also shows `#ci_group` with Open; the load row for `ci avatar`, which reads "Load more · N left", is disabled while its page loads with the directory stopped, turns into "Search failed, tap to retry", and loads more entries once the directory is back; and last, as a check that warns rather than fails, a long press on `#ci_group` that shows a menu with Open (screenshot `long-press-menu`).
It writes what it seeded to `<work dir>/seed.md` (the Claude sessions get it) and logs to `<work dir>/*.log`; `resume` (the same backend again from a stopped work dir, for the Claude job), `stop`, `directory-stop`, `directory-start` and `listings` are its other commands. The app reaches it through the simulator, which shares the runner's network; pointing the app at the names server is a few steps in Settings, described in `claude-prompt.md`.
+7 -5
View File
@@ -482,20 +482,22 @@ expect_no_join_row "the chat list after joining"
echo "after joining through Open, ci_cards is its chat row among the results and no longer a result to open"
clear_search
# the page's only result is that channel, which the list shows, so the search adds no row and no note; nothing marks
# its end, so the step waits out the request, which times out after 10 s
# its end, so the step waits it out: once the directory's link is fetched, the request fails after 10 s
type_search "ci_cards"
tap_search_in_directory
sleep 12
sleep 20
retry=$(labelled_containing "Search failed, tap to retry")
if [ -n "$retry" ]; then
tap_element "$retry"
sleep 12
sleep 20
fi
"$ui" shot page-in-chats
expect_in_tree -F 'StaticText "ci_cards"'
expect_no_join_row "the chat list listing ci_cards"
if "$ui" tree | grep -qF -e "All results on this page" -e "Search failed, tap to retry"; then
echo "error: the search whose only result is the listed ci_cards shows a note or failed"
tree=$("$ui" tree)
# "Search in Directory" hides once a search for the text runs
if grep -qF -e '"Search in Directory"' -e "All results on this page" -e "Search failed, tap to retry" <<< "$tree"; then
echo "error: the search whose only result is the listed ci_cards didn't run, failed or shows a note"
exit 1
fi
echo "the search whose only result is the listed ci_cards adds no row and no note"