diff --git a/scripts/ci/ios-sim/claude-prompt.md b/scripts/ci/ios-sim/claude-prompt.md index bd43329fdb..8576537dc6 100644 --- a/scripts/ci/ios-sim/claude-prompt.md +++ b/scripts/ci/ios-sim/claude-prompt.md @@ -24,7 +24,7 @@ You are testing the SimpleX Chat iOS app in an iPhone 17 simulator (iOS 26.5, sc - **Counting requests.** `ui requests` tells how many directory searches the app sent; compare before and after an action to prove that something was or was not sent (C6, C9, D12, D13). - **The test directory** can be stopped and started (`ui directory stop|start`). Use it instead of turning networking off (D8, D9; the simulator shares the runner's network). After `ui directory start`, wait about 15 s before searching again; while it is stopped, searches fail after 10 s. - **Text size** (D3): `ui text-size accessibility-large`, and back to `ui text-size medium` afterwards. Only one phone size is available. -- **Joining.** Tapping a directory row of an entry the profile does not know yet starts the join flow (an alert to join; confirm it to join); joining a channel works (the test relay is online), joining a group may stay "connecting". A joined or connecting entry is known: tapping it shows an alert to open it ("Open channel", "Open group") instead, and a sheet behind that alert stays until you choose "Open". So each case that joins uses the entry named for it under "This session", which no earlier case has touched. Joined chats appear in the chat list, and the profile is no longer "without conversations". +- **Joining** (the app's normal flow, from `NewChatView.swift`; an earlier session took its first step for a failure). Tapping a row of a group or channel the profile does not know yet shows an alert with its name (and subscriber count for a channel) and "Cancel" / "Open channel" or "Open group": that is the start of joining, not a sign that the entry is known. "Open …" opens the chat, not yet joined, with "Tap Join channel" in it and a "Join channel" / "Join group" bar at the bottom; tapping that bar joins (a channel then shows "you are subscriber"; a group may stay "connecting"). A contact works the same way: "Open new chat", then the chat's connect button. An entry already opened, joined or connecting shows the same kind of alert, plus your role ("You are a subscriber", "You are a member") once joined. So each case that joins uses the entry named for it under "This session", which no earlier case has touched. Opened and joined chats appear in the chat list, and the profile is no longer "without conversations". - **New chat and the pencil.** With the top bar the pencil is hidden while the search field is focused; tapping the "Search in Directory" row hides the keyboard, and the pencil shows again while that search runs. - **A second profile** (D14, and D15 once the first profile has chats): open the picker, "Your chat profiles", add a profile; a new profile has no conversations. Switch back to the first profile when done. - **If the app gets stuck**, `ui relaunch` restarts it (search text and results are cleared, everything else stays). @@ -45,11 +45,12 @@ The test backend resolves SimpleX names through its own SMP server, whose addres **What the names are for:** `cinews` is a listed channel (N1, N6, the full search); `cialice` and `cibobby` are contacts that accept a connection at once (N2, N5, N10, W7, W8, G1; `cibobby`'s profile is called "Robert", so its name is not in its display name); `ciboth` has both a contact address and a channel (N3); `cinone` is not registered (N4). N7 needs the second profile with "To resolve names" off for both operators (step 2 only, no server added). Which case uses which contact in this layout is under "This session". -**The full search** (in both layouts, after N1): with a local chat whose name contains "cinews" (join the group cinews_digest from its Directory row first, if it is not in your chats yet), type `cinews` and press the Search key. The plan's parts are the chats already loaded, a "Names" part and a "Directory" part (N1: the "Names" header above the Directory part; W4 for New chat: "Names" and "Directory" after "Your Contacts"). Record for this layout, from the tree, every header and row with its y, top to bottom: the local chats, "Names" with its rows, "Directory" with its rows (cinews_weekly; cinews should not repeat there, as it is the Names row), and say whether the part names and their order match the plan. Do the same in New chat (W4). +**The full search** (after N1): with a local chat whose name contains "cinews" (join the group cinews_digest from its Directory row first, if it is not in your chats yet), type `cinews` and press the Search key. The plan's parts are the chats already loaded, a "Names" part and a "Directory" part (N1: the "Names" header above the Directory part; W4 for New chat: "Names" and "Directory" after "Your Contacts"). Record for this layout, from the tree, every header and row with its y, top to bottom: the local chats, "Names" with its rows, "Directory" with its rows (cinews_weekly; cinews should not repeat there, as it is the Names row), and say whether the part names and their order match the plan. Do the same in New chat (W4) when the session's cases include New chat. ## How to run the cases - Run each case of the session's list, in the plan's order unless a case needs another first (for example D15 needs a profile without conversations, U1 and U2 need a joined channel from D6). +- The plan's tables (ID, Do, Expect) are at the end of `prompt.md` in the current directory: read the row before running a case and do exactly what it says. An earlier session did not find them and filed other actions under some IDs. - For each case, do what the plan says, decide pass or fail against the plan's "Expect" column, and take at least one `ui shot` at the moment that shows the result, named after the case and layout without a number, for example `ui shot D2-reachable` (the helper numbers the files). - Pass only what you saw: quote the evidence (element labels, values and coordinates from the tree, or what a screenshot shows). If an expectation cannot be checked here, say which part and why, and mark the case passed for the parts you checked only if those parts are the substance of the case; otherwise skipped. - **Skip** any case that needs something this runner lacks, saying exactly what is missing. Known: U3 (needs a listed group joined through another link), W10 (needs a notification from another profile), W11 unless "Archived contacts" is shown. Do not skip a case only because it takes several steps. @@ -64,7 +65,7 @@ Write `report.md` in the current directory as you go, so a session that is cut s 1. `# Device test report: ` and one line on the app state at the start. 2. `## Results`: a table with one row per case of the session's list: `| ID | Result | Evidence | Screenshots |`, Result being pass, fail or skipped (with the reason). 3. `## Directory header`: per screen with results, the heading's y and the first and last result's y, and the heading's place in the tree order. -4. `## Avatars and the size limit`: what each "Show more" added, compared with the table below, and how the avatars looked. +4. `## Avatars and the size limit` (when the session's cases include D2 or D7): what each "Show more" added, compared with the table below, and how the avatars looked. 5. `## Findings`: app bugs, each with steps, what you saw, what you expected, screenshots. 6. `## Harness notes`: anything about `ui` or the runner that got in the way. diff --git a/scripts/ci/ios-sim/claude-scope b/scripts/ci/ios-sim/claude-scope index 0818b85368..dd3057e342 100644 --- a/scripts/ci/ios-sim/claude-scope +++ b/scripts/ci/ios-sim/claude-scope @@ -1,9 +1,11 @@ -# What claude-test.sh runs: one Claude session per layout (reachable, top), the plan's cases, and each session's bounds. +# What claude-test.sh runs: one Claude session per layout (reachable, top), each with its cases, and each session's bounds. +# A case runs in both layouts only when the layouts render it differently; the rest run in one. # entries- tells that session which directory entries and names its cases use, so none meets an entry already known. model: claude-haiku-5-5 layouts: reachable top -cases: all of the plan's cases (C1-C9, D1-D15, U1-U3, N1-N11, W1-W11, G1-G4) and the full search, skipping those named under "How to run the cases" -entries-reachable: D6 joins ci_channel, which U1 then searches by name and U2 by "seed channel"; W6 joins ci_avatar_c2; D11 searches "private" (only the local chat "Private notes" has it); D14 and D15 use a second profile you add, which N7 then uses; N10 connects to @cialice.testing, which N5, W7 and G1 then use; W8 first connects to @cibobby.testing (Robert) from its Names row -entries-top: D6 joins ci_avatar_c1, which U1 then searches by name and U2 by "avatar channel 1"; W6 joins ci_avatar_c3; D11 searches "private"; D14, D15 and N7 use the second profile from the earlier session (add one if it is missing); the contacts cialice and Robert are already connected (N5, W7, W8, G1 use them); N10 taps the Names row of the channel #ciboth.testing +cases-reachable: the SimpleX names setup, then N1-N11, W4 with names, W7, W8, G1 and the full search (chat list and New chat) +cases-top: N1 and N4 (this layout places the "Connect to …" row and the Names and Directory parts differently) and the full search on the chat list only (New chat looks the same in both layouts); the names setup is already done +entries-reachable: N7 uses a second profile you add (the first profile has no other); N10 connects to @cialice.testing, which N5, W7 and G1 then use; W8 first connects to @cibobby.testing (Robert) from its Names row; the full search joins the group cinews_digest +entries-top: the earlier session joined cinews_digest and connected to cialice and Robert; N1 and N4 join nothing max-turns: 900 max-budget-usd: 10 diff --git a/scripts/ci/ios-sim/claude-test.sh b/scripts/ci/ios-sim/claude-test.sh index cf60b80a8e..9b132f79bf 100755 --- a/scripts/ci/ios-sim/claude-test.sh +++ b/scripts/ci/ios-sim/claude-test.sh @@ -24,7 +24,6 @@ scope() { model=$(scope model) layouts=$(scope layouts) -cases=$(scope cases) max_turns=$(scope max-turns) max_budget=$(scope max-budget-usd) @@ -59,7 +58,7 @@ write_prompt() { echo "## This session" echo echo "- Layout: $(layout_name "$layout")." - echo "- Cases: $cases." + echo "- Cases: $(scope "cases-$layout")." echo "- Entries to use: $(scope "entries-$layout")." echo "- Link to paste (C4, W3): $(cat "$BACKEND_DIR/directory-link") (the test directory's address)." echo "- Names server to add in the app: $(cat "$BACKEND_DIR/names-server")." @@ -82,7 +81,7 @@ for layout in $layouts; do mkdir -p "$session/shots" ui layout "$layout" write_prompt "$layout" "$previous" > "$session/prompt.md" - echo "::group::Claude session: $layout, cases $cases, $model, $max_turns turns, \$$max_budget" + echo "::group::Claude session: $layout, $model, $max_turns turns, \$$max_budget" session_status=0 # outside the session's directory, so the session cannot change it export CLAUDE_GUARD_LOG=$out/$layout-guard.log