ci: run the device test plan with Claude on the runner [claude-test]

After the smoke test, claude-test.sh runs one Claude Code session (pinned 2.1.300) per layout named in claude-scope, with the plan's cases, model, turns and budget from there. A session gets only Bash for the ui helper and Read/Write/Edit in its own directory (--restricted, dontAsk, no MCP), drives the app through ui, and writes report.md with screenshots; its transcript is stream-json. The app job declares the app-sim-claude environment; only the Claude step gets CLAUDE_CODE_OAUTH_TOKEN, and it runs only on workflow_dispatch or a head commit with [claude-test]. claude-usage.sh writes each session's turns, tokens and cost (estimated at API prices) to the job summary and claude-usage.json, even when the session fails. The plan is copied from the iOS directory search run. This run: C2, D2, D7 in the one-hand UI with Haiku 5.5, to validate the harness.
This commit is contained in:
spacedandy-agents-simplex committed 2026-10-15 11:46:04 +00:00
1 parent 368e8e1d65
commit 79a1f5f87d
6 files changed
+345 -2

No files matched your search

+36 -2
View File
@@ -193,7 +193,12 @@ jobs:
# core and backend are skipped when an earlier run built them
if: ${{ !cancelled() && !failure() }}
runs-on: macos-latest
timeout-minutes: 90
environment:
name: app-sim-claude
deployment: false
timeout-minutes: 330
env:
CLAUDE_RUN: ${{ github.event_name == 'workflow_dispatch' || contains(github.event.head_commit.message, '[claude-test]') }}
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -262,6 +267,24 @@ jobs:
- name: Run smoke test
run: scripts/ci/ios-sim/smoke-test.sh "$RUNNER_TEMP/DerivedData/Build/Products/Debug-iphonesimulator/SimpleX.app" "$RUNNER_TEMP/smoke-test"
- name: Install Claude Code
if: env.CLAUDE_RUN == 'true'
run: |
npm install -g @anthropic-ai/claude-code@2.1.300
claude --version
- name: Run the device test plan with Claude
if: env.CLAUDE_RUN == 'true'
timeout-minutes: 240
env:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
BACKEND_DIR: ${{ runner.temp }}/backend
run: scripts/ci/ios-sim/claude-test.sh "$RUNNER_TEMP/claude-test"
- name: Report Claude usage
if: ${{ always() && env.CLAUDE_RUN == 'true' }}
run: scripts/ci/ios-sim/claude-usage.sh "$RUNNER_TEMP/claude-test"
- name: Stop test backend
if: ${{ !cancelled() }}
run: scripts/ci/ios-sim/backend.sh stop "$RUNNER_TEMP/backend"
@@ -280,6 +303,17 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: ios-sim-backend-logs
path: ${{ runner.temp }}/backend/*.log
path: |
${{ runner.temp }}/backend/*.log
${{ runner.temp }}/backend/seed.md
retention-days: 3
if-no-files-found: warn
- name: Upload Claude test results
if: ${{ always() && env.CLAUDE_RUN == 'true' }}
uses: actions/upload-artifact@v4
with:
name: ios-sim-claude-test
path: ${{ runner.temp }}/claude-test
retention-days: 14
if-no-files-found: warn