mirror of
https://github.com/simplex-chat/simplex-chat.git
synced 2026-11-12 05:07:56 +00:00
ci: run the device test plan with Claude on the runner [claude-test]
After the smoke test, claude-test.sh runs one Claude Code session (pinned 2.1.300) per layout named in claude-scope, with the plan's cases, model, turns and budget from there. A session gets only Bash for the ui helper and Read/Write/Edit in its own directory (--restricted, dontAsk, no MCP), drives the app through ui, and writes report.md with screenshots; its transcript is stream-json. The app job declares the app-sim-claude environment; only the Claude step gets CLAUDE_CODE_OAUTH_TOKEN, and it runs only on workflow_dispatch or a head commit with [claude-test]. claude-usage.sh writes each session's turns, tokens and cost (estimated at API prices) to the job summary and claude-usage.json, even when the session fails. The plan is copied from the iOS directory search run. This run: C2, D2, D7 in the one-hand UI with Haiku 5.5, to validate the harness.
This commit is contained in:
1 parent
368e8e1d65
commit
79a1f5f87d
6 files changed
+345
-2
No files matched your search
@@ -193,7 +193,12 @@ jobs:
|
||||
# core and backend are skipped when an earlier run built them
|
||||
if: ${{ !cancelled() && !failure() }}
|
||||
runs-on: macos-latest
|
||||
timeout-minutes: 90
|
||||
environment:
|
||||
name: app-sim-claude
|
||||
deployment: false
|
||||
timeout-minutes: 330
|
||||
env:
|
||||
CLAUDE_RUN: ${{ github.event_name == 'workflow_dispatch' || contains(github.event.head_commit.message, '[claude-test]') }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -262,6 +267,24 @@ jobs:
|
||||
- name: Run smoke test
|
||||
run: scripts/ci/ios-sim/smoke-test.sh "$RUNNER_TEMP/DerivedData/Build/Products/Debug-iphonesimulator/SimpleX.app" "$RUNNER_TEMP/smoke-test"
|
||||
|
||||
- name: Install Claude Code
|
||||
if: env.CLAUDE_RUN == 'true'
|
||||
run: |
|
||||
npm install -g @anthropic-ai/claude-code@2.1.300
|
||||
claude --version
|
||||
|
||||
- name: Run the device test plan with Claude
|
||||
if: env.CLAUDE_RUN == 'true'
|
||||
timeout-minutes: 240
|
||||
env:
|
||||
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
BACKEND_DIR: ${{ runner.temp }}/backend
|
||||
run: scripts/ci/ios-sim/claude-test.sh "$RUNNER_TEMP/claude-test"
|
||||
|
||||
- name: Report Claude usage
|
||||
if: ${{ always() && env.CLAUDE_RUN == 'true' }}
|
||||
run: scripts/ci/ios-sim/claude-usage.sh "$RUNNER_TEMP/claude-test"
|
||||
|
||||
- name: Stop test backend
|
||||
if: ${{ !cancelled() }}
|
||||
run: scripts/ci/ios-sim/backend.sh stop "$RUNNER_TEMP/backend"
|
||||
@@ -280,6 +303,17 @@ jobs:
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ios-sim-backend-logs
|
||||
path: ${{ runner.temp }}/backend/*.log
|
||||
path: |
|
||||
${{ runner.temp }}/backend/*.log
|
||||
${{ runner.temp }}/backend/seed.md
|
||||
retention-days: 3
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Upload Claude test results
|
||||
if: ${{ always() && env.CLAUDE_RUN == 'true' }}
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ios-sim-claude-test
|
||||
path: ${{ runner.temp }}/claude-test
|
||||
retention-days: 14
|
||||
if-no-files-found: warn
|
||||
Reference in new issue
Block a user