Files
simplex-chat/scripts/ci/README.md
T
ad75fe706f store: sell and register SimpleX names (#7979)
* core: add prepaid name codes and length bounds

* store: run the names migration after badges

* store: issue and revoke prepaid name codes

* store: document issuing prepaid name codes

* store: move the ABI helpers to Names.Abi

* store: read the [names] and [chain] settings

* store: add the names chain client and reads

* store: refuse group name codes while names are off

* store: search names and preview links

* store: serve each site on its own port

* scripts: deploy the badges and names sites

* store: document search and the two sites

* store: add the drop_taken_codes migration

* store: read the names codes_only setting

* store: add the names store and order signatures

* store: take name orders by card, crypto or code

* store: document name orders and codes-only

* store: add the names transaction sender

* store: commit and register queued orders

* store: reveal and retry an owner's order

* store: document the gas key

* store: add the name balance migrations

* store: read crypto units, rates and confirmations

* store: keep a taken order's money as a balance

* store: spend a balance on another name

* store: record refunds with //refund

* store: document balances and refunds

* store: add the agreed terms migration

* store: read the terms and community names

* store: answer community and terms in search

* store: sign and store the agreed terms

* store: document the terms and community names

* store: add the invoiced units migration

* store: read the crypto an invoice asks

* store: credit late and overpaid crypto by share

* store: document crediting late crypto by share

* store: add the record changes migration

* store: find an owner's orders and move one

* store: relay owners' link edits on chain

* store: document link edits

* store: move isLoopbackHost to Config

* store: trust dev issuer keys on the local chain

* scripts: build the names site from a submodule

* scripts: add the dev setup and contract run

* store: document the dev setup and names image

* store: send a Stripe idempotency key for names

* tests: wait until a relay lets go of its edit

* store: refuse a link edit deadline under 10 min

* store: count reverted link edits in the year limit

* store: require https for a remote chain node

* store: refuse zero chain confirmations

* store: set up the dev stack from the names branch

* store: rerun a failed unpaid cleanup next pass

* store: never replace a succeeded Stripe intent

* store: refuse an ini setting written as key: value

* store: note the webapp submodule's pinned branch

* store: document confirmLanded judging by ownerOf

* store: update the names webapp submodule

* store-service: register names under the name policy, every name accepting its terms on chain (#7889)

* store: accept each name's terms on chain under the name policy

* ci: run the reduced build on sh/names-deploy

* store: address review: policy re-read, terms reads and startup log

* store: never credit an unconfirmed extra

* store: alert money sent to a paid name invoice

* store: refuse to start on another chain's node

* store: keep orders while the allowance is used up

* store: replace a stalled transaction after 3 blocks

* store: judge a name taken at confirmation depth

* store: bind the template's listener to loopback

* store: scan only the ids of active name orders

* store: refuse unknown [names] and [chain] settings

* store: fetch the webapp submodule in upgrade steps

* store: delete definitions only tests use

* store: lock only name invoices at settlement

* store: retry an unexpired commitment as usual

* store: share the helpers defined twice

* store: test route limits on the running service

* store: fix tests that could not fail

* ci: test the store service on Postgres

* store: count IPv6 clients by their /64

* store: sweep invoices read alone past a list cap

* store: cap checkouts of all clients together

* store: cap the long polls held at once

* store: stop the old badge project at upgrade

* store: keep search lookups free of previews

* store: read the ini as UTF-8 in any locale

* ci: run Postgres store tests in a UTF-8 locale

* store: cap the transactions one order sends

* store: cancel a card intent after three declines

* store: refuse a name price under a cent

* store: send each transaction with gas headroom

* store: cap searches and read terms once per block

* store: never log the group join link

* store: mask the database password at startup

* store: run one instance per database

* store: limit name codes to admins, 100 a day

* store: refuse HighSpeed settlement with names

* store: no unverified receipts with a trusted key

* store: cap the log lines of refused webhooks

* store: check Apple certificates at the signed date

* store: refuse a malformed Stripe publishable key

* store: check the owner before a refund

* store: fold the names migrations into one

* store: read onion name links through the proxy

* store: refund money after paid once confirmed

* store: limit the allowance wait to a dry run

* store: bind loopback when upgrading

* ci: cache and widen the store Postgres job

* store: fix stale and duplicate names tests

* store: fix the submodule path in the upgrade

* store: refuse unpaid name codes on the CLI

* store: apply the review's names test list

* store: finish the review's names test list

* store: restore the base registration tests

* store: pass link hosts that do not resolve here

* store: refuse a move once its sends are used up

* store: take the shared checkout at the provider

* store: hold a search slot for chain reads only

* store: state which refunds are safe on a reference

* store: check a failed order's name before refusing its move (#8060)

* store: test and document the shared checkout cap for name orders

* store: check a failed order's name on chain before refusing its move

* store: test name links whose host lookup fails

* store: test that a search holds its slot through the chain reads only

* store: restore the signed transaction check in the chain tests

* store: remove the chain tests' copy of the signed transaction check

* store: correct the search slot texts and the README's balance endpoints

* docs: system and sequence diagrams of the names service (#8068)

---------

Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: brenzi-agent <brenzi-agent@brenzi.ch>
Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
2026-11-07 10:45:19 +00:00

198 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CI on GitHub's runners
Three workflows build the apps from a branch's own code and test them, only when asked: `ios-sim` (the iOS app in a simulator on a macOS runner), `desktop-linux` (the desktop app on a Linux runner) and `android-emu` (the Android app in an emulator on a Linux runner). Each runs a scripted smoke test and, on request, a Claude session that follows a test plan, and a test backend (relay, directory, names server, peers). Details: [iOS](ios-sim/README.md), [desktop](desktop-linux/README.md), [Android](android/README.md).
## Reduced build
GitHub runs the reduced build of `build.yml` on a dispatch (workflow_dispatch), on each push to a draft pull request and on a push to master or stable in a fork. Once the pull request is marked ready for review, GitHub runs the full matrix, as on a push to master or stable in simplex-chat/simplex-chat.
On a push to master or stable in a fork, the release steps and tests are skipped; build-linux saves the -O0 build caches, and on master the Docker image cache, that the fork's dispatches restore.
On a draft pull request, the tests are selected by a fenced block with the info string `ci-tests` in the description, one hspec match pattern per line; the block is read at each push. `/` runs the whole suite; without a block, the reduced build runs no tests.
### Dispatch
```sh
curl -fsS -X POST \
-H "Authorization: Bearer $GH_TOKEN" \
-H "Accept: application/vnd.github+json" \
https://api.github.com/repos/<owner>/simplex-chat/actions/workflows/build.yml/dispatches \
-d '{"ref": "<branch>", "inputs": {"tests": "/SimpleX chat client/group tests/\n/Message batching/"}}'
```
Endpoint: https://docs.github.com/en/rest/actions/workflows#create-a-workflow-dispatch-event. Inputs:
| Input | Default | Value |
|---|---|---|
| `tests` | empty | hspec match patterns, one per line (`\n` in JSON); `/` runs the whole suite, and an empty value builds without tests |
| `base` | `master` | the branch that the changes are compared with:<br>`stable` for a branch cut from stable<br>the parent branch for a stacked branch |
When a newer dispatch on the same branch starts build-linux or build-ios, GitHub cancels the unfinished job of the older one (`cancel-in-progress`).
Results:
- Run and jobs: `GET /repos/<owner>/simplex-chat/actions/runs?head_sha=<commit>&event=workflow_dispatch` returns the run id; `GET /repos/<owner>/simplex-chat/actions/runs/<run id>/jobs` returns each step with its start, end and conclusion.
- Job log: `GET /repos/<owner>/simplex-chat/actions/jobs/<job id>/logs` returns a redirect to a text file, valid for 1 minute (https://docs.github.com/en/rest/actions/workflow-jobs#download-job-logs-for-a-workflow-run). The test summary is printed as "Finished in N seconds" and "N examples, N failures".
### Scopes
The reduced build runs one Linux job, build-linux (ubuntu-22.04 x86_64), for every scope except ios, and one macOS job, build-ios (macos-latest), for ios.
#### core
- Globs: `BUILD_CHANGES["haskell"]` in packages_release.py.
- CI, in build-linux:
- Docker image (step "Build and cache Docker image");
- caches (the steps "Restore ... from cache");
- `cabal build -j --enable-tests --ghc-options=-O0` (step "Build CLI");
- simplex-chat-test with the selected tests (step "Run tests"), each attempt limited to 35 minutes by `timeout`; an attempt that reaches the limit fails, and "Attempt <n> timed out." is printed.
- Local:
```sh
cp scripts/cabal.project.local.linux cabal.project.local
cabal build --enable-tests --ghc-options=-O0
mkdir -p tests/tmp
cabal test simplex-chat-test --ghc-options=-O0 --test-option=-m --test-option='/SimpleX chat client/group tests/'
```
The selection runs as in CI with scripts/ci/run-tests.sh, which appends the patterns of `TESTS` to the test binary:
```sh
TESTS='/SimpleX chat client/group tests/' scripts/ci/run-tests.sh "$(cabal list-bin simplex-chat-test --ghc-options=-O0)"
```
`--ghc-options=-O0` matches the reduced build; the library and test-suite stanzas of simplex-chat.cabal set -O2. Test directories are created under tests/tmp (`tmpBracket` in tests/Test.hs).
#### kotlin
- Globs: `BUILD_CHANGES["kotlin"]` in packages_release.py.
- CI: Gradle in the Docker container of build-linux (step "Build and test Kotlin apps", matrix field `kotlin`).
- Local:
```sh
cd apps/multiplatform
./gradlew --no-daemon --max-workers=1 :common:desktopTest :desktop:compileKotlinJvm -x :desktop:cmakeBuildAndCopy :android:compileGoogleDebugKotlin :android:testFossDebugUnitTest
```
JDK 17 and an Android SDK in ANDROID_HOME are required, as installed by Dockerfile.build.
#### ios
- Globs:
- apps/ios/**
- scripts/ci/ios-stub-libs.sh
- CI: build-ios: stub libraries, then xcodebuild for the iOS Simulator without signing.
- Local, on macOS: `scripts/ci/ios-stub-libs.sh`, then the xcodebuild command of the step "Build iOS app" of build-ios.
#### packages
- Globs:
- packages/simplex-chat-nodejs/**
- packages/simplex-chat-client/types/typescript/**
- packages/simplex-chat-python/**
- scripts/ci/package-tests.sh
- CI, in build-linux:
- Node 22 and Python 3.12 (steps "Setup Node.js" and "Setup Python");
- libsimplex, when `CORE_SOURCES` of packages_release.py also changed (step "Build library");
- `package-tests.sh build linux-x86_64` and `package-tests.sh test` (step "Package tests (pull request)").
- With libsimplex built, SIMPLEX_LIBS_DIR is set to it for the nodejs tests; the python tests run with it unset (package-tests.sh). Without SIMPLEX_LIBS_DIR, the pinned release libraries are downloaded (packages/simplex-chat-nodejs/src/libs.ts).
- Local:
```sh
scripts/ci/package-tests.sh build linux-x86_64
scripts/ci/package-tests.sh test
```
#### badge-web
- Globs: apps/simplex-store-service/web/**. These paths are also in the core globs (apps/simplex-store-service/**), and web/src/catalog.ts is read by core tests (tests/Bots/StoreService/CatalogTests.hs, WebTests.hs).
- CI, in build-linux: core build and tests, then `scope-tests.sh badge-web` (step "Test changed scopes").
- Selected core tests: `/SimpleX store service/`, which includes `/SimpleX store service e2e/`.
- Local:
```sh
cd apps/simplex-store-service/web
npm ci
npm test
```
#### ci-scripts
- Globs:
- scripts/ci/packages_release.py
- scripts/ci/test_packages_release.py
- CI, in build-linux: `scope-tests.sh ci-scripts` (step "Test changed scopes").
- Local: `python3 scripts/ci/test_packages_release.py`; Python 3.11 or newer is required for tomllib (packages_release.py).
#### store service on Postgres
- Job: test-store-postgres (ubuntu-22.04 x86_64), on a pull request that changes the store service, its tests, `tests/Test.hs`, `src/Simplex/Chat/PaymentService`, `src/Simplex/Chat/Badges`, `cabal.project`, `simplex-chat.cabal` or build.yml, and on every dispatch. Production runs the store service on Postgres only, and the SQLite jobs do not cover it.
- CI: the build-linux Docker image, a `postgres:17` container on the host network, `cabal build` with `+client_postgres`, then simplex-chat-test with `--fail-on=empty -m "Supporter badges" -m "SimpleX store service" -m "StoreService names"`.
- Local, with a Postgres 17 on 127.0.0.1 that trusts the `postgres` user and allows 1500 connections:
```sh
cabal build --builddir dist-postgres --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' simplex-chat-test
mkdir -p tests/tmp
PGHOST=127.0.0.1 LANG=C.UTF-8 "$(cabal list-bin --builddir dist-postgres --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' simplex-chat-test)" -m "Supporter badges" -m "SimpleX store service" -m "StoreService names"
```
#### Scopes lacking CI tests
| Scope | Paths | CI |
|---|---|---|
| ios | apps/ios/** | compile only |
| types package | packages/simplex-chat-client/types/typescript/** | compile only |
| website | website/**<br>images/**<br>blog/**<br>docs/**<br>PRIVACY.md | build and deploy on master pushes (web.yml) |
| ts-client | packages/simplex-chat-client/typescript/** | - |
| webrtc | packages/simplex-chat-webrtc/** | - |
| mcp | packages/simplex-chat-mcp/** | - |
| calculator-bot | apps/simplex-calculator-bot/** | - |
| support-bot | apps/simplex-support-bot/** | - |
| support-bot-light | apps/simplex-support-bot-light/** | - |
| prices | scripts/prices/** | - |
| nft | eth/nft/** | - |
CI coverage of such a scope is added in the pull request that changes it (Adding CI coverage).
### Adding CI coverage
For a scope lacking CI tests, in the pull request that changes it:
1. scripts/ci/packages_release.py: an entry `"<scope>": ["<glob>", ...]` in SCOPE_CHANGES.
2. build.yml, `on.pull_request.paths`: the same globs; the two are compared in `test_build_yml_runs_for_the_paths_of_its_builds` of test_packages_release.py.
3. scripts/ci/scope-tests.sh: a `<scope>)` case with the build and test commands, and the scope name in the usage line.
4. scripts/ci/test_packages_release.py, ChangesTest: a case asserting that the scope is printed in `scopes` for a change to one of its files.
5. This file, "Scopes": a subsection for the scope, in place of its row in "Scopes lacking CI tests".
6. Checks: `python3 scripts/ci/test_packages_release.py`, the new case of scope-tests.sh locally, then a dispatch of the reduced build.
The scope commands run in build-linux after the core tests, with Node 22 and Python 3.12 (steps "Setup Node.js" and "Setup Python"). For a scope that needs other versions, a setup step is added before "Test changed scopes", conditioned on the scope name in `needs.maybe-release.outputs.scopes`. A scope that needs macOS is a BUILD_CHANGES entry with a separate maybe-release output, as ios is, and a step in build-ios; the `if` of build-ios includes that output.
## Starting a run
Pushes start none of the app workflows: a run is started by hand, on any branch, with inputs that are off by default and combine.
| Input | iOS | Desktop | Android |
|---|---|---|---|
| no input | scripted test, the app uploaded | scripted test, the app uploaded | scripted test, the app uploaded |
| `claude` | scripted test, then Claude | scripted test, then Claude | scripted test, then Claude |
| `backend` | also the test backend | also the test backend | also the test backend |
| `release` | — | also the latest release's onboarding, to tell a branch's failure from the runner's | also the latest release's onboarding, to tell a branch's failure from the runner's |
| `armv7a` | — | — | also the core library for 32-bit Arm (armeabi-v7a), built only |
In the Actions tab: `ios-sim`, `desktop-linux` or `android-emu` → Run workflow, choose the branch and tick the inputs. With the GitHub CLI, each input is a field:
```sh
gh workflow run ios-sim.yml --ref <branch>
gh workflow run ios-sim.yml --ref <branch> -f claude=true
gh workflow run desktop-linux.yml --ref <branch> -f claude=true -f backend=true -f release=true
gh workflow run android-emu.yml --ref <branch> -f claude=true
```
A newer run of a workflow on the same branch cancels the older one.
Each run uploads the app it built, for manual tests, as an artifact kept 14 days: `ios-sim-app` (SimpleX.app.tar.gz, for the iOS simulator on an Apple-silicon Mac), `desktop-app` (desktop-app.tar.gz, for Linux x86-64) and `android-app` (the debug APK, arm64-v8a only). The run's page lists it under Artifacts; `gh run download <run id> -n <artifact>` downloads it.
An agent that runs these workflows in a fork follows [app-tests-agent.md](app-tests-agent.md).
## How long a run takes
| Run | iOS | Desktop | Android |
|---|---|---|---|
| Scripted test, builds reused | about 25 min, plus the wait for a macOS runner, which the team's builds share (minutes to an hour) | about 10 min | about 15 min |
| The core rebuilt, after Haskell changes | 30–45 min more | about 25 min more | about 27 min more |
| With the test backend | 20–45 min more when it is rebuilt, then about 10 min to start | about 30 min more when it is rebuilt, then a few minutes to start | as desktop, whose backend it shares |
| With Claude on the smoke plan | 10–15 min more | about 4 min more | about 5 min more |
## The Claude job
It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. The environment's branch rule alone decides which branches may use it: on a branch it doesn't allow, a run that asks for Claude fails at the Claude job's environment check, and the scripted test runs as usual. Each run spends the subscription's usage limits; the job summary gives its tokens and cost estimated at API prices.