mirror of
https://github.com/simplex-chat/simplex-chat.git
synced 2026-11-16 17:28:23 +00:00
* core: add prepaid name codes and length bounds * store: run the names migration after badges * store: issue and revoke prepaid name codes * store: document issuing prepaid name codes * store: move the ABI helpers to Names.Abi * store: read the [names] and [chain] settings * store: add the names chain client and reads * store: refuse group name codes while names are off * store: search names and preview links * store: serve each site on its own port * scripts: deploy the badges and names sites * store: document search and the two sites * store: add the drop_taken_codes migration * store: read the names codes_only setting * store: add the names store and order signatures * store: take name orders by card, crypto or code * store: document name orders and codes-only * store: add the names transaction sender * store: commit and register queued orders * store: reveal and retry an owner's order * store: document the gas key * store: add the name balance migrations * store: read crypto units, rates and confirmations * store: keep a taken order's money as a balance * store: spend a balance on another name * store: record refunds with //refund * store: document balances and refunds * store: add the agreed terms migration * store: read the terms and community names * store: answer community and terms in search * store: sign and store the agreed terms * store: document the terms and community names * store: add the invoiced units migration * store: read the crypto an invoice asks * store: credit late and overpaid crypto by share * store: document crediting late crypto by share * store: add the record changes migration * store: find an owner's orders and move one * store: relay owners' link edits on chain * store: document link edits * store: move isLoopbackHost to Config * store: trust dev issuer keys on the local chain * scripts: build the names site from a submodule * scripts: add the dev setup and contract run * store: document the dev setup and names image * store: send a Stripe idempotency key for names * tests: wait until a relay lets go of its edit * store: refuse a link edit deadline under 10 min * store: count reverted link edits in the year limit * store: require https for a remote chain node * store: refuse zero chain confirmations * store: set up the dev stack from the names branch * store: rerun a failed unpaid cleanup next pass * store: never replace a succeeded Stripe intent * store: refuse an ini setting written as key: value * store: note the webapp submodule's pinned branch * store: document confirmLanded judging by ownerOf * store: update the names webapp submodule * store-service: register names under the name policy, every name accepting its terms on chain (#7889) * store: accept each name's terms on chain under the name policy * ci: run the reduced build on sh/names-deploy * store: address review: policy re-read, terms reads and startup log * store: never credit an unconfirmed extra * store: alert money sent to a paid name invoice * store: refuse to start on another chain's node * store: keep orders while the allowance is used up * store: replace a stalled transaction after 3 blocks * store: judge a name taken at confirmation depth * store: bind the template's listener to loopback * store: scan only the ids of active name orders * store: refuse unknown [names] and [chain] settings * store: fetch the webapp submodule in upgrade steps * store: delete definitions only tests use * store: lock only name invoices at settlement * store: retry an unexpired commitment as usual * store: share the helpers defined twice * store: test route limits on the running service * store: fix tests that could not fail * ci: test the store service on Postgres * store: count IPv6 clients by their /64 * store: sweep invoices read alone past a list cap * store: cap checkouts of all clients together * store: cap the long polls held at once * store: stop the old badge project at upgrade * store: keep search lookups free of previews * store: read the ini as UTF-8 in any locale * ci: run Postgres store tests in a UTF-8 locale * store: cap the transactions one order sends * store: cancel a card intent after three declines * store: refuse a name price under a cent * store: send each transaction with gas headroom * store: cap searches and read terms once per block * store: never log the group join link * store: mask the database password at startup * store: run one instance per database * store: limit name codes to admins, 100 a day * store: refuse HighSpeed settlement with names * store: no unverified receipts with a trusted key * store: cap the log lines of refused webhooks * store: check Apple certificates at the signed date * store: refuse a malformed Stripe publishable key * store: check the owner before a refund * store: fold the names migrations into one * store: read onion name links through the proxy * store: refund money after paid once confirmed * store: limit the allowance wait to a dry run * store: bind loopback when upgrading * ci: cache and widen the store Postgres job * store: fix stale and duplicate names tests * store: fix the submodule path in the upgrade * store: refuse unpaid name codes on the CLI * store: apply the review's names test list * store: finish the review's names test list * store: restore the base registration tests * store: pass link hosts that do not resolve here * store: refuse a move once its sends are used up * store: take the shared checkout at the provider * store: hold a search slot for chain reads only * store: state which refunds are safe on a reference * store: check a failed order's name before refusing its move (#8060) * store: test and document the shared checkout cap for name orders * store: check a failed order's name on chain before refusing its move * store: test name links whose host lookup fails * store: test that a search holds its slot through the chain reads only * store: restore the signed transaction check in the chain tests * store: remove the chain tests' copy of the signed transaction check * store: correct the search slot texts and the README's balance endpoints * docs: system and sequence diagrams of the names service (#8068) --------- Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com> Co-authored-by: brenzi-agent <brenzi-agent@brenzi.ch> Co-authored-by: Evgeny @ SimpleX Chat <259188159+evgeny-simplex@users.noreply.github.com>
198 lines
12 KiB
Markdown
198 lines
12 KiB
Markdown
# CI on GitHub's runners
|
||
|
||
Three workflows build the apps from a branch's own code and test them, only when asked: `ios-sim` (the iOS app in a simulator on a macOS runner), `desktop-linux` (the desktop app on a Linux runner) and `android-emu` (the Android app in an emulator on a Linux runner). Each runs a scripted smoke test and, on request, a Claude session that follows a test plan, and a test backend (relay, directory, names server, peers). Details: [iOS](ios-sim/README.md), [desktop](desktop-linux/README.md), [Android](android/README.md).
|
||
|
||
## Reduced build
|
||
|
||
GitHub runs the reduced build of `build.yml` on a dispatch (workflow_dispatch), on each push to a draft pull request and on a push to master or stable in a fork. Once the pull request is marked ready for review, GitHub runs the full matrix, as on a push to master or stable in simplex-chat/simplex-chat.
|
||
|
||
On a push to master or stable in a fork, the release steps and tests are skipped; build-linux saves the -O0 build caches, and on master the Docker image cache, that the fork's dispatches restore.
|
||
|
||
On a draft pull request, the tests are selected by a fenced block with the info string `ci-tests` in the description, one hspec match pattern per line; the block is read at each push. `/` runs the whole suite; without a block, the reduced build runs no tests.
|
||
|
||
### Dispatch
|
||
|
||
```sh
|
||
curl -fsS -X POST \
|
||
-H "Authorization: Bearer $GH_TOKEN" \
|
||
-H "Accept: application/vnd.github+json" \
|
||
https://api.github.com/repos/<owner>/simplex-chat/actions/workflows/build.yml/dispatches \
|
||
-d '{"ref": "<branch>", "inputs": {"tests": "/SimpleX chat client/group tests/\n/Message batching/"}}'
|
||
```
|
||
|
||
Endpoint: https://docs.github.com/en/rest/actions/workflows#create-a-workflow-dispatch-event. Inputs:
|
||
|
||
| Input | Default | Value |
|
||
|---|---|---|
|
||
| `tests` | empty | hspec match patterns, one per line (`\n` in JSON); `/` runs the whole suite, and an empty value builds without tests |
|
||
| `base` | `master` | the branch that the changes are compared with:<br>`stable` for a branch cut from stable<br>the parent branch for a stacked branch |
|
||
|
||
When a newer dispatch on the same branch starts build-linux or build-ios, GitHub cancels the unfinished job of the older one (`cancel-in-progress`).
|
||
|
||
Results:
|
||
- Run and jobs: `GET /repos/<owner>/simplex-chat/actions/runs?head_sha=<commit>&event=workflow_dispatch` returns the run id; `GET /repos/<owner>/simplex-chat/actions/runs/<run id>/jobs` returns each step with its start, end and conclusion.
|
||
- Job log: `GET /repos/<owner>/simplex-chat/actions/jobs/<job id>/logs` returns a redirect to a text file, valid for 1 minute (https://docs.github.com/en/rest/actions/workflow-jobs#download-job-logs-for-a-workflow-run). The test summary is printed as "Finished in N seconds" and "N examples, N failures".
|
||
|
||
### Scopes
|
||
|
||
The reduced build runs one Linux job, build-linux (ubuntu-22.04 x86_64), for every scope except ios, and one macOS job, build-ios (macos-latest), for ios.
|
||
|
||
#### core
|
||
|
||
- Globs: `BUILD_CHANGES["haskell"]` in packages_release.py.
|
||
- CI, in build-linux:
|
||
- Docker image (step "Build and cache Docker image");
|
||
- caches (the steps "Restore ... from cache");
|
||
- `cabal build -j --enable-tests --ghc-options=-O0` (step "Build CLI");
|
||
- simplex-chat-test with the selected tests (step "Run tests"), each attempt limited to 35 minutes by `timeout`; an attempt that reaches the limit fails, and "Attempt <n> timed out." is printed.
|
||
- Local:
|
||
```sh
|
||
cp scripts/cabal.project.local.linux cabal.project.local
|
||
cabal build --enable-tests --ghc-options=-O0
|
||
mkdir -p tests/tmp
|
||
cabal test simplex-chat-test --ghc-options=-O0 --test-option=-m --test-option='/SimpleX chat client/group tests/'
|
||
```
|
||
The selection runs as in CI with scripts/ci/run-tests.sh, which appends the patterns of `TESTS` to the test binary:
|
||
```sh
|
||
TESTS='/SimpleX chat client/group tests/' scripts/ci/run-tests.sh "$(cabal list-bin simplex-chat-test --ghc-options=-O0)"
|
||
```
|
||
`--ghc-options=-O0` matches the reduced build; the library and test-suite stanzas of simplex-chat.cabal set -O2. Test directories are created under tests/tmp (`tmpBracket` in tests/Test.hs).
|
||
|
||
#### kotlin
|
||
|
||
- Globs: `BUILD_CHANGES["kotlin"]` in packages_release.py.
|
||
- CI: Gradle in the Docker container of build-linux (step "Build and test Kotlin apps", matrix field `kotlin`).
|
||
- Local:
|
||
```sh
|
||
cd apps/multiplatform
|
||
./gradlew --no-daemon --max-workers=1 :common:desktopTest :desktop:compileKotlinJvm -x :desktop:cmakeBuildAndCopy :android:compileGoogleDebugKotlin :android:testFossDebugUnitTest
|
||
```
|
||
JDK 17 and an Android SDK in ANDROID_HOME are required, as installed by Dockerfile.build.
|
||
|
||
#### ios
|
||
|
||
- Globs:
|
||
- apps/ios/**
|
||
- scripts/ci/ios-stub-libs.sh
|
||
- CI: build-ios: stub libraries, then xcodebuild for the iOS Simulator without signing.
|
||
- Local, on macOS: `scripts/ci/ios-stub-libs.sh`, then the xcodebuild command of the step "Build iOS app" of build-ios.
|
||
|
||
#### packages
|
||
|
||
- Globs:
|
||
- packages/simplex-chat-nodejs/**
|
||
- packages/simplex-chat-client/types/typescript/**
|
||
- packages/simplex-chat-python/**
|
||
- scripts/ci/package-tests.sh
|
||
- CI, in build-linux:
|
||
- Node 22 and Python 3.12 (steps "Setup Node.js" and "Setup Python");
|
||
- libsimplex, when `CORE_SOURCES` of packages_release.py also changed (step "Build library");
|
||
- `package-tests.sh build linux-x86_64` and `package-tests.sh test` (step "Package tests (pull request)").
|
||
- With libsimplex built, SIMPLEX_LIBS_DIR is set to it for the nodejs tests; the python tests run with it unset (package-tests.sh). Without SIMPLEX_LIBS_DIR, the pinned release libraries are downloaded (packages/simplex-chat-nodejs/src/libs.ts).
|
||
- Local:
|
||
```sh
|
||
scripts/ci/package-tests.sh build linux-x86_64
|
||
scripts/ci/package-tests.sh test
|
||
```
|
||
|
||
#### badge-web
|
||
|
||
- Globs: apps/simplex-store-service/web/**. These paths are also in the core globs (apps/simplex-store-service/**), and web/src/catalog.ts is read by core tests (tests/Bots/StoreService/CatalogTests.hs, WebTests.hs).
|
||
- CI, in build-linux: core build and tests, then `scope-tests.sh badge-web` (step "Test changed scopes").
|
||
- Selected core tests: `/SimpleX store service/`, which includes `/SimpleX store service e2e/`.
|
||
- Local:
|
||
```sh
|
||
cd apps/simplex-store-service/web
|
||
npm ci
|
||
npm test
|
||
```
|
||
|
||
#### ci-scripts
|
||
|
||
- Globs:
|
||
- scripts/ci/packages_release.py
|
||
- scripts/ci/test_packages_release.py
|
||
- CI, in build-linux: `scope-tests.sh ci-scripts` (step "Test changed scopes").
|
||
- Local: `python3 scripts/ci/test_packages_release.py`; Python 3.11 or newer is required for tomllib (packages_release.py).
|
||
|
||
#### store service on Postgres
|
||
|
||
- Job: test-store-postgres (ubuntu-22.04 x86_64), on a pull request that changes the store service, its tests, `tests/Test.hs`, `src/Simplex/Chat/PaymentService`, `src/Simplex/Chat/Badges`, `cabal.project`, `simplex-chat.cabal` or build.yml, and on every dispatch. Production runs the store service on Postgres only, and the SQLite jobs do not cover it.
|
||
- CI: the build-linux Docker image, a `postgres:17` container on the host network, `cabal build` with `+client_postgres`, then simplex-chat-test with `--fail-on=empty -m "Supporter badges" -m "SimpleX store service" -m "StoreService names"`.
|
||
- Local, with a Postgres 17 on 127.0.0.1 that trusts the `postgres` user and allows 1500 connections:
|
||
```sh
|
||
cabal build --builddir dist-postgres --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' simplex-chat-test
|
||
mkdir -p tests/tmp
|
||
PGHOST=127.0.0.1 LANG=C.UTF-8 "$(cabal list-bin --builddir dist-postgres --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' simplex-chat-test)" -m "Supporter badges" -m "SimpleX store service" -m "StoreService names"
|
||
```
|
||
|
||
#### Scopes lacking CI tests
|
||
|
||
| Scope | Paths | CI |
|
||
|---|---|---|
|
||
| ios | apps/ios/** | compile only |
|
||
| types package | packages/simplex-chat-client/types/typescript/** | compile only |
|
||
| website | website/**<br>images/**<br>blog/**<br>docs/**<br>PRIVACY.md | build and deploy on master pushes (web.yml) |
|
||
| ts-client | packages/simplex-chat-client/typescript/** | - |
|
||
| webrtc | packages/simplex-chat-webrtc/** | - |
|
||
| mcp | packages/simplex-chat-mcp/** | - |
|
||
| calculator-bot | apps/simplex-calculator-bot/** | - |
|
||
| support-bot | apps/simplex-support-bot/** | - |
|
||
| support-bot-light | apps/simplex-support-bot-light/** | - |
|
||
| prices | scripts/prices/** | - |
|
||
| nft | eth/nft/** | - |
|
||
|
||
CI coverage of such a scope is added in the pull request that changes it (Adding CI coverage).
|
||
|
||
### Adding CI coverage
|
||
|
||
For a scope lacking CI tests, in the pull request that changes it:
|
||
1. scripts/ci/packages_release.py: an entry `"<scope>": ["<glob>", ...]` in SCOPE_CHANGES.
|
||
2. build.yml, `on.pull_request.paths`: the same globs; the two are compared in `test_build_yml_runs_for_the_paths_of_its_builds` of test_packages_release.py.
|
||
3. scripts/ci/scope-tests.sh: a `<scope>)` case with the build and test commands, and the scope name in the usage line.
|
||
4. scripts/ci/test_packages_release.py, ChangesTest: a case asserting that the scope is printed in `scopes` for a change to one of its files.
|
||
5. This file, "Scopes": a subsection for the scope, in place of its row in "Scopes lacking CI tests".
|
||
6. Checks: `python3 scripts/ci/test_packages_release.py`, the new case of scope-tests.sh locally, then a dispatch of the reduced build.
|
||
|
||
The scope commands run in build-linux after the core tests, with Node 22 and Python 3.12 (steps "Setup Node.js" and "Setup Python"). For a scope that needs other versions, a setup step is added before "Test changed scopes", conditioned on the scope name in `needs.maybe-release.outputs.scopes`. A scope that needs macOS is a BUILD_CHANGES entry with a separate maybe-release output, as ios is, and a step in build-ios; the `if` of build-ios includes that output.
|
||
|
||
## Starting a run
|
||
|
||
Pushes start none of the app workflows: a run is started by hand, on any branch, with inputs that are off by default and combine.
|
||
|
||
| Input | iOS | Desktop | Android |
|
||
|---|---|---|---|
|
||
| no input | scripted test, the app uploaded | scripted test, the app uploaded | scripted test, the app uploaded |
|
||
| `claude` | scripted test, then Claude | scripted test, then Claude | scripted test, then Claude |
|
||
| `backend` | also the test backend | also the test backend | also the test backend |
|
||
| `release` | — | also the latest release's onboarding, to tell a branch's failure from the runner's | also the latest release's onboarding, to tell a branch's failure from the runner's |
|
||
| `armv7a` | — | — | also the core library for 32-bit Arm (armeabi-v7a), built only |
|
||
|
||
In the Actions tab: `ios-sim`, `desktop-linux` or `android-emu` → Run workflow, choose the branch and tick the inputs. With the GitHub CLI, each input is a field:
|
||
|
||
```sh
|
||
gh workflow run ios-sim.yml --ref <branch>
|
||
gh workflow run ios-sim.yml --ref <branch> -f claude=true
|
||
gh workflow run desktop-linux.yml --ref <branch> -f claude=true -f backend=true -f release=true
|
||
gh workflow run android-emu.yml --ref <branch> -f claude=true
|
||
```
|
||
|
||
A newer run of a workflow on the same branch cancels the older one.
|
||
|
||
Each run uploads the app it built, for manual tests, as an artifact kept 14 days: `ios-sim-app` (SimpleX.app.tar.gz, for the iOS simulator on an Apple-silicon Mac), `desktop-app` (desktop-app.tar.gz, for Linux x86-64) and `android-app` (the debug APK, arm64-v8a only). The run's page lists it under Artifacts; `gh run download <run id> -n <artifact>` downloads it.
|
||
|
||
An agent that runs these workflows in a fork follows [app-tests-agent.md](app-tests-agent.md).
|
||
|
||
## How long a run takes
|
||
|
||
| Run | iOS | Desktop | Android |
|
||
|---|---|---|---|
|
||
| Scripted test, builds reused | about 25 min, plus the wait for a macOS runner, which the team's builds share (minutes to an hour) | about 10 min | about 15 min |
|
||
| The core rebuilt, after Haskell changes | 30–45 min more | about 25 min more | about 27 min more |
|
||
| With the test backend | 20–45 min more when it is rebuilt, then about 10 min to start | about 30 min more when it is rebuilt, then a few minutes to start | as desktop, whose backend it shares |
|
||
| With Claude on the smoke plan | 10–15 min more | about 4 min more | about 5 min more |
|
||
|
||
## The Claude job
|
||
|
||
It needs the GitHub environment `app-sim-claude` with the secret `CLAUDE_CODE_OAUTH_TOKEN`, a Claude subscription token. The environment's branch rule alone decides which branches may use it: on a branch it doesn't allow, a run that asks for Claude fails at the Claude job's environment check, and the scripted test runs as usual. Each run spends the subscription's usage limits; the job summary gives its tokens and cost estimated at API prices.
|