* core, ui: validate short link type, show contact info for long addresses and rejected contacts (#8184) * tests: short link type, QR code capacity and contact statuses * core: validate the type of short links * android, desktop: show contact info for addresses above QR code capacity * ui: show chats with rejected contact requests * tests: short link types with group, channel and contact data * core: validate short link type against link data * tests: fix short link data test * tests: group data in invitation link before its first use * tests: connect via address in contact profile with group link * core: connect via contact address only with address links * core: parse short link data once, by link type * core: types of short link data parser * core: test short link types and contact rejection, record app fixtures * android, desktop: test contact info, link alerts and rejected contacts * ios: test invalid link alerts and rejected contacts in the chat list * desktop: contact info tests find texts without app resources * ui: core responses of invalid link plans and a rejected contact request * desktop: contact info tests provide the app bar handler * core: normalise file and message keys in app test fixtures * core: normalise deletion times in app test fixtures * desktop: test that a QR code above capacity is not shared on tap * core: test relay link checks and known channel refresh from link data * libs: nodejs 7.1.4 (#8263) * nodejs: live test reads the updated item under the key of 7.1 and 7.2 cores * libs: nodejs 7.1.4 * mcp: session states coloured by what they need, green for working (#8265) * mcp: session states in the colours of what they need: green working, orange waiting in the chat, red idle * mcp: the space after waiting for you in the tests * mcp: a session with a background command shows working in the background * mcp: a readable README, and the roadmap lines of 0.3.0 (#8264) * mcp: readme by what the plugin does for its users, roadmap lines of 0.3.0 * mcp: almost every waiting state * mcp: reply, edit and react in one line * mcp: flat layout without the avatar note * mcp: sessions use flat layout * mcp: shorter lines on answers, the Main group and approvals * mcp: approvals line without the taps * mcp: live stream line * mcp: status line * mcp: shorter lines * mcp: 0.3.1 with simplex-chat 7.1.4 (#8266) * docs: testing approaches of the codebase (#8255) * docs: app changes come with app tests on the core's recorded responses (#8252) * ci: decide the library claim from the written releases (#8137) * ci: test the library claim when reads return the previous releases * ci: decide the library claim from the written releases * tests: record name fixtures after the channel join settles (#8250) * core, ci: update sqlcipher-simple to 0.4.19.0 and simplexmq (#8270) * update sqlcipher-simple to 0.4.19.0 * sqlcipher-simple from simplex-chat/sqlcipher-simple * update simplexmq * ci: restore source repositories only for the same cabal.project * ci: keep restored source checkouts only at their pinned commits * android, desktop, ios: copy markdown files with line breaks, and the whole file with one button (#8269) * android, desktop: test copying lines from the markdown text view * android, desktop: keep line breaks in text copied from markdown files * android, desktop: check the last copied text in the markdown view test * android, desktop, ios: button to copy the whole markdown file * android, desktop: test the markdown file copy button * android, desktop, ios: ask before copying markdown files over 128 KB * android, desktop, ios: title the markdown copy alerts File too large * ios: test the markdown file copy button * android, desktop, ios: preview the markdown file view and test the copy limit * android, desktop, ios: snapshot references of the markdown file view * docs: names conditions describe upgradeable registry contracts (#8277) * docs: names conditions describe upgradeable registry contracts * docs: names conditions without the immutability after stabilization --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> * ci: reproducible Linux builds of 7.1 releases (#8276) * ci: plan reproducible Linux builds of 7.1 releases * ci: reproduce Linux builds with the cabal store path of releases * ci: reproduce releases with the current reproduce script * ci: build in the image with the cabal directory of releases * ci: reproduce releases on the Ubuntu 24.04 runner * ci: correct the failure history and rollout in the plan --------- Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> * website: remove the webinar promotion (#8281) * website: remove the webinar invitation from the home page * website: redirect the livestream page to Wefunder * mcp server: recall the chat history in a new agent session (#8141) * mcp server: recall the chat history in a new agent session * docs: chat history in receive for a new agent session * mcp server: type the history page for the compiler * mcp server: recall the chat history after a compaction too * docs: history plan changes with the message helpers of master Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75 * docs: drop anchor remapping from the docs table of the history plan Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75 --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> * core: prevent stalled join of channel subscribers to a new relay (#8239) * core: save relay connection before requesting relay link data * core: share relay connection creation between join and retry * tests: connect to a re-added relay when its connection is saved late * tests: delay only the relay connection record, not the command * ci, tests: run all tests in CI (#8238) * bots, tests: run broadcast bot test in CI * tests: test moderation that arrives before the message * ci, tests: run all tests in CI * tests: update re-enabled tests, restore query plans, PostgreSQL schema from CI * tests: remove CI skip helpers no longer used * tests: expect XFTP AUTH error of cancelled standalone file * tests: update query plans from CI * tests: update query plans from CI * tests: expect group history and member contacts in async group test * tests: accept subscription summary after first events on restart * tests: make AUTH and client timeout direct tests deterministic * tests: keep bob's badge clock before renewal until the move back * tests: keep the quota of 2 messages only on the member's receive queue * tests: keep query plans files with repeated entries of one query * tests: update query plans from CI * tests: stop the host before a member requests to join with a badge * tests: remove repeated entries of one query from query plans files * tests: accept the refusal of an unknown remote host after its session code * ci: state when the query plans test fails * tests: skip multicast discovery on macOS CI runners * tests: accept a remote host started with one local address * tests: restore the client timeout test except its zero timeout * tests: fail when a remote host prints another line than its other addresses * ci: name the tests that overwrite generated files * tests: move the badge clock after the restoring device arms its wake * tests: keep both members on the quota server in the member inactivity test * tests: accept subscription summaries in any order in the async group test * tests: update query plans from CI * tests: quota of 2 only on bob's queues in member inactivity test * tests: connect to a deleted link after its queue is deleted * core: save relay connection before requesting relay link data * core: share relay connection creation between join and retry * tests: connect to a re-added relay when its connection is saved late * tests: delay only the relay connection record, not the command * tests: record API fixtures on SQLite only * core: update query plans * ci: run PostgreSQL tests with the steps of build-linux * desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts (#8288) * desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts * desktop: record time offsets and snapshots rendered with Skia m138 * desktop: name the time offset test, skip only a missing fc-cache, update plan --------- Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> * core, directory: stop cancelled actions instead of continuing them (#8279) * ci: retry npm install once after a failure (#8280) * core: query plans of the store group tests that master now verifies --------- Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com> Co-authored-by: sh-agents <justcheckingifitwillwork@gmail.com> Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com> Co-authored-by: brenzi-agent <brenzi-agent@brenzi.ch> Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
27 KiB
Testing
The testing approaches of this repository: what is checked, how a test is written and run, and where CI runs it. The CI scopes, the reduced build and the ci-tests block of a pull request: CI on GitHub's runners.
Tests for a change
| Change | Tests |
|---|---|
| Behaviour of the chat core: messages, groups, files, profiles, connections | chat scenario test |
| Encoding of a protocol message or a JSON type, markdown, a pure function of the core | unit test |
| Command, response, event or type of the core API | Bot API docs, regenerated app fixture, when an app processes it |
| Database migration | schema dumps of SQLite and PostgreSQL query plans |
| Directory service, store service, remote sessions | the suite of the area |
| App processing of a core response or event | app fixture, replayed in Kotlin and iOS tests |
| App screen | a preview, rendered by the desktop and iOS snapshot tests |
| Flow across app screens | a case of an app workflow |
| Kotlin or Swift code without views | desktopTest or commonTest SimpleX Unit Tests |
| Package, bot, website or CI script | the tests of its scope |
| Scope lacking CI tests | its CI coverage, in the same pull request (Adding CI coverage) |
Core
The hspec suite simplex-chat-test of simplex-chat.cabal, with the entry point tests/Test.hs. Building it, running it and selecting tests by hspec match patterns: CI guide, core. A pattern is a path of describe and it names, for example /SimpleX chat client/direct tests/.
- A new test module is added to
other-modulesof the test suite in simplex-chat.cabal, and its spec tomainof Test.hs. tests/tmpcontains a directory per test (tmpBracketin Test.hs) and is emptied at the start and the end of a run (withTmpFilesin ChatClient.hs).- With
CI=true, set on GitHub's runners, the tests markedxdescribe''andxit''are skipped, and on macOS those markedxitMacCI(ChatTests/Utils.hs); the tests markedxit'are skipped on Linux. - tests/fixtures: images and files sent in the tests, TLS credentials of the test servers, and the app fixtures in
api. - In the full matrix of build.yml, the whole suite runs on Linux x86_64, with GHC 9.6.3 and 8.10.7, and on macOS aarch64; on a draft pull request, the tests matched by its
ci-testsblock run. - Coverage:
-fhpcadded toghc-optionsof the test suite in simplex-chat.cabal.
Suites
| Pattern | Module | Checked |
|---|---|---|
/SimpleX chat client/ |
ChatTests.hs | chat scenarios: direct, forward, group, chat relay, local, file, profile and chat list tests |
/names tests/ |
ChatTests/Names.hs | SimpleX names, resolved by a local resolver (NameResolver.hs); SQLite only |
/SimpleX chat protocol/ |
ProtocolTests.hs | chat protocol messages, short link data, batch limits, preferences JSON |
/JSON Tests/ |
JSONTests.hs, JSONFixtures.hs | conversion of the Swift JSON form to the tagged form, with QuickCheck; JSON of markdown |
/SimpleX chat markdown/ |
MarkdownTests.hs | the markdown parser |
/SimpleX chat view/ |
ViewTests.hs | functions of the CLI view |
/Mobile API Tests/ |
MobileTests.hs | the API of the core library (src/Simplex/Chat/Mobile.hs) and its C functions; SQLite only |
/Message batching/ |
MessageBatching.hs | batches of messages and of relay deliveries |
/Valid names/ |
ValidNames.hs | display names made valid |
/Random servers/ |
RandomServers.hs | the random choice of preset servers |
/Operators/ |
OperatorTests.hs | server operators, usage conditions, validation of user servers |
/Member relations/ |
MemberRelationsTests.hs | member relation vectors |
/Image resize/ |
ImageTests.hs | image resizing |
/Supporter badges/ |
BadgeTests.hs | badge credentials and proofs |
/WebRTC encryption/ |
WebRTCTests.hs | encryption of call media frames; SQLite only |
/Remote session/ |
RemoteTests.hs | a desktop controlling a mobile: handshake, messages, files |
/SimpleX Directory service bot//SimpleX Directory names/ |
Bots/DirectoryTests.hs | the directory service with chat clients (withDirectoryService); names: SQLite only |
/SimpleX Broadcast bot/ |
Bots/BroadcastTests.hs | the broadcast bot; skipped when CI=true |
/SimpleX store service/ |
Bots/StoreService/*Tests.hs | configuration, web server and schema, catalog, groups, invoice waiters, payment providers, store receipts; the providers are local fakes (Fake*.hs) |
/SimpleX store service e2e/ |
Bots/StoreService/BotTests.hs, GroupIntegrationTests.hs | the store service bot with chat clients |
/Schema dump//Postgres schema dump//Bot API docs//Save query plans/ |
SchemaDump.hs, PostgresSchemaDump.hs, APIDocs.hs | committed files |
Chat scenario tests
-
Checked: chat cores of several clients connected through a test SMP server, by the terminal output of each client, and by queries of a client's database for a state absent from the output.
-
Pattern, registered with
it "<name>" testDirectMessagein the spec of its module:testDirectMessage :: HasCallStack => TestParams -> IO () testDirectMessage = testChat2 aliceProfile bobProfile $ \alice bob -> do connectUsers alice bob alice #> "@bob hello" bob <# "alice> hello" -
Helpers of ChatClient.hs and ChatTests/Utils.hs:
Helper Use testChat2…testChat5,testChatCfgOpts2clients with new databases withNewTestChat,withTestChatone client with a new database; a restart with its database withNewTestChatCfgOptsa client with a ChatConfigandChatOpts, such aschatHooksormarkRead = FalseconnectUsers,createGroup2,createGroup3a contact; a group of 2 or 3 members ##>,#>,senda command and its echo; a message and its sent item, time removed; a command alone <##,<#,<##.,.<##the next line; the next line, time removed; its prefix; its suffix <###,concurrentlyN_lines in any order; lines of several clients at once #$>a command and a function of its output line, as in ("/_get chat @2 count=100", chat, [...])withCCTransaction,shouldEventuallyReturna query of the client's database; a value polled for 20 s withXFTPServer,withSmpServer',withSmpServerAndNamesan XFTP server; another SMP server; an SMP server with a names resolver versionTestMatrix2,testCfgVPrev,testCfgV1clients of the previous and of the first supported versions withTestOutputthe terminal output of the clients printed -
Run:
/SimpleX chat client/<area> tests/, for example/SimpleX chat client/group tests/. -
CI: build-linux, step "Run tests".
-
Pitfalls:
- The output of every client is read to its end: a test fails when a client prints a line within 0.1 s after the test (
<//intestChatNand thewith…TestChat…brackets). - The lines of one client are read in order; the lines of several clients are awaited with
concurrently_orconcurrentlyN_. - A test fails when a line is awaited for 20 s (
getTermLine') and when it runs longer than 180 s (itof ChatTests/Utils.hs, imported in place of hspec's). - Each test has its own ports (
withPortBase): the ports 7001–7003 of server addresses inChatOptsare mapped to them (testPortsCfg), andsmpServerStr,smpServer2StrandxftpServerStrreturn the addresses for commands and expected lines. - One SMP server is started per test by
testBracketof Test.hs; XFTP and other servers are started by the test.
- The output of every client is read to its end: a test fails when a client prints a line within 0.1 s after the test (
Unit tests
- Checked: functions and encodings of the core, in the suites without clients.
- Pattern: a
Specofdescribeanditwith hspec expectations, as in ViewTests.hs and ProtocolTests.hs. - Pitfall: in MobileTests, the expected JSON is the Swift form on macOS with the
swiftflag and the tagged form otherwise; both forms are in JSONFixtures.hs.
Checks against committed files
Each of these tests writes its file, then compares the new text with the previous one: the first run after a change of the text fails, and the next passes. The rewritten file is reviewed and committed with the change.
| Pattern | Written | Source |
|---|---|---|
/Bot API docs/ |
bots/api/COMMANDS.md, EVENTS.md, TYPES.md packages/simplex-chat-client/types/typescript/src/.ts packages/simplex-chat-python/src/simplex_chat/types/_.py |
the API docs in bots/src/API/Docs (testGenerate in APIDocs.hs) |
/Schema dump/ |
src/Simplex/Chat/Store/SQLite/Migrations/chat_schema.sql | SQLite migrations, dumped with sqlite3 |
/Postgres schema dump/ |
src/Simplex/Chat/Store/Postgres/Migrations/chat_schema.sql | PostgreSQL migrations, dumped with pg_dump |
/Save query plans/ |
chat_query_plans.txt and agent_query_plans.txt in src/Simplex/Chat/Store/SQLite/Migrations | the queries recorded in the run (chatQueryStats, agentQueryStats) |
| recording tests | tests/fixtures/api | app fixtures |
- Bot API docs also fail on a command, response or event of the API that is neither documented in bots/src/API/Docs nor listed in
cliCommands,undocumentedCommands,undocumentedResponsesorundocumentedEvents, on a type of a documented one missing from Docs/Types.hs, and on a documented command without responses. They run with GHC 9.6 and later (MIN_VERSION_base(4,18,0)), in SQLite builds. - Schema dump also checks down migrations (exceptions in
skipComparisonForDownMigrations), STRICT tables, and the output of.lint fkey-indexesagainst chat_lint.sql, an empty file, so that the test fails for a foreign key without an index. - Save query plans is skipped when
CI=true. The plans are of the queries of that run only, so the files are committed from a local run of the whole suite.
PostgreSQL
-
Run:
cabal test simplex-chat-test --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' --ghc-options=-O0 --test-option=-m --test-option='/SimpleX chat client/'The flags
client_postgresof simplexmq and simplex-chat, as in scripts/desktop/build-lib-linux.sh;dbPostgresis defined for the tests. -
Needed: a PostgreSQL server that accepts the role
postgreswithout a password, andpg_dump. The roletest_chat_userand the databasetest_chat_dbare created at the start of a run and dropped at its end (withTestDBin Test.hs); each client has its own schema, dropped after each test (dropTestSchemasin ChatClient.hs). -
Left out of PostgreSQL builds: Schema dump, Bot API docs, WebRTC encryption, Mobile API Tests, both names suites, Save query plans.
-
CI: local runs only; build.yml builds the PostgreSQL library for releases (build-linux-postgres).
-
Pitfall: the Postgres schema dump calls
sed -i ''(BSD sed) unlessCI=true.
App fixtures
The core's responses and events are recorded by core tests in tests/fixtures/api and replayed by Kotlin and iOS tests in place of the core library, so that app code is tested on the output of the current core.
flowchart LR
CT["core test<br>tests/ChatTests"] -->|"apiFixture<br>eventsFixture"| F["tests/fixtures/api"]
F --> T["Kotlin desktopTest<br>iOS SimpleX Unit Tests"]
T -->|"recorded responses<br>in place of the core"| V["app functions and views"]
T -->|"recorded events"| P["processReceivedMsg"]
V --> M["ChatModel, alerts, notifications"]
P --> M
Recording
Helpers of ChatTests/Utils.hs, called in a chat scenario test:
| Helper | Written to tests/fixtures/api/<fixture>.json |
|---|---|
apiFixture fixture cc placeholders FixtureRequests {frCommands, frMarkdown, frUris} |
an object with commands, markdown and uris, keyed by the request: the response to each command, encoded by chatSendCmd of the core library's API; chatParseMarkdown of each text; chatParseUri of each link |
nameFixture fixture cc name |
apiFixture of /_connect plan 1 <name> and of the markdown of the name, in names/ |
eventsFixture fixture events steps |
an array of the events appended by recordItemEvent events, in eventHook of chatHooks, during steps: newChatItems and chatItemUpdated, encoded as chatRecvMsg encodes them |
nameMovedFixture |
a nameMoved event, built from the IDs of the named contacts and groups |
Examples: testPlanKnownNameAvailable in ChatTests/Names.hs, testDirectMessageUpdatePrevItem and testMissedCall in ChatTests/Direct.hs.
Stable recordings:
-
Values of each run are replaced by
stableJSON:Fields Replaced by createdAtupdatedAtchatTsuserMemberProfileSentAtitemTs1970-01-01T00:00:00ZagentConnIdxContactIdconnFullLinkconnShortLinkcontactLinkgroupLinkviaGroupLinkUrisimplexUrimemberIdmemberPubKeypublicGroupIdbusinessIdcustomerIditemSharedMsgIdthe field name groupRelays"groupRelay"per elementA field with a new value in each run is added to this list.
-
Strings of the test, such as links, are replaced by the
placeholdersofapiFixture, in keys and values:[(cLink, "CONTACT_LINK")]intestPlanAddressContactViaAddress, so that the app test connects toCONTACT_LINK. -
The recorded state is awaited first: the response to an update is recorded after the item status in the database is
snd_rcvd ok complete(shouldEventuallyReturnintestDirectMessageUpdatePrevItem), and the steps ofeventsFixtureend after the expected number of events (testMissedCall). -
The client whose events are recorded has
markRead = False: the CLI marks received items read when it prints them (runTerminalOutputin src/Simplex/Chat/Terminal/Output.hs), while the apps receive unread items. -
Recordings are in the tagged JSON form of builds without the
swiftflag.
Replay
| Kotlin, common/src/commonMain | iOS | |
|---|---|---|
| Commands | chatSendCmdRetry, platform/Core.kt |
sendSimpleXCmdStr, SimpleXChat/API.swift |
| Markdown | chatParseMarkdown, platform/Core.kt |
parseSimpleXMarkdownText, SimpleXChat/API.swift |
| Links | chatParseUri, platform/Core.kt |
parseSanitizeUri, SimpleXChat/API.swift |
| Events | ChatController.processReceivedMsg, model/SimpleXAPI.kt |
processReceivedMsg, Shared/Model/SimpleXAPI.swift |
| Notifications | ntfManager, platform/NtfManager.kt |
NtfManager.shared, Shared/Model/NtfManager.swift |
| Recording read by | fixtureText |
fixtureJSON, then singleFieldJSON to the Swift JSON form |
| Tests | NameAlertsTest.kt, ChatItemUpdateTest.kt | NameAlertsTests.swift, ChatItemUpdateTests.swift |
- Pattern:
recorded(fixture)in the NameAlerts tests sets the first three seams to lookups in the recording; the test fails at tear-down on a request without a recorded response, unless the request is checked withassertRequested. The ChatItemUpdate tests decode the recorded events, pass each toprocessReceivedMsgand compareChatModel. Tear-down restores the seams and clears the model. - Run and CI: Kotlin apps, SimpleX Unit Tests. CI runs the kotlin and ios scopes for a change in tests/fixtures/api (
BUILD_CHANGESin scripts/ci/packages_release.py). The recording tests are core tests, selected in theci-testsblock by the patterns of the CI guide, core. - Pitfalls:
singleFieldJSONconverts each object with atypefield, exceptMsgContent, recognized by its tags. The core encodes other types withtypein both JSON forms: the types oftaggedObjectJSONin src, such asMsgChatLink,LinkContent,MsgScopeandChatBotCommand, andMsgReaction, by its own instance. A recording with them requires another conversion rule.- The tests use the app's singletons, such as
ChatModelandAlertManager: each test restores what it changes. - The core is reached only through the replaced seams: the Kotlin test JVM does not load the core library, and the iOS app host in CI is linked with stub libraries.
Kotlin apps
In apps/multiplatform:
| Source set | Gradle task | Contents |
|---|---|---|
| common/src/commonTest | :common:desktopTest |
tests of common code (kotlin.test) |
| common/src/desktopTest | :common:desktopTest |
JVM tests: Compose views with runComposeUiTest, layout positions with renderScene and Modifier.record (LayoutRecorder.kt), files with withTempDir, app fixtures, snapshots |
| android/src/test | :android:testFossDebugUnitTest |
a template test, ExampleUnitTest |
| android/src/androidTest | connectedFossDebugAndroidTest, with a device or an emulator |
a template test, ExampleInstrumentedTest |
- Run:
./gradlew :common:desktopTest; one class:./gradlew :common:desktopTest --tests chat.simplex.app.ChatItemUpdateTest. The CI command: CI guide, kotlin. - CI: build.yml, step "Build and test Kotlin apps" (kotlin scope); desktop-linux, app job.
- Pitfalls:
- The working directory is apps/multiplatform/common: recordings are read from
../../../tests/fixtures/api, sources fromsrc/. - The tests use the app's singletons, such as
ChatModel,appPrefsandsimplexWindowState: they are restored in@AfterTest. - The test JVM does not load the core library: code that calls it is tested with the replaced seams of app fixtures.
- The working directory is apps/multiplatform/common: recordings are read from
Desktop snapshots
-
Checked: each
@Previewof commonMain and desktopMain, rendered in the light and dark themes bySnapshotTest(SnapshotTest.kt), equals its reference in common/src/desktopTest/snapshots, within 8 of 255 per colour channel (snapshots-compare.py). -
Write: a new
@Previewis added topreviewScenes, withchatItemFontScalesfor a chat item, or topreviewsCallingCorewhen its layout calls the core library;everyPreviewIsRenderedfails otherwise. A preview usespreviewDatein place of the current time. -
Record and check: with
scripts/ci/snapshots-remote.py desktop record, desktop-linux renders the previews of the pushed branch, and the renders replace the references, for review and commit; withcheck, the renders that differ are listed. -
Run, from apps/multiplatform, then from the repository root:
SNAPSHOT_DIR=<absolute dir> ./gradlew -Psimplex.assets.dir=../../assets :common:desktopTest --tests chat.simplex.app.SnapshotTest python3 scripts/ci/snapshots-compare.py apps/multiplatform/common/src/desktopTest/snapshots <absolute dir> <diff dir> -
CI: desktop-linux, step "Compare snapshots"; in build.yml, the previews are rendered without a comparison.
-
Details: desktop-linux README, "Snapshots".
iOS app
In apps/ios, SimpleX.xcodeproj:
| Target | Kind | Scheme | CI |
|---|---|---|---|
| SimpleX Unit Tests | unit tests hosted by the app | SimpleX Unit Tests | build.yml, build-ios, step "Run iOS unit tests" (ios scope) |
| SimpleX Snapshots | unit tests hosted by the app: preview renders and tests of a running chat | SimpleX (iOS) | ios-sim, after the smoke test |
| Tests iOS | UI tests of Xcode's template | SimpleX (iOS) | local runs only |
A new test file is added to its target in project.pbxproj: a PBXBuildFile, a PBXFileReference, the target's group and its Sources phase.
SimpleX Unit Tests
- Checked: app code on app fixtures and other logic of the app.
- Write: an
XCTestCasein apps/ios/SimpleX Unit Tests;fixtureJSON,singleFieldJSONandwaitForare at file scope of NameAlertsTests.swift. - Run, on macOS:
scripts/ci/ios-stub-libs.sh, then thexcodebuild testcommand of the step "Run iOS unit tests" of build-ios in build.yml; or Product > Test with the scheme SimpleX Unit Tests. - Pitfalls:
- The stubs replace the libraries in apps/ios/Libraries/sim, where scripts/ios/prepare.sh puts the core built for the simulator.
- With the stubs, the app opens its database error screen and calls the core no more.
- The tests run with
-testLanguage en -testRegion US; a test of dates setsNSTimeZone.defaultto UTC.
SimpleX Snapshots
- Checked: each
PreviewProviderof apps/ios, rendered in the light and dark themes, equals its reference in References, within 8 of 255 per colour channel; previews markedallSizesare rendered at three content sizes. SnapshotTests.swift also contains assertion tests of layout functions, calls and VoIP pushes; the VoIP push tests require a running chat (waitUntilChatActive). - Write: a new
PreviewProvideris added topreviewScenes(), or toEXCLUDEDof check-previews.py when its renders differ between runs; check-previews.py fails on a#Preview. A preview usespreviewDatein place of the current time. - Record and check:
scripts/ci/snapshots-remote.py ios record|check, as for desktop. - Run, from the repository root, with the core in apps/ios/Libraries/sim and a simulator where the app has a running chat:
SIM_UDID=<udid> scripts/ci/ios-sim/snapshots.sh <derived data> <out dir>(snapshots.sh). - CI: build-ios runs check-previews.py; ios-sim renders and compares at the end of the smoke test, whose step fails on a difference.
- Details: ios-sim README, "Snapshots".
Tests iOS
Xcode's UI test template: testExample launches the app, testLaunchPerformance measures its launch, testLaunch attaches a screenshot of the launch screen. It is in the test action of the scheme SimpleX (iOS), with SimpleX Snapshots: xcodebuild test -scheme "SimpleX (iOS)" of apps/ios/README.md runs both.
App workflows
The app built from a branch, its core included, driven on a device by scripts and, on request, by Claude following a test plan, with an optional test backend (relay, directory, names server, peers).
| Workflow | App | Smoke test | Claude plan | README |
|---|---|---|---|---|
ios-sim |
iOS app in an iPhone simulator | onboarding images sent in a group snapshots |
device-test-plan.md, cases per layout in claude-scope | ios-sim |
desktop-linux |
desktop app on a virtual display, Ubuntu | desktopTest with snapshots onboarding a local group both layouts images sent |
claude-test.md, claude-scope | desktop-linux |
android-emu |
debug APK in an Android emulator | onboarding a local group both layouts images sent |
claude-test.md, claude-scope | android |
- Write: a scripted case in the workflow's
smoke-test.sh, with itsuidriver; a Claude plan in the files of the table, withbackend: yesin claude-scope when the cases use the test backend. - Run: by hand only, on any branch: CI guide, Starting a run. Each run uploads the app for manual tests.
- Pitfalls:
- A run with
claudeandbackend: yesin claude-scope also requires the inputbackend, or its plan job fails. - A Claude run requires the branch rule of the GitHub environment
app-sim-claudeto include the branch. - The Android app runs under ARM translation: with the input
release, the released app runs in the same emulator, to tell an app failure from the emulator's.
- A run with
Packages and other scopes
| Scope | Tests | Run |
|---|---|---|
| packages: Node.js library | packages/simplex-chat-nodejs/tests, jest: unit tests *.unit.test.ts, with core functions mockedtests of the add-on and of library downloads live tests with a local smp-server ( describeIfSmpServerAvailable: SMP_SERVER, or a download on Linux) |
scripts/ci/package-tests.sh build linux-x86_64scripts/ci/package-tests.sh test |
| packages: Python library | packages/simplex-chat-python/tests, pytest, with a fake libsimplex (FakeLib) |
as above |
| packages: TypeScript types | generated by Bot API docs, compiled by package-tests.sh | as above |
| mcp | packages/simplex-chat-mcp/test, jest:test/unittest/e2e, with a local smp-server and xftp-servermanual checks: TESTING.md |
scripts/ci/scope-tests.sh mcp |
| support-bot | apps/simplex-support-bot/bot.test.ts, vitest, with simplex-chat and @simplex-chat/types replaced by test/mocks |
scripts/ci/scope-tests.sh support-bot |
| support-bot-light | apps/simplex-support-bot-light/tests, pytest, with FakeChatApi of conftest.py; ruff, pyright |
scripts/ci/scope-tests.sh support-bot-light |
| website | website/test, node --test of src/js/simplex-lib.jsc |
npm test in website |
| badge-web | apps/simplex-store-service/web/test, node --test of the compiled tests, in UTC |
npm ci, npm test in apps/simplex-store-service/web |
| ci-scripts | scripts/ci/test_packages_release.py, unittest of packages_release.py | python3 scripts/ci/test_packages_release.py |
CI of each scope: CI guide, Scopes. Scopes lacking CI tests, with tests run locally:
| Scope | Run | Notes |
|---|---|---|
| ts-client | npm test in packages/simplex-chat-client/typescript |
prettier, eslint and jest; client.test.ts is skipped |
| webrtc | npm test in packages/simplex-chat-webrtc |
fullstack.test.js requires SIMPLEX_CHAT and SMP_SERVER, the paths of the chat CLI and of smp-server |
| calculator-bot | npm test in apps/simplex-calculator-bot |
calculatorBot.test.ts: smp-server from SMP_SERVER or a download on Linux, and tests/fixtures/tls |