27 KiB
Testing
The testing approaches of this repository: what is checked, how a test is written and run, and where CI runs it. The CI scopes, the reduced build and the ci-tests block of a pull request: CI on GitHub's runners.
Tests for a change
| Change | Tests |
|---|---|
| Behaviour of the chat core: messages, groups, files, profiles, connections | chat scenario test |
| Encoding of a protocol message or a JSON type, markdown, a pure function of the core | unit test |
| Command, response, event or type of the core API | Bot API docs, regenerated app fixture, when an app processes it |
| Database migration | schema dumps of SQLite and PostgreSQL query plans |
| Directory service, badge service, remote sessions | the suite of the area |
| App processing of a core response or event | app fixture, replayed in Kotlin and iOS tests |
| App screen | a preview, rendered by the desktop and iOS snapshot tests |
| Flow across app screens | a case of an app workflow |
| Kotlin or Swift code without views | desktopTest or commonTest SimpleX Unit Tests |
| Package, bot, website or CI script | the tests of its scope |
| Scope lacking CI tests | its CI coverage, in the same pull request (Adding CI coverage) |
Core
The hspec suite simplex-chat-test of simplex-chat.cabal, with the entry point tests/Test.hs. Building it, running it and selecting tests by hspec match patterns: CI guide, core. A pattern is a path of describe and it names, for example /SimpleX chat client/direct tests/.
- A new test module is added to
other-modulesof the test suite in simplex-chat.cabal, and its spec tomainof Test.hs. tests/tmpcontains a directory per test (tmpBracketin Test.hs) and is emptied at the start and the end of a run (withTmpFilesin ChatClient.hs).- With
CI=true, set on GitHub's runners, the tests markedxdescribe''andxit''are skipped, and on macOS those markedxitMacCI(ChatTests/Utils.hs); the tests markedxit'are skipped on Linux. - tests/fixtures: images and files sent in the tests, TLS credentials of the test servers, and the app fixtures in
api. - In the full matrix of build.yml, the whole suite runs on Linux x86_64, with GHC 9.6.3 and 8.10.7, and on macOS aarch64; on a draft pull request, the tests matched by its
ci-testsblock run. - Coverage:
-fhpcadded toghc-optionsof the test suite in simplex-chat.cabal.
Suites
| Pattern | Module | Checked |
|---|---|---|
/SimpleX chat client/ |
ChatTests.hs | chat scenarios: direct, forward, group, chat relay, local, file, profile and chat list tests |
/names tests/ |
ChatTests/Names.hs | SimpleX names, resolved by a local resolver (NameResolver.hs); SQLite only |
/SimpleX chat protocol/ |
ProtocolTests.hs | chat protocol messages, short link data, batch limits, preferences JSON |
/JSON Tests/ |
JSONTests.hs, JSONFixtures.hs | conversion of the Swift JSON form to the tagged form, with QuickCheck; JSON of markdown |
/SimpleX chat markdown/ |
MarkdownTests.hs | the markdown parser |
/SimpleX chat view/ |
ViewTests.hs | functions of the CLI view |
/Mobile API Tests/ |
MobileTests.hs | the API of the core library (src/Simplex/Chat/Mobile.hs) and its C functions; SQLite only |
/Message batching/ |
MessageBatching.hs | batches of messages and of relay deliveries |
/Valid names/ |
ValidNames.hs | display names made valid |
/Random servers/ |
RandomServers.hs | the random choice of preset servers |
/Operators/ |
OperatorTests.hs | server operators, usage conditions, validation of user servers |
/Member relations/ |
MemberRelationsTests.hs | member relation vectors |
/Image resize/ |
ImageTests.hs | image resizing |
/Supporter badges/ |
BadgeTests.hs | badge credentials and proofs |
/WebRTC encryption/ |
WebRTCTests.hs | encryption of call media frames; SQLite only |
/Remote session/ |
RemoteTests.hs | a desktop controlling a mobile: handshake, messages, files |
/SimpleX Directory service bot//SimpleX Directory names/ |
Bots/DirectoryTests.hs | the directory service with chat clients (withDirectoryService); names: SQLite only |
/SimpleX Broadcast bot/ |
Bots/BroadcastTests.hs | the broadcast bot; skipped when CI=true |
/SimpleX badge service/ |
Bots/BadgeService/*Tests.hs | configuration, web server and schema, catalog, groups, invoice waiters, payment providers, store receipts; the providers are local fakes (Fake*.hs) |
/SimpleX badge service e2e/ |
Bots/BadgeService/BotTests.hs, GroupIntegrationTests.hs | the badge service bot with chat clients |
/Schema dump//Postgres schema dump//Bot API docs//Save query plans/ |
SchemaDump.hs, PostgresSchemaDump.hs, APIDocs.hs | committed files |
Chat scenario tests
-
Checked: chat cores of several clients connected through a test SMP server, by the terminal output of each client, and by queries of a client's database for a state absent from the output.
-
Pattern, registered with
it "<name>" testDirectMessagein the spec of its module:testDirectMessage :: HasCallStack => TestParams -> IO () testDirectMessage = testChat2 aliceProfile bobProfile $ \alice bob -> do connectUsers alice bob alice #> "@bob hello" bob <# "alice> hello" -
Helpers of ChatClient.hs and ChatTests/Utils.hs:
Helper Use testChat2…testChat5,testChatCfgOpts2clients with new databases withNewTestChat,withTestChatone client with a new database; a restart with its database withNewTestChatCfgOptsa client with a ChatConfigandChatOpts, such aschatHooksormarkRead = FalseconnectUsers,createGroup2,createGroup3a contact; a group of 2 or 3 members ##>,#>,senda command and its echo; a message and its sent item, time removed; a command alone <##,<#,<##.,.<##the next line; the next line, time removed; its prefix; its suffix <###,concurrentlyN_lines in any order; lines of several clients at once #$>a command and a function of its output line, as in ("/_get chat @2 count=100", chat, [...])withCCTransaction,shouldEventuallyReturna query of the client's database; a value polled for 20 s withXFTPServer,withSmpServer',withSmpServerAndNamesan XFTP server; another SMP server; an SMP server with a names resolver versionTestMatrix2,testCfgVPrev,testCfgV1clients of the previous and of the first supported versions withTestOutputthe terminal output of the clients printed -
Run:
/SimpleX chat client/<area> tests/, for example/SimpleX chat client/group tests/. -
CI: build-linux, step "Run tests".
-
Pitfalls:
- The output of every client is read to its end: a test fails when a client prints a line within 0.1 s after the test (
<//intestChatNand thewith…TestChat…brackets). - The lines of one client are read in order; the lines of several clients are awaited with
concurrently_orconcurrentlyN_. - A test fails when a line is awaited for 20 s (
getTermLine') and when it runs longer than 180 s (itof ChatTests/Utils.hs, imported in place of hspec's). - Each test has its own ports (
withPortBase): the ports 7001–7003 of server addresses inChatOptsare mapped to them (testPortsCfg), andsmpServerStr,smpServer2StrandxftpServerStrreturn the addresses for commands and expected lines. - One SMP server is started per test by
testBracketof Test.hs; XFTP and other servers are started by the test.
- The output of every client is read to its end: a test fails when a client prints a line within 0.1 s after the test (
Unit tests
- Checked: functions and encodings of the core, in the suites without clients.
- Pattern: a
Specofdescribeanditwith hspec expectations, as in ViewTests.hs and ProtocolTests.hs. - Pitfall: in MobileTests, the expected JSON is the Swift form on macOS with the
swiftflag and the tagged form otherwise; both forms are in JSONFixtures.hs.
Checks against committed files
Each of these tests writes its file, then compares the new text with the previous one: the first run after a change of the text fails, and the next passes. The rewritten file is reviewed and committed with the change.
| Pattern | Written | Source |
|---|---|---|
/Bot API docs/ |
bots/api/COMMANDS.md, EVENTS.md, TYPES.md packages/simplex-chat-client/types/typescript/src/.ts packages/simplex-chat-python/src/simplex_chat/types/_.py |
the API docs in bots/src/API/Docs (testGenerate in APIDocs.hs) |
/Schema dump/ |
src/Simplex/Chat/Store/SQLite/Migrations/chat_schema.sql | SQLite migrations, dumped with sqlite3 |
/Postgres schema dump/ |
src/Simplex/Chat/Store/Postgres/Migrations/chat_schema.sql | PostgreSQL migrations, dumped with pg_dump |
/Save query plans/ |
chat_query_plans.txt and agent_query_plans.txt in src/Simplex/Chat/Store/SQLite/Migrations | the queries recorded in the run (chatQueryStats, agentQueryStats) |
| recording tests | tests/fixtures/api | app fixtures |
- Bot API docs also fail on a command, response or event of the API that is neither documented in bots/src/API/Docs nor listed in
cliCommands,undocumentedCommands,undocumentedResponsesorundocumentedEvents, on a type of a documented one missing from Docs/Types.hs, and on a documented command without responses. They run with GHC 9.6 and later (MIN_VERSION_base(4,18,0)), in SQLite builds. - Schema dump also checks down migrations (exceptions in
skipComparisonForDownMigrations), STRICT tables, and the output of.lint fkey-indexesagainst chat_lint.sql, an empty file, so that the test fails for a foreign key without an index. - Save query plans is skipped when
CI=true. The plans are of the queries of that run only, so the files are committed from a local run of the whole suite.
PostgreSQL
-
Run:
cabal test simplex-chat-test --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' --ghc-options=-O0 --test-option=-m --test-option='/SimpleX chat client/'The flags
client_postgresof simplexmq and simplex-chat, as in scripts/desktop/build-lib-linux.sh;dbPostgresis defined for the tests. -
Needed: a PostgreSQL server that accepts the role
postgreswithout a password, andpg_dump. The roletest_chat_userand the databasetest_chat_dbare created at the start of a run and dropped at its end (withTestDBin Test.hs); each client has its own schema, dropped after each test (dropTestSchemasin ChatClient.hs). -
Left out of PostgreSQL builds: Schema dump, Bot API docs, WebRTC encryption, Mobile API Tests, both names suites, Save query plans.
-
CI: local runs only; build.yml builds the PostgreSQL library for releases (build-linux-postgres).
-
Pitfall: the Postgres schema dump calls
sed -i ''(BSD sed) unlessCI=true.
App fixtures
The core's responses and events are recorded by core tests in tests/fixtures/api and replayed by Kotlin and iOS tests in place of the core library, so that app code is tested on the output of the current core.
flowchart LR
CT["core test<br>tests/ChatTests"] -->|"apiFixture<br>eventsFixture"| F["tests/fixtures/api"]
F --> T["Kotlin desktopTest<br>iOS SimpleX Unit Tests"]
T -->|"recorded responses<br>in place of the core"| V["app functions and views"]
T -->|"recorded events"| P["processReceivedMsg"]
V --> M["ChatModel, alerts, notifications"]
P --> M
Recording
Helpers of ChatTests/Utils.hs, called in a chat scenario test:
| Helper | Written to tests/fixtures/api/<fixture>.json |
|---|---|
apiFixture fixture cc placeholders FixtureRequests {frCommands, frMarkdown, frUris} |
an object with commands, markdown and uris, keyed by the request: the response to each command, encoded by chatSendCmd of the core library's API; chatParseMarkdown of each text; chatParseUri of each link |
nameFixture fixture cc name |
apiFixture of /_connect plan 1 <name> and of the markdown of the name, in names/ |
eventsFixture fixture events steps |
an array of the events appended by recordItemEvent events, in eventHook of chatHooks, during steps: newChatItems and chatItemUpdated, encoded as chatRecvMsg encodes them |
nameMovedFixture |
a nameMoved event, built from the IDs of the named contacts and groups |
Examples: testPlanKnownNameAvailable in ChatTests/Names.hs, testDirectMessageUpdatePrevItem and testMissedCall in ChatTests/Direct.hs.
Stable recordings:
-
Values of each run are replaced by
stableJSON:Fields Replaced by createdAtupdatedAtchatTsuserMemberProfileSentAtitemTs1970-01-01T00:00:00ZagentConnIdxContactIdconnFullLinkconnShortLinkcontactLinkgroupLinkviaGroupLinkUrisimplexUrimemberIdmemberPubKeypublicGroupIdbusinessIdcustomerIditemSharedMsgIdthe field name groupRelays"groupRelay"per elementA field with a new value in each run is added to this list.
-
Strings of the test, such as links, are replaced by the
placeholdersofapiFixture, in keys and values:[(cLink, "CONTACT_LINK")]intestPlanAddressContactViaAddress, so that the app test connects toCONTACT_LINK. -
The recorded state is awaited first: the response to an update is recorded after the item status in the database is
snd_rcvd ok complete(shouldEventuallyReturnintestDirectMessageUpdatePrevItem), and the steps ofeventsFixtureend after the expected number of events (testMissedCall). -
The client whose events are recorded has
markRead = False: the CLI marks received items read when it prints them (runTerminalOutputin src/Simplex/Chat/Terminal/Output.hs), while the apps receive unread items. -
Recordings are in the tagged JSON form of builds without the
swiftflag.
Replay
| Kotlin, common/src/commonMain | iOS | |
|---|---|---|
| Commands | chatSendCmdRetry, platform/Core.kt |
sendSimpleXCmdStr, SimpleXChat/API.swift |
| Markdown | chatParseMarkdown, platform/Core.kt |
parseSimpleXMarkdownText, SimpleXChat/API.swift |
| Links | chatParseUri, platform/Core.kt |
parseSanitizeUri, SimpleXChat/API.swift |
| Events | ChatController.processReceivedMsg, model/SimpleXAPI.kt |
processReceivedMsg, Shared/Model/SimpleXAPI.swift |
| Notifications | ntfManager, platform/NtfManager.kt |
NtfManager.shared, Shared/Model/NtfManager.swift |
| Recording read by | fixtureText |
fixtureJSON, then singleFieldJSON to the Swift JSON form |
| Tests | NameAlertsTest.kt, ChatItemUpdateTest.kt | NameAlertsTests.swift, ChatItemUpdateTests.swift |
- Pattern:
recorded(fixture)in the NameAlerts tests sets the first three seams to lookups in the recording; the test fails at tear-down on a request without a recorded response, unless the request is checked withassertRequested. The ChatItemUpdate tests decode the recorded events, pass each toprocessReceivedMsgand compareChatModel. Tear-down restores the seams and clears the model. - Run and CI: Kotlin apps, SimpleX Unit Tests. CI runs the kotlin and ios scopes for a change in tests/fixtures/api (
BUILD_CHANGESin scripts/ci/packages_release.py). The recording tests are core tests, selected in theci-testsblock by the patterns of the CI guide, core. - Pitfalls:
singleFieldJSONconverts each object with atypefield, exceptMsgContent, recognized by its tags. The core encodes other types withtypein both JSON forms: the types oftaggedObjectJSONin src, such asMsgChatLink,LinkContent,MsgScopeandChatBotCommand, andMsgReaction, by its own instance. A recording with them requires another conversion rule.- The tests use the app's singletons, such as
ChatModelandAlertManager: each test restores what it changes. - The core is reached only through the replaced seams: the Kotlin test JVM does not load the core library, and the iOS app host in CI is linked with stub libraries.
Kotlin apps
In apps/multiplatform:
| Source set | Gradle task | Contents |
|---|---|---|
| common/src/commonTest | :common:desktopTest |
tests of common code (kotlin.test) |
| common/src/desktopTest | :common:desktopTest |
JVM tests: Compose views with runComposeUiTest, layout positions with renderScene and Modifier.record (LayoutRecorder.kt), files with withTempDir, app fixtures, snapshots |
| android/src/test | :android:testFossDebugUnitTest |
a template test, ExampleUnitTest |
| android/src/androidTest | connectedFossDebugAndroidTest, with a device or an emulator |
a template test, ExampleInstrumentedTest |
- Run:
./gradlew :common:desktopTest; one class:./gradlew :common:desktopTest --tests chat.simplex.app.ChatItemUpdateTest. The CI command: CI guide, kotlin. - CI: build.yml, step "Build and test Kotlin apps" (kotlin scope); desktop-linux, app job.
- Pitfalls:
- The working directory is apps/multiplatform/common: recordings are read from
../../../tests/fixtures/api, sources fromsrc/. - The tests use the app's singletons, such as
ChatModel,appPrefsandsimplexWindowState: they are restored in@AfterTest. - The test JVM does not load the core library: code that calls it is tested with the replaced seams of app fixtures.
- The working directory is apps/multiplatform/common: recordings are read from
Desktop snapshots
-
Checked: each
@Previewof commonMain and desktopMain, rendered in the light and dark themes bySnapshotTest(SnapshotTest.kt), equals its reference in common/src/desktopTest/snapshots, within 8 of 255 per colour channel (snapshots-compare.py). -
Write: a new
@Previewis added topreviewScenes, withchatItemFontScalesfor a chat item, or topreviewsCallingCorewhen its layout calls the core library;everyPreviewIsRenderedfails otherwise. A preview usespreviewDatein place of the current time. -
Record and check: with
scripts/ci/snapshots-remote.py desktop record, desktop-linux renders the previews of the pushed branch, and the renders replace the references, for review and commit; withcheck, the renders that differ are listed. -
Run, from apps/multiplatform, then from the repository root:
SNAPSHOT_DIR=<absolute dir> ./gradlew -Psimplex.assets.dir=../../assets :common:desktopTest --tests chat.simplex.app.SnapshotTest python3 scripts/ci/snapshots-compare.py apps/multiplatform/common/src/desktopTest/snapshots <absolute dir> <diff dir> -
CI: desktop-linux, step "Compare snapshots"; in build.yml, the previews are rendered without a comparison.
-
Details: desktop-linux README, "Snapshots".
iOS app
In apps/ios, SimpleX.xcodeproj:
| Target | Kind | Scheme | CI |
|---|---|---|---|
| SimpleX Unit Tests | unit tests hosted by the app | SimpleX Unit Tests | build.yml, build-ios, step "Run iOS unit tests" (ios scope) |
| SimpleX Snapshots | unit tests hosted by the app: preview renders and tests of a running chat | SimpleX (iOS) | ios-sim, after the smoke test |
| Tests iOS | UI tests of Xcode's template | SimpleX (iOS) | local runs only |
A new test file is added to its target in project.pbxproj: a PBXBuildFile, a PBXFileReference, the target's group and its Sources phase.
SimpleX Unit Tests
- Checked: app code on app fixtures and other logic of the app.
- Write: an
XCTestCasein apps/ios/SimpleX Unit Tests;fixtureJSON,singleFieldJSONandwaitForare at file scope of NameAlertsTests.swift. - Run, on macOS:
scripts/ci/ios-stub-libs.sh, then thexcodebuild testcommand of the step "Run iOS unit tests" of build-ios in build.yml; or Product > Test with the scheme SimpleX Unit Tests. - Pitfalls:
- The stubs replace the libraries in apps/ios/Libraries/sim, where scripts/ios/prepare.sh puts the core built for the simulator.
- With the stubs, the app opens its database error screen and calls the core no more.
- The tests run with
-testLanguage en -testRegion US; a test of dates setsNSTimeZone.defaultto UTC.
SimpleX Snapshots
- Checked: each
PreviewProviderof apps/ios, rendered in the light and dark themes, equals its reference in References, within 8 of 255 per colour channel; previews markedallSizesare rendered at three content sizes. SnapshotTests.swift also contains assertion tests of layout functions, calls and VoIP pushes; the VoIP push tests require a running chat (waitUntilChatActive). - Write: a new
PreviewProvideris added topreviewScenes(), or toEXCLUDEDof check-previews.py when its renders differ between runs; check-previews.py fails on a#Preview. A preview usespreviewDatein place of the current time. - Record and check:
scripts/ci/snapshots-remote.py ios record|check, as for desktop. - Run, from the repository root, with the core in apps/ios/Libraries/sim and a simulator where the app has a running chat:
SIM_UDID=<udid> scripts/ci/ios-sim/snapshots.sh <derived data> <out dir>(snapshots.sh). - CI: build-ios runs check-previews.py; ios-sim renders and compares at the end of the smoke test, whose step fails on a difference.
- Details: ios-sim README, "Snapshots".
Tests iOS
Xcode's UI test template: testExample launches the app, testLaunchPerformance measures its launch, testLaunch attaches a screenshot of the launch screen. It is in the test action of the scheme SimpleX (iOS), with SimpleX Snapshots: xcodebuild test -scheme "SimpleX (iOS)" of apps/ios/README.md runs both.
App workflows
The app built from a branch, its core included, driven on a device by scripts and, on request, by Claude following a test plan, with an optional test backend (relay, directory, names server, peers).
| Workflow | App | Smoke test | Claude plan | README |
|---|---|---|---|---|
ios-sim |
iOS app in an iPhone simulator | onboarding images sent in a group snapshots |
device-test-plan.md, cases per layout in claude-scope | ios-sim |
desktop-linux |
desktop app on a virtual display, Ubuntu | desktopTest with snapshots onboarding a local group both layouts images sent |
claude-test.md, claude-scope | desktop-linux |
android-emu |
debug APK in an Android emulator | onboarding a local group both layouts images sent |
claude-test.md, claude-scope | android |
- Write: a scripted case in the workflow's
smoke-test.sh, with itsuidriver; a Claude plan in the files of the table, withbackend: yesin claude-scope when the cases use the test backend. - Run: by hand only, on any branch: CI guide, Starting a run. Each run uploads the app for manual tests.
- Pitfalls:
- A run with
claudeandbackend: yesin claude-scope also requires the inputbackend, or its plan job fails. - A Claude run requires the branch rule of the GitHub environment
app-sim-claudeto include the branch. - The Android app runs under ARM translation: with the input
release, the released app runs in the same emulator, to tell an app failure from the emulator's.
- A run with
Packages and other scopes
| Scope | Tests | Run |
|---|---|---|
| packages: Node.js library | packages/simplex-chat-nodejs/tests, jest: unit tests *.unit.test.ts, with core functions mockedtests of the add-on and of library downloads live tests with a local smp-server ( describeIfSmpServerAvailable: SMP_SERVER, or a download on Linux) |
scripts/ci/package-tests.sh build linux-x86_64scripts/ci/package-tests.sh test |
| packages: Python library | packages/simplex-chat-python/tests, pytest, with a fake libsimplex (FakeLib) |
as above |
| packages: TypeScript types | generated by Bot API docs, compiled by package-tests.sh | as above |
| mcp | packages/simplex-chat-mcp/test, jest:test/unittest/e2e, with a local smp-server and xftp-servermanual checks: TESTING.md |
scripts/ci/scope-tests.sh mcp |
| support-bot | apps/simplex-support-bot/bot.test.ts, vitest, with simplex-chat and @simplex-chat/types replaced by test/mocks |
scripts/ci/scope-tests.sh support-bot |
| support-bot-light | apps/simplex-support-bot-light/tests, pytest, with FakeChatApi of conftest.py; ruff, pyright |
scripts/ci/scope-tests.sh support-bot-light |
| website | website/test, node --test of src/js/simplex-lib.jsc |
npm test in website |
| badge-web | apps/simplex-badge-service/web/test, node --test of the compiled tests, in UTC |
npm ci, npm test in apps/simplex-badge-service/web |
| ci-scripts | scripts/ci/test_packages_release.py, unittest of packages_release.py | python3 scripts/ci/test_packages_release.py |
CI of each scope: CI guide, Scopes. Scopes lacking CI tests, with tests run locally:
| Scope | Run | Notes |
|---|---|---|
| ts-client | npm test in packages/simplex-chat-client/typescript |
prettier, eslint and jest; client.test.ts is skipped |
| webrtc | npm test in packages/simplex-chat-webrtc |
fullstack.test.js requires SIMPLEX_CHAT and SMP_SERVER, the paths of the chat CLI and of smp-server |
| calculator-bot | npm test in apps/simplex-calculator-bot |
calculatorBot.test.ts: smp-server from SMP_SERVER or a download on Linux, and tests/fixtures/tls |