Files
simplex-chat/docs/contributing/TESTS.md
T
533bfd8ff5 core, store: merge master into names (#8298)
* core, ui: validate short link type, show contact info for long addresses and rejected contacts (#8184)

* tests: short link type, QR code capacity and contact statuses

* core: validate the type of short links

* android, desktop: show contact info for addresses above QR code capacity

* ui: show chats with rejected contact requests

* tests: short link types with group, channel and contact data

* core: validate short link type against link data

* tests: fix short link data test

* tests: group data in invitation link before its first use

* tests: connect via address in contact profile with group link

* core: connect via contact address only with address links

* core: parse short link data once, by link type

* core: types of short link data parser

* core: test short link types and contact rejection, record app fixtures

* android, desktop: test contact info, link alerts and rejected contacts

* ios: test invalid link alerts and rejected contacts in the chat list

* desktop: contact info tests find texts without app resources

* ui: core responses of invalid link plans and a rejected contact request

* desktop: contact info tests provide the app bar handler

* core: normalise file and message keys in app test fixtures

* core: normalise deletion times in app test fixtures

* desktop: test that a QR code above capacity is not shared on tap

* core: test relay link checks and known channel refresh from link data

* libs: nodejs 7.1.4 (#8263)

* nodejs: live test reads the updated item under the key of 7.1 and 7.2 cores

* libs: nodejs 7.1.4

* mcp: session states coloured by what they need, green for working (#8265)

* mcp: session states in the colours of what they need: green working, orange waiting in the chat, red idle

* mcp: the space after waiting for you in the tests

* mcp: a session with a background command shows working in the background

* mcp: a readable README, and the roadmap lines of 0.3.0 (#8264)

* mcp: readme by what the plugin does for its users, roadmap lines of 0.3.0

* mcp: almost every waiting state

* mcp: reply, edit and react in one line

* mcp: flat layout without the avatar note

* mcp: sessions use flat layout

* mcp: shorter lines on answers, the Main group and approvals

* mcp: approvals line without the taps

* mcp: live stream line

* mcp: status line

* mcp: shorter lines

* mcp: 0.3.1 with simplex-chat 7.1.4 (#8266)

* docs: testing approaches of the codebase (#8255)

* docs: app changes come with app tests on the core's recorded responses (#8252)

* ci: decide the library claim from the written releases (#8137)

* ci: test the library claim when reads return the previous releases

* ci: decide the library claim from the written releases

* tests: record name fixtures after the channel join settles (#8250)

* core, ci: update sqlcipher-simple to 0.4.19.0 and simplexmq (#8270)

* update sqlcipher-simple to 0.4.19.0

* sqlcipher-simple from simplex-chat/sqlcipher-simple

* update simplexmq

* ci: restore source repositories only for the same cabal.project

* ci: keep restored source checkouts only at their pinned commits

* android, desktop, ios: copy markdown files with line breaks, and the whole file with one button (#8269)

* android, desktop: test copying lines from the markdown text view

* android, desktop: keep line breaks in text copied from markdown files

* android, desktop: check the last copied text in the markdown view test

* android, desktop, ios: button to copy the whole markdown file

* android, desktop: test the markdown file copy button

* android, desktop, ios: ask before copying markdown files over 128 KB

* android, desktop, ios: title the markdown copy alerts File too large

* ios: test the markdown file copy button

* android, desktop, ios: preview the markdown file view and test the copy limit

* android, desktop, ios: snapshot references of the markdown file view

* docs: names conditions describe upgradeable registry contracts (#8277)

* docs: names conditions describe upgradeable registry contracts

* docs: names conditions without the immutability after stabilization

---------

Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com>

* ci: reproducible Linux builds of 7.1 releases (#8276)

* ci: plan reproducible Linux builds of 7.1 releases

* ci: reproduce Linux builds with the cabal store path of releases

* ci: reproduce releases with the current reproduce script

* ci: build in the image with the cabal directory of releases

* ci: reproduce releases on the Ubuntu 24.04 runner

* ci: correct the failure history and rollout in the plan

---------

Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com>

* website: remove the webinar promotion (#8281)

* website: remove the webinar invitation from the home page

* website: redirect the livestream page to Wefunder

* mcp server: recall the chat history in a new agent session (#8141)

* mcp server: recall the chat history in a new agent session

* docs: chat history in receive for a new agent session

* mcp server: type the history page for the compiler

* mcp server: recall the chat history after a compaction too

* docs: history plan changes with the message helpers of master

Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75

* docs: drop anchor remapping from the docs table of the history plan

Claude-Session: https://claude.ai/code/session_01AV2VpomCCQNciqggT9gr75

---------

Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com>

* core: prevent stalled join of channel subscribers to a new relay (#8239)

* core: save relay connection before requesting relay link data

* core: share relay connection creation between join and retry

* tests: connect to a re-added relay when its connection is saved late

* tests: delay only the relay connection record, not the command

* ci, tests: run all tests in CI (#8238)

* bots, tests: run broadcast bot test in CI

* tests: test moderation that arrives before the message

* ci, tests: run all tests in CI

* tests: update re-enabled tests, restore query plans, PostgreSQL schema from CI

* tests: remove CI skip helpers no longer used

* tests: expect XFTP AUTH error of cancelled standalone file

* tests: update query plans from CI

* tests: update query plans from CI

* tests: expect group history and member contacts in async group test

* tests: accept subscription summary after first events on restart

* tests: make AUTH and client timeout direct tests deterministic

* tests: keep bob's badge clock before renewal until the move back

* tests: keep the quota of 2 messages only on the member's receive queue

* tests: keep query plans files with repeated entries of one query

* tests: update query plans from CI

* tests: stop the host before a member requests to join with a badge

* tests: remove repeated entries of one query from query plans files

* tests: accept the refusal of an unknown remote host after its session code

* ci: state when the query plans test fails

* tests: skip multicast discovery on macOS CI runners

* tests: accept a remote host started with one local address

* tests: restore the client timeout test except its zero timeout

* tests: fail when a remote host prints another line than its other addresses

* ci: name the tests that overwrite generated files

* tests: move the badge clock after the restoring device arms its wake

* tests: keep both members on the quota server in the member inactivity test

* tests: accept subscription summaries in any order in the async group test

* tests: update query plans from CI

* tests: quota of 2 only on bob's queues in member inactivity test

* tests: connect to a deleted link after its queue is deleted

* core: save relay connection before requesting relay link data

* core: share relay connection creation between join and retry

* tests: connect to a re-added relay when its connection is saved late

* tests: delay only the relay connection record, not the command

* tests: record API fixtures on SQLite only

* core: update query plans

* ci: run PostgreSQL tests with the steps of build-linux

* desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts (#8288)

* desktop: prevent crash on Linux with bitmap and WOFF2 fallback fonts

* desktop: record time offsets and snapshots rendered with Skia m138

* desktop: name the time offset test, skip only a missing fc-cache, update plan

---------

Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com>

* core, directory: stop cancelled actions instead of continuing them (#8279)

* ci: retry npm install once after a failure (#8280)

* core: query plans of the store group tests that master now verifies

---------

Co-authored-by: brenzi-agent <346870783+brenzi-agent@users.noreply.github.com>
Co-authored-by: sh-agents <justcheckingifitwillwork@gmail.com>
Co-authored-by: sh-agents <350651734+sh-agents@users.noreply.github.com>
Co-authored-by: brenzi-agent <brenzi-agent@brenzi.ch>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
2026-11-21 23:35:25 +00:00

27 KiB
Raw Blame History

Testing

The testing approaches of this repository: what is checked, how a test is written and run, and where CI runs it. The CI scopes, the reduced build and the ci-tests block of a pull request: CI on GitHub's runners.

Tests for a change

Change Tests
Behaviour of the chat core: messages, groups, files, profiles, connections chat scenario test
Encoding of a protocol message or a JSON type, markdown, a pure function of the core unit test
Command, response, event or type of the core API Bot API docs, regenerated
app fixture, when an app processes it
Database migration schema dumps of SQLite and PostgreSQL
query plans
Directory service, store service, remote sessions the suite of the area
App processing of a core response or event app fixture, replayed in Kotlin and iOS tests
App screen a preview, rendered by the desktop and iOS snapshot tests
Flow across app screens a case of an app workflow
Kotlin or Swift code without views desktopTest or commonTest
SimpleX Unit Tests
Package, bot, website or CI script the tests of its scope
Scope lacking CI tests its CI coverage, in the same pull request (Adding CI coverage)

Core

The hspec suite simplex-chat-test of simplex-chat.cabal, with the entry point tests/Test.hs. Building it, running it and selecting tests by hspec match patterns: CI guide, core. A pattern is a path of describe and it names, for example /SimpleX chat client/direct tests/.

  • A new test module is added to other-modules of the test suite in simplex-chat.cabal, and its spec to main of Test.hs.
  • tests/tmp contains a directory per test (tmpBracket in Test.hs) and is emptied at the start and the end of a run (withTmpFiles in ChatClient.hs).
  • With CI=true, set on GitHub's runners, the tests marked xdescribe'' and xit'' are skipped, and on macOS those marked xitMacCI (ChatTests/Utils.hs); the tests marked xit' are skipped on Linux.
  • tests/fixtures: images and files sent in the tests, TLS credentials of the test servers, and the app fixtures in api.
  • In the full matrix of build.yml, the whole suite runs on Linux x86_64, with GHC 9.6.3 and 8.10.7, and on macOS aarch64; on a draft pull request, the tests matched by its ci-tests block run.
  • Coverage: -fhpc added to ghc-options of the test suite in simplex-chat.cabal.

Suites

Pattern Module Checked
/SimpleX chat client/ ChatTests.hs chat scenarios: direct, forward, group, chat relay, local, file, profile and chat list tests
/names tests/ ChatTests/Names.hs SimpleX names, resolved by a local resolver (NameResolver.hs); SQLite only
/SimpleX chat protocol/ ProtocolTests.hs chat protocol messages, short link data, batch limits, preferences JSON
/JSON Tests/ JSONTests.hs, JSONFixtures.hs conversion of the Swift JSON form to the tagged form, with QuickCheck; JSON of markdown
/SimpleX chat markdown/ MarkdownTests.hs the markdown parser
/SimpleX chat view/ ViewTests.hs functions of the CLI view
/Mobile API Tests/ MobileTests.hs the API of the core library (src/Simplex/Chat/Mobile.hs) and its C functions; SQLite only
/Message batching/ MessageBatching.hs batches of messages and of relay deliveries
/Valid names/ ValidNames.hs display names made valid
/Random servers/ RandomServers.hs the random choice of preset servers
/Operators/ OperatorTests.hs server operators, usage conditions, validation of user servers
/Member relations/ MemberRelationsTests.hs member relation vectors
/Image resize/ ImageTests.hs image resizing
/Supporter badges/ BadgeTests.hs badge credentials and proofs
/WebRTC encryption/ WebRTCTests.hs encryption of call media frames; SQLite only
/Remote session/ RemoteTests.hs a desktop controlling a mobile: handshake, messages, files
/SimpleX Directory service bot/
/SimpleX Directory names/
Bots/DirectoryTests.hs the directory service with chat clients (withDirectoryService); names: SQLite only
/SimpleX Broadcast bot/ Bots/BroadcastTests.hs the broadcast bot; skipped when CI=true
/SimpleX store service/ Bots/StoreService/*Tests.hs configuration, web server and schema, catalog, groups, invoice waiters, payment providers, store receipts; the providers are local fakes (Fake*.hs)
/SimpleX store service e2e/ Bots/StoreService/BotTests.hs, GroupIntegrationTests.hs the store service bot with chat clients
/Schema dump/
/Postgres schema dump/
/Bot API docs/
/Save query plans/
SchemaDump.hs, PostgresSchemaDump.hs, APIDocs.hs committed files

Chat scenario tests

  • Checked: chat cores of several clients connected through a test SMP server, by the terminal output of each client, and by queries of a client's database for a state absent from the output.

  • Pattern, registered with it "<name>" testDirectMessage in the spec of its module:

    testDirectMessage :: HasCallStack => TestParams -> IO ()
    testDirectMessage =
      testChat2 aliceProfile bobProfile $ \alice bob -> do
        connectUsers alice bob
        alice #> "@bob hello"
        bob <# "alice> hello"
    
  • Helpers of ChatClient.hs and ChatTests/Utils.hs:

    Helper Use
    testChat2 … testChat5, testChatCfgOpts2 clients with new databases
    withNewTestChat, withTestChat one client with a new database; a restart with its database
    withNewTestChatCfgOpts a client with a ChatConfig and ChatOpts, such as chatHooks or markRead = False
    connectUsers, createGroup2, createGroup3 a contact; a group of 2 or 3 members
    ##>, #>, send a command and its echo; a message and its sent item, time removed; a command alone
    <##, <#, <##., .<## the next line; the next line, time removed; its prefix; its suffix
    <###, concurrentlyN_ lines in any order; lines of several clients at once
    #$> a command and a function of its output line, as in ("/_get chat @2 count=100", chat, [...])
    withCCTransaction, shouldEventuallyReturn a query of the client's database; a value polled for 20 s
    withXFTPServer, withSmpServer', withSmpServerAndNames an XFTP server; another SMP server; an SMP server with a names resolver
    versionTestMatrix2, testCfgVPrev, testCfgV1 clients of the previous and of the first supported versions
    withTestOutput the terminal output of the clients printed
  • Run: /SimpleX chat client/<area> tests/, for example /SimpleX chat client/group tests/.

  • CI: build-linux, step "Run tests".

  • Pitfalls:

    • The output of every client is read to its end: a test fails when a client prints a line within 0.1 s after the test (<// in testChatN and the with…TestChat… brackets).
    • The lines of one client are read in order; the lines of several clients are awaited with concurrently_ or concurrentlyN_.
    • A test fails when a line is awaited for 20 s (getTermLine') and when it runs longer than 180 s (it of ChatTests/Utils.hs, imported in place of hspec's).
    • Each test has its own ports (withPortBase): the ports 7001–7003 of server addresses in ChatOpts are mapped to them (testPortsCfg), and smpServerStr, smpServer2Str and xftpServerStr return the addresses for commands and expected lines.
    • One SMP server is started per test by testBracket of Test.hs; XFTP and other servers are started by the test.

Unit tests

  • Checked: functions and encodings of the core, in the suites without clients.
  • Pattern: a Spec of describe and it with hspec expectations, as in ViewTests.hs and ProtocolTests.hs.
  • Pitfall: in MobileTests, the expected JSON is the Swift form on macOS with the swift flag and the tagged form otherwise; both forms are in JSONFixtures.hs.

Checks against committed files

Each of these tests writes its file, then compares the new text with the previous one: the first run after a change of the text fails, and the next passes. The rewritten file is reviewed and committed with the change.

Pattern Written Source
/Bot API docs/ bots/api/COMMANDS.md, EVENTS.md, TYPES.md
packages/simplex-chat-client/types/typescript/src/.ts
packages/simplex-chat-python/src/simplex_chat/types/_
.py
the API docs in bots/src/API/Docs (testGenerate in APIDocs.hs)
/Schema dump/ src/Simplex/Chat/Store/SQLite/Migrations/chat_schema.sql SQLite migrations, dumped with sqlite3
/Postgres schema dump/ src/Simplex/Chat/Store/Postgres/Migrations/chat_schema.sql PostgreSQL migrations, dumped with pg_dump
/Save query plans/ chat_query_plans.txt and agent_query_plans.txt in src/Simplex/Chat/Store/SQLite/Migrations the queries recorded in the run (chatQueryStats, agentQueryStats)
recording tests tests/fixtures/api app fixtures
  • Bot API docs also fail on a command, response or event of the API that is neither documented in bots/src/API/Docs nor listed in cliCommands, undocumentedCommands, undocumentedResponses or undocumentedEvents, on a type of a documented one missing from Docs/Types.hs, and on a documented command without responses. They run with GHC 9.6 and later (MIN_VERSION_base(4,18,0)), in SQLite builds.
  • Schema dump also checks down migrations (exceptions in skipComparisonForDownMigrations), STRICT tables, and the output of .lint fkey-indexes against chat_lint.sql, an empty file, so that the test fails for a foreign key without an index.
  • Save query plans is skipped when CI=true. The plans are of the queries of that run only, so the files are committed from a local run of the whole suite.

PostgreSQL

  • Run:

    cabal test simplex-chat-test --constraint 'simplexmq +client_postgres' --constraint 'simplex-chat +client_postgres' --ghc-options=-O0 --test-option=-m --test-option='/SimpleX chat client/'
    

    The flags client_postgres of simplexmq and simplex-chat, as in scripts/desktop/build-lib-linux.sh; dbPostgres is defined for the tests.

  • Needed: a PostgreSQL server that accepts the role postgres without a password, and pg_dump. The role test_chat_user and the database test_chat_db are created at the start of a run and dropped at its end (withTestDB in Test.hs); each client has its own schema, dropped after each test (dropTestSchemas in ChatClient.hs).

  • Left out of PostgreSQL builds: Schema dump, Bot API docs, WebRTC encryption, Mobile API Tests, both names suites, Save query plans.

  • CI: local runs only; build.yml builds the PostgreSQL library for releases (build-linux-postgres).

  • Pitfall: the Postgres schema dump calls sed -i '' (BSD sed) unless CI=true.

App fixtures

The core's responses and events are recorded by core tests in tests/fixtures/api and replayed by Kotlin and iOS tests in place of the core library, so that app code is tested on the output of the current core.

flowchart LR
  CT["core test<br>tests/ChatTests"] -->|"apiFixture<br>eventsFixture"| F["tests/fixtures/api"]
  F --> T["Kotlin desktopTest<br>iOS SimpleX Unit Tests"]
  T -->|"recorded responses<br>in place of the core"| V["app functions and views"]
  T -->|"recorded events"| P["processReceivedMsg"]
  V --> M["ChatModel, alerts, notifications"]
  P --> M

Recording

Helpers of ChatTests/Utils.hs, called in a chat scenario test:

Helper Written to tests/fixtures/api/<fixture>.json
apiFixture fixture cc placeholders FixtureRequests {frCommands, frMarkdown, frUris} an object with commands, markdown and uris, keyed by the request: the response to each command, encoded by chatSendCmd of the core library's API; chatParseMarkdown of each text; chatParseUri of each link
nameFixture fixture cc name apiFixture of /_connect plan 1 <name> and of the markdown of the name, in names/
eventsFixture fixture events steps an array of the events appended by recordItemEvent events, in eventHook of chatHooks, during steps: newChatItems and chatItemUpdated, encoded as chatRecvMsg encodes them
nameMovedFixture a nameMoved event, built from the IDs of the named contacts and groups

Examples: testPlanKnownNameAvailable in ChatTests/Names.hs, testDirectMessageUpdatePrevItem and testMissedCall in ChatTests/Direct.hs.

Stable recordings:

  • Values of each run are replaced by stableJSON:

    Fields Replaced by
    createdAt
    updatedAt
    chatTs
    userMemberProfileSentAt
    itemTs
    1970-01-01T00:00:00Z
    agentConnId
    xContactId
    connFullLink
    connShortLink
    contactLink
    groupLink
    viaGroupLinkUri
    simplexUri
    memberId
    memberPubKey
    publicGroupId
    businessId
    customerId
    itemSharedMsgId
    the field name
    groupRelays "groupRelay" per element

    A field with a new value in each run is added to this list.

  • Strings of the test, such as links, are replaced by the placeholders of apiFixture, in keys and values: [(cLink, "CONTACT_LINK")] in testPlanAddressContactViaAddress, so that the app test connects to CONTACT_LINK.

  • The recorded state is awaited first: the response to an update is recorded after the item status in the database is snd_rcvd ok complete (shouldEventuallyReturn in testDirectMessageUpdatePrevItem), and the steps of eventsFixture end after the expected number of events (testMissedCall).

  • The client whose events are recorded has markRead = False: the CLI marks received items read when it prints them (runTerminalOutput in src/Simplex/Chat/Terminal/Output.hs), while the apps receive unread items.

  • Recordings are in the tagged JSON form of builds without the swift flag.

Replay

Kotlin, common/src/commonMain iOS
Commands chatSendCmdRetry, platform/Core.kt sendSimpleXCmdStr, SimpleXChat/API.swift
Markdown chatParseMarkdown, platform/Core.kt parseSimpleXMarkdownText, SimpleXChat/API.swift
Links chatParseUri, platform/Core.kt parseSanitizeUri, SimpleXChat/API.swift
Events ChatController.processReceivedMsg, model/SimpleXAPI.kt processReceivedMsg, Shared/Model/SimpleXAPI.swift
Notifications ntfManager, platform/NtfManager.kt NtfManager.shared, Shared/Model/NtfManager.swift
Recording read by fixtureText fixtureJSON, then singleFieldJSON to the Swift JSON form
Tests NameAlertsTest.kt, ChatItemUpdateTest.kt NameAlertsTests.swift, ChatItemUpdateTests.swift
  • Pattern: recorded(fixture) in the NameAlerts tests sets the first three seams to lookups in the recording; the test fails at tear-down on a request without a recorded response, unless the request is checked with assertRequested. The ChatItemUpdate tests decode the recorded events, pass each to processReceivedMsg and compare ChatModel. Tear-down restores the seams and clears the model.
  • Run and CI: Kotlin apps, SimpleX Unit Tests. CI runs the kotlin and ios scopes for a change in tests/fixtures/api (BUILD_CHANGES in scripts/ci/packages_release.py). The recording tests are core tests, selected in the ci-tests block by the patterns of the CI guide, core.
  • Pitfalls:
    • singleFieldJSON converts each object with a type field, except MsgContent, recognized by its tags. The core encodes other types with type in both JSON forms: the types of taggedObjectJSON in src, such as MsgChatLink, LinkContent, MsgScope and ChatBotCommand, and MsgReaction, by its own instance. A recording with them requires another conversion rule.
    • The tests use the app's singletons, such as ChatModel and AlertManager: each test restores what it changes.
    • The core is reached only through the replaced seams: the Kotlin test JVM does not load the core library, and the iOS app host in CI is linked with stub libraries.

Kotlin apps

In apps/multiplatform:

Source set Gradle task Contents
common/src/commonTest :common:desktopTest tests of common code (kotlin.test)
common/src/desktopTest :common:desktopTest JVM tests: Compose views with runComposeUiTest, layout positions with renderScene and Modifier.record (LayoutRecorder.kt), files with withTempDir, app fixtures, snapshots
android/src/test :android:testFossDebugUnitTest a template test, ExampleUnitTest
android/src/androidTest connectedFossDebugAndroidTest, with a device or an emulator a template test, ExampleInstrumentedTest
  • Run: ./gradlew :common:desktopTest; one class: ./gradlew :common:desktopTest --tests chat.simplex.app.ChatItemUpdateTest. The CI command: CI guide, kotlin.
  • CI: build.yml, step "Build and test Kotlin apps" (kotlin scope); desktop-linux, app job.
  • Pitfalls:
    • The working directory is apps/multiplatform/common: recordings are read from ../../../tests/fixtures/api, sources from src/.
    • The tests use the app's singletons, such as ChatModel, appPrefs and simplexWindowState: they are restored in @AfterTest.
    • The test JVM does not load the core library: code that calls it is tested with the replaced seams of app fixtures.

Desktop snapshots

  • Checked: each @Preview of commonMain and desktopMain, rendered in the light and dark themes by SnapshotTest (SnapshotTest.kt), equals its reference in common/src/desktopTest/snapshots, within 8 of 255 per colour channel (snapshots-compare.py).

  • Write: a new @Preview is added to previewScenes, with chatItemFontScales for a chat item, or to previewsCallingCore when its layout calls the core library; everyPreviewIsRendered fails otherwise. A preview uses previewDate in place of the current time.

  • Record and check: with scripts/ci/snapshots-remote.py desktop record, desktop-linux renders the previews of the pushed branch, and the renders replace the references, for review and commit; with check, the renders that differ are listed.

  • Run, from apps/multiplatform, then from the repository root:

    SNAPSHOT_DIR=<absolute dir> ./gradlew -Psimplex.assets.dir=../../assets :common:desktopTest --tests chat.simplex.app.SnapshotTest
    python3 scripts/ci/snapshots-compare.py apps/multiplatform/common/src/desktopTest/snapshots <absolute dir> <diff dir>
    
  • CI: desktop-linux, step "Compare snapshots"; in build.yml, the previews are rendered without a comparison.

  • Details: desktop-linux README, "Snapshots".

iOS app

In apps/ios, SimpleX.xcodeproj:

Target Kind Scheme CI
SimpleX Unit Tests unit tests hosted by the app SimpleX Unit Tests build.yml, build-ios, step "Run iOS unit tests" (ios scope)
SimpleX Snapshots unit tests hosted by the app: preview renders and tests of a running chat SimpleX (iOS) ios-sim, after the smoke test
Tests iOS UI tests of Xcode's template SimpleX (iOS) local runs only

A new test file is added to its target in project.pbxproj: a PBXBuildFile, a PBXFileReference, the target's group and its Sources phase.

SimpleX Unit Tests

  • Checked: app code on app fixtures and other logic of the app.
  • Write: an XCTestCase in apps/ios/SimpleX Unit Tests; fixtureJSON, singleFieldJSON and waitFor are at file scope of NameAlertsTests.swift.
  • Run, on macOS: scripts/ci/ios-stub-libs.sh, then the xcodebuild test command of the step "Run iOS unit tests" of build-ios in build.yml; or Product > Test with the scheme SimpleX Unit Tests.
  • Pitfalls:
    • The stubs replace the libraries in apps/ios/Libraries/sim, where scripts/ios/prepare.sh puts the core built for the simulator.
    • With the stubs, the app opens its database error screen and calls the core no more.
    • The tests run with -testLanguage en -testRegion US; a test of dates sets NSTimeZone.default to UTC.

SimpleX Snapshots

  • Checked: each PreviewProvider of apps/ios, rendered in the light and dark themes, equals its reference in References, within 8 of 255 per colour channel; previews marked allSizes are rendered at three content sizes. SnapshotTests.swift also contains assertion tests of layout functions, calls and VoIP pushes; the VoIP push tests require a running chat (waitUntilChatActive).
  • Write: a new PreviewProvider is added to previewScenes(), or to EXCLUDED of check-previews.py when its renders differ between runs; check-previews.py fails on a #Preview. A preview uses previewDate in place of the current time.
  • Record and check: scripts/ci/snapshots-remote.py ios record|check, as for desktop.
  • Run, from the repository root, with the core in apps/ios/Libraries/sim and a simulator where the app has a running chat: SIM_UDID=<udid> scripts/ci/ios-sim/snapshots.sh <derived data> <out dir> (snapshots.sh).
  • CI: build-ios runs check-previews.py; ios-sim renders and compares at the end of the smoke test, whose step fails on a difference.
  • Details: ios-sim README, "Snapshots".

Tests iOS

Xcode's UI test template: testExample launches the app, testLaunchPerformance measures its launch, testLaunch attaches a screenshot of the launch screen. It is in the test action of the scheme SimpleX (iOS), with SimpleX Snapshots: xcodebuild test -scheme "SimpleX (iOS)" of apps/ios/README.md runs both.

App workflows

The app built from a branch, its core included, driven on a device by scripts and, on request, by Claude following a test plan, with an optional test backend (relay, directory, names server, peers).

Workflow App Smoke test Claude plan README
ios-sim iOS app in an iPhone simulator onboarding
images sent in a group
snapshots
device-test-plan.md, cases per layout in claude-scope ios-sim
desktop-linux desktop app on a virtual display, Ubuntu desktopTest with snapshots
onboarding
a local group
both layouts
images sent
claude-test.md, claude-scope desktop-linux
android-emu debug APK in an Android emulator onboarding
a local group
both layouts
images sent
claude-test.md, claude-scope android
  • Write: a scripted case in the workflow's smoke-test.sh, with its ui driver; a Claude plan in the files of the table, with backend: yes in claude-scope when the cases use the test backend.
  • Run: by hand only, on any branch: CI guide, Starting a run. Each run uploads the app for manual tests.
  • Pitfalls:
    • A run with claude and backend: yes in claude-scope also requires the input backend, or its plan job fails.
    • A Claude run requires the branch rule of the GitHub environment app-sim-claude to include the branch.
    • The Android app runs under ARM translation: with the input release, the released app runs in the same emulator, to tell an app failure from the emulator's.

Packages and other scopes

Scope Tests Run
packages: Node.js library packages/simplex-chat-nodejs/tests, jest:
unit tests *.unit.test.ts, with core functions mocked
tests of the add-on and of library downloads
live tests with a local smp-server (describeIfSmpServerAvailable: SMP_SERVER, or a download on Linux)
scripts/ci/package-tests.sh build linux-x86_64
scripts/ci/package-tests.sh test
packages: Python library packages/simplex-chat-python/tests, pytest, with a fake libsimplex (FakeLib) as above
packages: TypeScript types generated by Bot API docs, compiled by package-tests.sh as above
mcp packages/simplex-chat-mcp/test, jest:
test/unit
test/e2e, with a local smp-server and xftp-server
manual checks: TESTING.md
scripts/ci/scope-tests.sh mcp
support-bot apps/simplex-support-bot/bot.test.ts, vitest, with simplex-chat and @simplex-chat/types replaced by test/mocks scripts/ci/scope-tests.sh support-bot
support-bot-light apps/simplex-support-bot-light/tests, pytest, with FakeChatApi of conftest.py; ruff, pyright scripts/ci/scope-tests.sh support-bot-light
website website/test, node --test of src/js/simplex-lib.jsc npm test in website
badge-web apps/simplex-store-service/web/test, node --test of the compiled tests, in UTC npm ci, npm test in apps/simplex-store-service/web
ci-scripts scripts/ci/test_packages_release.py, unittest of packages_release.py python3 scripts/ci/test_packages_release.py

CI of each scope: CI guide, Scopes. Scopes lacking CI tests, with tests run locally:

Scope Run Notes
ts-client npm test in packages/simplex-chat-client/typescript prettier, eslint and jest; client.test.ts is skipped
webrtc npm test in packages/simplex-chat-webrtc fullstack.test.js requires SIMPLEX_CHAT and SMP_SERVER, the paths of the chat CLI and of smp-server
calculator-bot npm test in apps/simplex-calculator-bot calculatorBot.test.ts: smp-server from SMP_SERVER or a download on Linux, and tests/fixtures/tls