mirror of
https://github.com/simplex-chat/simplex-chat.git
synced 2026-11-12 15:58:03 +00:00
ui tests: about 300 more tests across all areas, server control, relays, remote desktop, CI workflow
- Tests for connection plans, prepared chats, edit/delete/selection, forwarding, drafts, group preferences and admission, group lifecycle, compose, live messages, item info, reports and moderation, privacy and SimpleX Lock, chat data and database, address and contact preferences, profiles, contact info, unread and search, files and reactions, group info and events, chat shell and statuses, joining, support and business chats, settings and appearance, network and servers, channels, remote desktop, Android intents and notifications. - Harness: server stop/start, smp2/xftp2, SOCKS stub, CLI chat relays, CLI remote host/controller, chat scroll and drag helpers, file chooser answers, clipboard in popups. - ui-tests workflow (manual dispatch only) running both suites on GitHub runners. - App findings recorded in plan section 15.5 (items 7-23), each covered by a test marked @KnownAppIssue.
This commit is contained in:
Narasimha-sc committed
1 parent
8927c18925
commit
69b3cc4f0e
256 files changed
+21715
-1093
No files matched your search
@@ -0,0 +1,71 @@
|
||||
# UI tests on GitHub's runners
|
||||
|
||||
`.github/workflows/ui-tests.yml` runs the Compose UI tests on GitHub runners (Ubuntu 24.04): the desktop app's `:common:desktopUiTest`, and the Android app's instrumented tests in an emulator. It uses the same runners as local runs, `apps/multiplatform/scripts/uitest.sh` and `apps/multiplatform/scripts/uitest-android.sh`. How the tests are written: `apps/multiplatform/common/src/uiTestShared/README.md`.
|
||||
|
||||
## Running it
|
||||
|
||||
The workflow runs only when started by hand, on any branch. In the Actions tab, `ui-tests` → Run workflow, then choose the branch and the inputs. With the GitHub CLI:
|
||||
|
||||
```sh
|
||||
gh workflow run ui-tests.yml --ref <branch> # all desktop tests
|
||||
gh workflow run ui-tests.yml --ref <branch> -f platform=both
|
||||
gh workflow run ui-tests.yml --ref <branch> -f platform=android -f tests=chat.simplex.app.ui.main.ChatListRowsTest
|
||||
gh workflow run ui-tests.yml --ref <branch> -f tests='*ChatListRowsTest, chat.simplex.app.ui.chat.ComposeTest#sendText_sendButton_peerReceives'
|
||||
```
|
||||
|
||||
| Input | Default | Does |
|
||||
|---|---|---|
|
||||
| `platform` | `desktop` | `desktop`, `android` or `both` |
|
||||
| `tests` | all tests | comma- or space-separated test classes (`chat.simplex.app.ui.main.ChatListRowsTest`) or methods (`Class#method`). Desktop also takes Gradle patterns with `*`; Android doesn't, and the plan job fails when it gets one |
|
||||
|
||||
GitHub offers Run workflow, and accepts `gh workflow run`, only for a workflow whose file is on the default branch. Until `ui-tests.yml` is there, a branch can't be tested with it; once it is, the run uses the workflow file of the branch it is started on. A newer run on the same branch cancels the older one.
|
||||
|
||||
Every run builds what the tests need from the branch's own code, and reuses an earlier build only when its sources are unchanged, as an artifact of the same branch named after a hash of its sources (30 days). The core libraries have the names and sources of `desktop-linux` and `android-emu`, so the three workflows reuse each other's builds on a branch. The CLI and `uitest-servers` (`ui-tests-bin-<hash>`) are built from `src/**`, `apps/simplex-chat/**`, the files under `tests/` that `uitest-servers` compiles (`UITestServers.hs`, `ChatClient.hs`, `NameResolver.hs`, `ChatTests/DBUtils.hs`, `ChatTests/DBUtils/**`), `cabal.project`, `simplex-chat.cabal`, `PRIVACY.md` and `build-bin.sh`. The workflow saves no Actions caches.
|
||||
|
||||
## Jobs
|
||||
|
||||
Without `tests`, each platform's tests run in four jobs at once; with `tests`, in one job per platform, named `selected`.
|
||||
|
||||
| Job | Does | Time |
|
||||
|---|---|---|
|
||||
| `plan` | `plan.sh`: the platforms and the shards of each; finds earlier builds | 10 s |
|
||||
| `desktop-core` | desktop-linux's `build-core.sh`, when the desktop tests run | skipped when reused, about 24 min not |
|
||||
| `android-core` | android-emu's `build-core.sh`, when the Android tests run | skipped when reused, about 27 min not |
|
||||
| `bin` | `build-bin.sh`: `exe:simplex-chat` and `exe:uitest-servers` with `-O0 -f uitest`, as `uitest.sh` builds them locally | skipped when reused; to be measured |
|
||||
| `desktop` | one job per shard: `uitest.sh` with the prebuilt CLI and servers (`UITEST_CLI`, `UITEST_SERVERS`) | to be measured |
|
||||
| `android` | one job per shard: the debug APK and the test APK, the emulator, `uitest-android.sh` | to be measured |
|
||||
|
||||
### Desktop shards
|
||||
|
||||
Without `tests`, the desktop tests run in four jobs, one per test package: `chat`, `groups`, `main`, and `settings-desktop` (`settings`, the desktop-only `desktop` tests and the harness's own tests). Fixed packages keep a shard's name telling which area failed, need no timings, and don't move tests between shards as tests are added. The plan job fails when a test package that no shard names appears, so a new package can't go untested; add it to `plan.sh`. In each job `uitest.sh` runs as many test JVMs as memory allows (at most 4).
|
||||
|
||||
`uitest.sh` runs the tests in their own user, PID and network namespaces (`unshare -rn --pid`), which Ubuntu 24.04 restricts: the job turns `kernel.apparmor_restrict_unprivileged_userns` off first, as android-emu's core build does for Nix, and checks that `unshare` works.
|
||||
|
||||
### Android
|
||||
|
||||
An emulator runs one test at a time, and the Android tests (about 450) run under ARM translation, likely too long for one job's 6 hours: without `tests` they are split by AndroidJUnitRunner's own sharding (`-e numShards 4 -e shardIndex <i>`), which assigns each test method to a shard by a hash of its name, the same in every pass of `uitest-android.sh`. Each shard job builds the APKs itself, since it compiles the host-side bridge anyway, whose classpath names files in the job's Gradle cache.
|
||||
|
||||
The emulator is android-emu's (`install-sdk.sh`, `emulator.sh`): Google APIs x86_64, API 34, with KVM, running the app's arm64-v8a code under ARM translation. The APKs are built as android-emu builds its APK (`-Pandroid.injected.build.abi=arm64-v8a`, `:android:assembleFossDebug`), with `:android:assembleFossDebugAndroidTest` and `:common:uitestBridgeClasspath` in the same Gradle run, before `install-sdk.sh` points `ANDROID_HOME` at an SDK that has only the emulator. Without `local.properties` the debug app's id is `chat.simplex.app` (`UITEST_APP_ID`). `uitest.sh --prepare-only` writes the peer template and `build/uitest/env.properties` that the host-side bridge reads, without the desktop build; `uitest-android.sh` then runs with `UITEST_SERIAL` set to the emulator and `ADB` to the SDK's adb, and forwards the bridge's and the servers' ports with `adb reverse`.
|
||||
|
||||
## Results
|
||||
|
||||
The job summaries list the results: per test class for desktop (`desktop-linux/test-summary.py` on the JUnit XML; known-issue tests are counted as skipped), and per class with the failed and crashed tests for Android (`android-summary.py` on `results.txt`). A job fails when a test fails or, on Android, crashes. The artifacts (7 days):
|
||||
|
||||
- `ui-tests-desktop-<shard>`: the HTML report and the JUnit XML (`common/build/{reports,test-results}/desktopUiTest`), and from the run directory the failure screenshots (`failures/`), the transcripts of the failed tests' CLI peers (`jvm-*/t<n>/peers/<name>/transcript.log`: a test's directory is kept only when it failed) and JVM crash logs;
|
||||
- `ui-tests-android-<shard>`: the run directory of `uitest-android.sh` (`results.txt`, `instrument-<pass>.txt`, `logcat.txt`, `timings.txt`, `failures/`, `bridge.log`), `emulator/emulator.log`, and the transcripts of the bridge's peers (all tests).
|
||||
|
||||
## Diagnosing a failure
|
||||
|
||||
Start from the failure message in the summary or the test report: a timed-out wait names what it waited for, the API calls still in flight and the first visible texts. Its screenshot in `failures/` is named after the JVM and the wait. The peers' transcripts show what each CLI peer printed, with timestamps, to tell an app failure from a peer or server one. The step log has each test's standard output. On Android, `instrument-<pass>.txt` has the stack traces, `logcat.txt` the app's log and the in-process watchdog's thread dumps, and `timings.txt` each test's phases; a test reported as `crashed` was running when the app process died, and the run resumed without it.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **ARM translation on Android**, as android-emu's README describes: the app runs slower than on a phone and can crash in the translator. `uitest-android.sh` resumes after a crash (at most 4 passes), and force-stops the app when the instrumentation prints nothing for 420 s (`UITEST_STALL_S`).
|
||||
- **Memory**: `uitest.sh` needs 3000 MB free and `uitest-android.sh` 4500 MB besides the emulator; the runners have 16 GB.
|
||||
- The test JVMs per desktop job follow free memory, not the runner's 4 CPUs.
|
||||
|
||||
## Scripts
|
||||
|
||||
- `plan.sh`: the plan job's choices and the shards of both platforms.
|
||||
- `build-bin.sh`, `install-bin.sh`: the CLI and `uitest-servers`, built and installed for `uitest.sh` (`UITEST_CLI`, `UITEST_SERVERS`).
|
||||
- `android-summary.py`: the Android job summary.
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# Prints the results of an Android UI test run of apps/multiplatform/scripts/uitest-android.sh (its results.txt), and
|
||||
# adds them to the job summary on GitHub.
|
||||
# Usage: scripts/ci/ui-tests/android-summary.py <run dir>
|
||||
|
||||
import collections
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
|
||||
run_dir = sys.argv[1]
|
||||
results = os.path.join(run_dir, "results.txt")
|
||||
if not os.path.isfile(results):
|
||||
sys.exit(f"no results.txt in {run_dir}")
|
||||
|
||||
outcomes = {"0": "passed", "-2": "failed", "-3": "skipped", "-4": "skipped", "crashed": "crashed"}
|
||||
columns = ["passed", "failed", "crashed", "skipped"]
|
||||
classes = collections.defaultdict(collections.Counter)
|
||||
problems = []
|
||||
with open(results) as f:
|
||||
for line in f:
|
||||
test, _, code = line.strip().rpartition(" ")
|
||||
if not test:
|
||||
continue
|
||||
outcome = outcomes.get(code, f"code {code}")
|
||||
cls = test.split("#")[0].removeprefix("chat.simplex.app.ui.")
|
||||
classes[cls][outcome if outcome in columns else "failed"] += 1
|
||||
if outcome not in ("passed", "skipped"):
|
||||
problems.append(f"{test.removeprefix('chat.simplex.app.ui.')}: {outcome}")
|
||||
|
||||
totals = collections.Counter()
|
||||
for counts in classes.values():
|
||||
totals.update(counts)
|
||||
rows = ["| Class | " + " | ".join(c.capitalize() for c in columns) + " |", "|---" * (len(columns) + 1) + "|"]
|
||||
for cls in sorted(classes):
|
||||
rows.append(f"| {cls} | " + " | ".join(str(classes[cls][c]) for c in columns) + " |")
|
||||
|
||||
summary = [f"### Android UI tests: {sum(totals.values())} tests, " + ", ".join(f"{totals[c]} {c}" for c in columns), ""]
|
||||
summary += rows if classes else ["No test finished."]
|
||||
if problems:
|
||||
summary += ["", "Failed or crashed:", ""] + [f"- {p}" for p in problems]
|
||||
shots = glob.glob(os.path.join(run_dir, "failures", "*"))
|
||||
if shots:
|
||||
summary += ["", f"{len(shots)} failure screenshots are in the `failures` directory of the run's artifact."]
|
||||
print("\n".join(summary))
|
||||
if os.environ.get("GITHUB_STEP_SUMMARY"):
|
||||
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
|
||||
f.write("\n".join(summary) + "\n")
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Builds the CLI and uitest-servers, the UI tests' peers and local servers, as apps/multiplatform/scripts/uitest.sh
|
||||
# builds them (-O0, flag uitest), and packs the two executables.
|
||||
# Usage: scripts/ci/ui-tests/build-bin.sh <output .tar.gz>
|
||||
# The configuration is build-core.sh's but without client_library, which makes uitest-servers unbuildable.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
mkdir -p "$(dirname "$1")"
|
||||
out="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"
|
||||
flags=(-O0 -f uitest)
|
||||
exes=(simplex-chat uitest-servers)
|
||||
|
||||
ghc --version
|
||||
cabal --version
|
||||
|
||||
cat > cabal.project.local << EOF
|
||||
ignore-project: False
|
||||
jobs: \$ncpus
|
||||
|
||||
package direct-sqlcipher
|
||||
flags: +openssl
|
||||
EOF
|
||||
|
||||
cabal build "${flags[@]}" "${exes[@]/#/exe:}"
|
||||
|
||||
pkg=$(mktemp -d)
|
||||
for exe in "${exes[@]}"; do
|
||||
# in a fresh project list-bin first prints git's output from syncing the source-repository-packages
|
||||
cp "$(cabal list-bin "${flags[@]}" "exe:$exe" | tail -n 1)" "$pkg"
|
||||
done
|
||||
ldd "$pkg"/*
|
||||
if ldd "$pkg"/* | grep 'not found'; then
|
||||
echo "error: the executables need these libraries"
|
||||
exit 1
|
||||
fi
|
||||
du -sh "$pkg"/*
|
||||
|
||||
tar -C "$pkg" -czf "$out" .
|
||||
ls -la "$out"
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Unpacks the CLI and uitest-servers from build-bin.sh into <dir>, checks that the runner has their libraries, and puts
|
||||
# UITEST_CLI and UITEST_SERVERS in GITHUB_ENV, where apps/multiplatform/scripts/uitest.sh uses them instead of building.
|
||||
# Usage: scripts/ci/ui-tests/install-bin.sh <bin .tar.gz> <dir>
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
mkdir -p "$2"
|
||||
dir=$(cd "$2" && pwd)
|
||||
tar -C "$dir" -xzf "$1"
|
||||
ls -la "$dir"
|
||||
if ldd "$dir/simplex-chat" "$dir/uitest-servers" | grep 'not found'; then
|
||||
echo "error: the executables need these libraries"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -n "${GITHUB_ENV:-}" ]; then
|
||||
{
|
||||
echo "UITEST_CLI=$dir/simplex-chat"
|
||||
echo "UITEST_SERVERS=$dir/uitest-servers"
|
||||
} >> "$GITHUB_ENV"
|
||||
fi
|
||||
Executable
+71
@@ -0,0 +1,71 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Reads the ui-tests workflow's inputs and writes to GITHUB_OUTPUT which platforms run, and the desktop and Android
|
||||
# shards as matrices.
|
||||
# Usage: PLATFORM=desktop|android|both TESTS='<classes or patterns>' scripts/ci/ui-tests/plan.sh
|
||||
# Without TESTS, the desktop tests run in one shard per test package; a test package no shard names fails the plan, so
|
||||
# a new package can't go untested. The Android tests run in android_shards shards of AndroidJUnitRunner's own sharding
|
||||
# (-e numShards -e shardIndex). With TESTS (comma- or space-separated), one job per platform runs them: as Gradle
|
||||
# --tests filters on desktop, with Class#method written Class.method, and as `am instrument -e class` on Android.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
out=${GITHUB_OUTPUT:-/dev/null}
|
||||
ui=apps/multiplatform/common/src
|
||||
roots=("$ui/uiTestShared/kotlin/chat/simplex/app/ui" "$ui/desktopTest/kotlin/chat/simplex/app/ui")
|
||||
android_shards=4
|
||||
shards='[
|
||||
{"shard": "chat", "tests": "chat.simplex.app.ui.chat.*"},
|
||||
{"shard": "groups", "tests": "chat.simplex.app.ui.groups.*"},
|
||||
{"shard": "main", "tests": "chat.simplex.app.ui.main.*"},
|
||||
{"shard": "settings-desktop", "tests": "chat.simplex.app.ui.settings.* chat.simplex.app.ui.desktop.* chat.simplex.app.ui.harness.*"}
|
||||
]'
|
||||
|
||||
case "${PLATFORM:-}" in
|
||||
desktop) desktop=true android=false ;;
|
||||
android) desktop=false android=true ;;
|
||||
both) desktop=true android=true ;;
|
||||
*)
|
||||
echo "::error::platform must be desktop, android or both, not \"${PLATFORM:-}\""
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
allowed='^[A-Za-z0-9_.*#, ]*$'
|
||||
if ! [[ "${TESTS:-}" =~ $allowed ]]; then
|
||||
echo "::error::tests may have only class names, patterns (* . # _), commas and spaces"
|
||||
exit 1
|
||||
fi
|
||||
read -r -a tests <<< "${TESTS//,/ }"
|
||||
|
||||
if [ "${#tests[@]}" -eq 0 ]; then
|
||||
covered=$(jq -r '[.[].tests | split(" ")[] | sub("^chat\\.simplex\\.app\\.ui\\."; "") | sub("\\.\\*$"; "")] | join(" ")' <<< "$shards")
|
||||
missing=$(find "${roots[@]}" -name '*Test.kt' | sed 's|.*/chat/simplex/app/ui/||' | cut -d/ -f1 | sort -u | while read -r pkg; do
|
||||
case " $covered " in
|
||||
*" $pkg "*) ;;
|
||||
*) echo "chat.simplex.app.ui.$pkg" ;;
|
||||
esac
|
||||
done)
|
||||
if [ -n "$missing" ]; then
|
||||
echo "::error::no desktop shard runs the tests in $(paste -sd ' ' <<< "$missing") (add a shard in scripts/ci/ui-tests/plan.sh)"
|
||||
exit 1
|
||||
fi
|
||||
matrix=$(jq -c . <<< "$shards")
|
||||
android_matrix=$(jq -cn --argjson n "$android_shards" '[range($n) | {shard: "\(. + 1)-of-\($n)", args: "-e numShards \($n) -e shardIndex \(.)"}]')
|
||||
else
|
||||
if [ "$android" = true ] && [[ "${tests[*]}" == *"*"* ]]; then
|
||||
echo "::error::Android runs whole class names (or Class#method), not patterns with *"
|
||||
exit 1
|
||||
fi
|
||||
desktop_tests="${tests[*]}"
|
||||
matrix=$(jq -cn --arg tests "${desktop_tests//#/.}" '[{shard: "selected", tests: $tests}]')
|
||||
android_classes=$(IFS=,; echo "${tests[*]}")
|
||||
android_matrix=$(jq -cn --arg classes "$android_classes" '[{shard: "selected", args: "-e class \($classes)"}]')
|
||||
fi
|
||||
|
||||
{
|
||||
echo "desktop=$desktop"
|
||||
echo "android=$android"
|
||||
echo "desktop-shards=$matrix"
|
||||
echo "android-shards=$android_matrix"
|
||||
} | tee -a "$out"
|
||||
Reference in new issue
Block a user