From cf58bf6d9e1bc6ac1e26686a2f50cdae2e8edd81 Mon Sep 17 00:00:00 2001 From: shum Date: Thu, 22 Oct 2026 12:26:17 +0000 Subject: [PATCH] docs: describe native desktop calls --- apps/multiplatform/product/README.md | 2 +- apps/multiplatform/product/flows/calling.md | 29 +++++++-------------- apps/multiplatform/product/glossary.md | 6 ++--- apps/multiplatform/product/rules.md | 6 ++--- apps/multiplatform/product/views/call.md | 19 +++++++------- apps/multiplatform/spec/impact.md | 3 ++- apps/multiplatform/spec/services/calls.md | 29 +++++++++++---------- 7 files changed, 42 insertions(+), 52 deletions(-) diff --git a/apps/multiplatform/product/README.md b/apps/multiplatform/product/README.md index 3dff974366..66b58829f8 100644 --- a/apps/multiplatform/product/README.md +++ b/apps/multiplatform/product/README.md @@ -118,7 +118,7 @@ End-to-end encrypted audio and video communication. | Incoming call view | Dedicated UI for incoming call notifications | `common/src/commonMain/kotlin/chat/simplex/common/views/call/IncomingCallAlertView.kt` | | Android CallService | Foreground service for active calls on Android | `android/src/main/java/chat/simplex/app/CallService.kt` | | Android CallActivity | Dedicated Activity for call UI on Android | `android/src/main/java/chat/simplex/app/views/call/CallActivity.kt` | -| Desktop inline calls | Browser-based WebRTC rendered inline in desktop window | `common/src/commonMain/kotlin/chat/simplex/common/views/call/CallView.kt` | +| Desktop inline calls | Native WebRTC rendered inline in desktop window | `common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt` | ### 5. Privacy & Security diff --git a/apps/multiplatform/product/flows/calling.md b/apps/multiplatform/product/flows/calling.md index d29c1d4478..7e7ecc1ad4 100644 --- a/apps/multiplatform/product/flows/calling.md +++ b/apps/multiplatform/product/flows/calling.md @@ -8,7 +8,7 @@ SimpleX Chat supports audio and video calls using WebRTC, with signaling deliver The architecture differs by platform: - **Android**: Calls run in a dedicated `CallActivity` (separate from `MainActivity`) with a `WebView` hosting the WebRTC JavaScript. A foreground `CallService` keeps the process alive and shows a persistent notification. -- **Desktop**: Calls open the system browser pointed at a local NanoHTTPD/NanoWSD embedded server on `localhost:50395`, which serves the WebRTC HTML/JS and communicates with the app via WebSocket. +- **Desktop**: Calls run in-process in `NativeCallEngine` (webrtc-java), shown full-window in the main window. Both platforms share a common signaling flow through the Haskell core API. @@ -133,7 +133,7 @@ Reconnection: Connected -> Reconnecting -> Connected | Ended 4. `ChatModel.showCallView` is set to `false`. 5. `ChatModel.activeCall` is set to `null`. 6. On Android, `CallService` is stopped and the `WebView` is destroyed. -7. On Desktop, `WCallCommand.End` is sent to the browser via WebSocket, and the NanoWSD server is stopped. +7. On Desktop, the call view leaves composition and `NativeCallEngine` closes the peer connection and releases devices. --- @@ -165,24 +165,13 @@ Reconnection: Connected -> Reconnecting -> Connected | Ended --- -## 6. Desktop-Specific: Browser-Based WebRTC +## 6. Desktop-Specific: Native WebRTC -### 6.1 NanoWSD Embedded Server - -1. When a call starts, `startServer(onResponse)` creates a `NanoWSD` server on `localhost:50395`. -2. The server serves static WebRTC HTML/JS from bundled resources at `/assets/www/desktop/call.html`. -3. The system browser is opened to `http://localhost:50395/simplex/call/`. - -### 6.2 WebSocket Communication - -1. The browser page connects back via WebSocket to the same `localhost:50395` server. -2. Commands from the app to the browser are serialized as `WVAPICall(corrId, command)` JSON. -3. Responses from the browser arrive as `WVAPIMessage(corrId, resp, command)` JSON. -4. The `WebRTCController` composable manages the command queue: - - Collects commands from `ChatModel.callCommand` (a `SnapshotStateList`). - - Sends them to the browser via the WebSocket connection. - - Processes responses through the same `WCallResponse` handling as Android. -5. On dispose, `WCallCommand.End` is sent, the server is stopped, and connections are cleared. +1. When the call view is shown, `NativeCallController` creates a `NativeCallEngine`. +2. Commands from `ChatModel.callCommand` (a `SnapshotStateList`) are passed to the engine in order. +3. The engine reports `WVAPIMessage(corrId, resp, command)` and the responses go through the same `WCallResponse` handling as Android. +4. The engine interoperates with `call.ts` (Android) and iOS peers: four transceivers (mic, camera, screen audio, screen video), VP8, AES-GCM frame encryption in the `call.ts` layout, SDP and ICE candidates as LZString-compressed JSON. +5. On dispose, the engine closes the peer connection and releases camera, screen capture and audio devices. --- @@ -227,4 +216,4 @@ Camera switching (front/back) is done via `WCallCommand.Camera(VideoCamera.User | `WCallResponse` | `views/call/WebRTC.kt` | Responses from WebRTC: Capabilities, Offer, Answer, Ice, Connection, Connected, End | | `CallActivity` | `android/.../views/call/CallActivity.kt` | Android Activity hosting the call UI and WebView | | `CallService` | `android/.../CallService.kt` | Android foreground Service for call persistence | -| `NanoWSD` | `desktopMain/.../views/call/CallView.desktop.kt` | Desktop embedded HTTP+WebSocket server | +| `NativeCallEngine` | `desktopMain/.../views/call/NativeCallEngine.kt` | Desktop native WebRTC engine (webrtc-java) | diff --git a/apps/multiplatform/product/glossary.md b/apps/multiplatform/product/glossary.md index 15ed40d3a1..8c80ddae34 100644 --- a/apps/multiplatform/product/glossary.md +++ b/apps/multiplatform/product/glossary.md @@ -335,10 +335,10 @@ An Android foreground `Service` that keeps the call alive when the app is in the *See:* `android/src/main/java/chat/simplex/app/CallService.kt` -### Desktop: Browser-based WebRTC via NanoWSD -On Desktop, calls are implemented by opening the system browser to a locally-hosted WebSocket server. A `NanoHTTPD`/`NanoWSD` server runs on `localhost:50395`, serving the WebRTC call page and communicating with the Kotlin app via WebSocket messages. Commands are sent as JSON-serialized `WVAPICall` objects; responses are parsed as `WVAPIMessage` objects. +### Desktop: Native WebRTC via webrtc-java +On Desktop, calls run in-process in `NativeCallEngine`, built on webrtc-java (native libwebrtc). It consumes the same `WCallCommand` commands and reports the same `WCallResponse` responses as the Android WebView, so the shared state machine and peers on other platforms are unchanged. -*See:* `common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt` +*See:* `common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt` ### ICE Servers STUN/TURN servers used for WebRTC NAT traversal. Configurable via `AppPreferences.webrtcIceServers`. The relay policy (`AppPreferences.webrtcPolicyRelay`) controls whether calls must use TURN relays (for IP privacy) or can attempt direct connections. diff --git a/apps/multiplatform/product/rules.md b/apps/multiplatform/product/rules.md index 90a2dadada..b4bf60553e 100644 --- a/apps/multiplatform/product/rules.md +++ b/apps/multiplatform/product/rules.md @@ -242,9 +242,9 @@ This document specifies invariants enforced by the Android and Desktop (Kotlin/C - Lock-screen behavior controlled by `AppPreferences.callOnLockScreen` (DISABLE / SHOW / ACCEPT). **Desktop enforcement:** -- Calls run in the system browser via the NanoWSD WebSocket server on `localhost:50395`. -- The `WebRTCController` composable manages the WebSocket lifecycle. -- On dispose, `WCallCommand.End` is sent and the server is stopped. +- Calls run in-process in `NativeCallEngine` (webrtc-java), one engine per call. +- `NativeCallController` creates the engine and passes it the `callCommand` queue. +- On dispose, the engine closes the peer connection and releases devices. **Location:** - `common/src/commonMain/kotlin/chat/simplex/common/views/call/CallManager.kt` diff --git a/apps/multiplatform/product/views/call.md b/apps/multiplatform/product/views/call.md index 1d4f3b881b..31a6046fe8 100644 --- a/apps/multiplatform/product/views/call.md +++ b/apps/multiplatform/product/views/call.md @@ -4,7 +4,7 @@ ## Purpose -Make and receive end-to-end encrypted audio and video calls over WebRTC. The implementation differs significantly between Android (WebView-based with `CallActivity` and PiP support) and Desktop (browser-based WebRTC via NanoHTTPD server on localhost). +Make and receive end-to-end encrypted audio and video calls over WebRTC. The implementation differs significantly between Android (WebView-based with `CallActivity` and PiP support) and Desktop (native WebRTC in-process via webrtc-java, shown in the main window). ## Route / Navigation @@ -18,11 +18,11 @@ Make and receive end-to-end encrypted audio and video calls over WebRTC. The imp | Aspect | Android | Desktop | |---|---|---| -| WebRTC host | `WebView` with `WebViewAssetLoader` serving local assets | NanoHTTPD server on `localhost:50395` opened in system browser | -| Call activity | `CallActivity` (separate Android Activity) with lifecycle management | Inline composable with `WebRTCController` | +| WebRTC host | `WebView` with `WebViewAssetLoader` serving local assets | `NativeCallEngine` (webrtc-java, native libwebrtc) | +| Call activity | `CallActivity` (separate Android Activity) with lifecycle management | Full-window composable over the app, collapsible to the floating call avatar | | PiP support | Native Android PiP via `CallActivity` | Not supported | -| Audio management | `CallAudioDeviceManager` with Android `AudioManager`, proximity wake lock | System browser audio routing | -| WebSocket | N/A | `NanoWSD` WebSocket server for bidirectional WebRTC signaling | +| Audio management | `CallAudioDeviceManager` with Android `AudioManager`, proximity wake lock | System default microphone and speaker | +| Screen sharing | Not supported | Screen or window picker | ## Page Sections @@ -58,11 +58,10 @@ Sound: `SoundPlayer.start()` plays ringtone while banner is visible (unless call | Element | Description | |---|---| -| NanoHTTPD server | HTTP server on `localhost:50395` serving `call.html` and assets | -| NanoWSD WebSocket | WebSocket endpoint for bidirectional signaling between Kotlin and browser JS | -| `WebRTCController` | Processes `WCallCommand`/`WCallResponse` messages via `chatModel.callCommand` channel | -| Browser launch | `LocalUriHandler.openUri("http://localhost:50395/call.html")` opens system browser | -| Connection list | `connections: ArrayList` tracks active WebSocket connections | +| `NativeCallController` | Creates `NativeCallEngine` for the call and passes it commands from the `chatModel.callCommand` queue | +| `NativeCallEngine` | Native WebRTC with `call.ts` semantics: transceivers, VP8, frame encryption, ICE batching, peer media detection | +| `ActiveCallLayout` | Remote video or avatar, local camera preview, call state and encryption info | +| Controls | Mic, peer audio, camera, screen share picker, camera picker, hang up; back button collapses the view | ### WebRTC Signaling Flow diff --git a/apps/multiplatform/spec/impact.md b/apps/multiplatform/spec/impact.md index 6ae1e8d9f6..4c35e18d6d 100644 --- a/apps/multiplatform/spec/impact.md +++ b/apps/multiplatform/spec/impact.md @@ -442,7 +442,8 @@ Path prefix: `common/src/desktopMain/kotlin/chat/simplex/common/` | `platform/Log.desktop.kt` | — | Low | Desktop log output | | `platform/Modifier.desktop.kt` | — | Low | Desktop modifier extensions | | `platform/Resources.desktop.kt` | — | Low | Desktop resource loading | -| `views/call/CallView.desktop.kt` | PC17 | Medium | Desktop WebView-based WebRTC call | +| `views/call/CallView.desktop.kt` | PC17 | Medium | Desktop call view and state machine | +| `views/call/NativeCallEngine.kt` | PC17 | High | Desktop native WebRTC engine (webrtc-java) | | `views/chat/ComposeView.desktop.kt` | PC4, PC10 | Low | Desktop compose view (drag-and-drop, paste) | | `views/chat/SendMsgView.desktop.kt` | PC4 | Low | Desktop send shortcut (Enter key handling) | | `views/chat/ComposeEmojiPicker.desktop.kt` | PC4 | Low | Desktop emoji picker popup (data from `resources/emoji/emoji.txt`, search) | diff --git a/apps/multiplatform/spec/services/calls.md b/apps/multiplatform/spec/services/calls.md index a46de4f45c..b0a8eb0cdc 100644 --- a/apps/multiplatform/spec/services/calls.md +++ b/apps/multiplatform/spec/services/calls.md @@ -12,7 +12,7 @@ ## Executive Summary -WebRTC calling in SimpleX Chat operates over SMP (SimpleX Messaging Protocol) for signaling, with platform-specific WebRTC media implementations. Android uses a WebView-based approach with a dedicated `CallActivity` and foreground `CallService`, while Desktop opens the system browser and communicates via a NanoWSD WebSocket server on localhost. Both platforms share a common `CallManager` for call lifecycle and a `CallState` enum for state tracking. Call commands and responses are serialized as JSON and exchanged between the native layer and the WebRTC JavaScript layer. +WebRTC calling in SimpleX Chat operates over SMP (SimpleX Messaging Protocol) for signaling, with platform-specific WebRTC media implementations. Android uses a WebView-based approach with a dedicated `CallActivity` and foreground `CallService`, while Desktop runs native WebRTC in-process through webrtc-java. Both platforms share a common `CallManager` for call lifecycle and a `CallState` enum for state tracking. Both use the same `WCallCommand`/`WCallResponse` protocol: Android exchanges it as JSON with the WebRTC JavaScript layer, Desktop passes it to `NativeCallEngine`. --- @@ -21,7 +21,7 @@ WebRTC calling in SimpleX Chat operates over SMP (SimpleX Messaging Protocol) fo Call signaling uses the same SMP protocol on all platforms -- call invitations, offers, answers, ICE candidates, and status updates flow through the chat backend via API commands. The WebRTC media plane, however, is implemented differently per platform: - **Android**: WebView loads `call.html` from bundled assets; a `@JavascriptInterface` bridge (`WebRTCInterface`) forwards JSON messages between Kotlin and JavaScript. -- **Desktop**: The system browser opens `http://localhost:50395/simplex/call/`; a NanoWSD HTTP+WebSocket server serves `call.html` from classpath resources and relays JSON commands/responses over WebSocket. +- **Desktop**: `NativeCallEngine` handles the same commands with webrtc-java (native libwebrtc), interoperating with `call.ts` and iOS peers: same transceiver layout, VP8, frame encryption format and SDP/ICE encoding. Both platforms share the [`CallManager`](../../common/src/commonMain/kotlin/chat/simplex/common/views/call/CallManager.kt) class (119 lines), which orchestrates incoming call acceptance, call ending, and notification management. @@ -116,19 +116,19 @@ The `actual` platform implementation of `ActiveCallView()` and supporting compos ## 4. Desktop Implementation -### 4.1 CallView.desktop.kt (308 lines) +### 4.1 CallView.desktop.kt and NativeCallEngine.kt -[`CallView.desktop.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt) +[`CallView.desktop.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt) shows the call in the main window; [`NativeCallEngine.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt) runs WebRTC in-process: -Desktop calls run WebRTC in the system browser, not an embedded WebView: - -- **NanoWSD server** ([line 215](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L215)): `startServer()` creates a `NanoWSD` instance bound to `localhost:50395`. If that port is already in use it falls back to an OS-assigned free port (`port 0`); `WebRTCController` reads `server.listeningPort` for the browser URL. The server serves `call.html` from JAR resources at `/assets/www/desktop/call.html` for the path `/simplex/call/` when the request includes a valid `token` query parameter. All other paths serve resources from `/assets/www/`. -- **Call token** ([line 271](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L271)): `newCallServerToken()` returns 32 random bytes, base64url-encoded. `hasValidCallServerToken()` ([line 277](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L277)) checks the `token` query parameter. -- **WebSocket upgrade** ([line 238](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L238)): Each server instance accepts one WebSocket upgrade. The upgrade request must include `Origin: http://localhost:` and a valid token. Other upgrade requests receive `401 Unauthorized`. -- **WebSocket communication** ([line 283](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L283)): `MyWebSocket` handles WebSocket frames from the browser. `onMessage` deserializes JSON into `WVAPIMessage` and forwards to the response handler. `onClose` triggers `WCallResponse.End`. -- **WebRTCController** ([line 157](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L157)): Generates the call token, starts the server, then opens `http://localhost:/simplex/call/?token=` (normally `50395`) via `LocalUriHandler`. Processes `WCallCommand` queue by sending JSON over the WebSocket. On dispose, sends `WCallCommand.End` and stops the server. -- **SendStateUpdates** ([line 141](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L141)): Sends `WCallCommand.Description` with call state and encryption info text to the browser for display. -- **ActiveCallView** ([line 32](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L32)): Handles `WCallResponse` messages identically to Android (same state machine), plus a `WCallCommand.Permission` message on `Capabilities` error for browser permission denial guidance. +- **ActiveCallView** ([line 56](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L56)): Handles `WCallResponse` messages with the same state machine as Android. A `Capabilities` error (no microphone) shows an alert and ends the call. +- **NativeCallController** ([line 169](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L169)): Creates the engine for the call, drains the `chatModel.callCommand` queue into it, and closes it on dispose. +- **ActiveCallLayout** ([line 200](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt#L200)): Full-window call view over the app: remote screen or camera video, local camera preview, state and encryption info, and controls for mic, peer audio, camera, screen share (screen or window picker), camera picker and hang up. The back button collapses the view to the floating call avatar, which expands it again. +- **NativeCallEngine** ([line 29](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt#L29)): Processes `WCallCommand` on a single thread and reports `WCallResponse` with `call.ts` semantics: four transceivers (mic, camera, screen audio, screen video), VP8 preferred, ICE candidates batched as in `call.ts`, connection state and peer media changes reported. +- **Audio sending** ([line 305](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt#L305)): Device-captured audio reaches every audio sender, so the mic is muted and the screen audio sender silenced by deactivating the sender's encoding, which stops RTP. +- **Peer media** ([line 545](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt#L545)): A received source is reported muted after 3 seconds without received bytes, as in `call.ts`. +- **Reconnection** ([line 442](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt#L442)): As in `call.ts`, a connected call that loses its connection is reported as `"reconnecting"`. The party that sent the initial offer restarts ICE every 10 seconds and sends the restart offer in `WCallResponse.Ice` with `session`; the other party answers it the same way. The call ends as `"disconnected"` if it does not reconnect in 30 seconds. +- **Frame encryption** ([`CallFrameCipher.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallFrameCipher.kt)): AES-256-GCM in the `call.ts` frame layout, applied as encoded-frame transforms on senders and receivers. +- **Audio devices**: Calls use the system default microphone and speaker. webrtc-java selects audio devices by GUID, which the Linux and macOS backends of libwebrtc do not report. --- @@ -171,7 +171,8 @@ An in-app notification banner shown when a call invitation arrives while the app |---|---|---|---| | `CallView.kt` | [`common/src/commonMain/.../views/call/CallView.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/views/call/CallView.kt) | 28 | `expect fun ActiveCallView()`, delivery receipt waiting | | `CallView.android.kt` | [`common/src/androidMain/.../views/call/CallView.android.kt`](../../common/src/androidMain/kotlin/chat/simplex/common/views/call/CallView.android.kt) | 891 | Android WebView WebRTC, overlay, permissions | -| `CallView.desktop.kt` | [`common/src/desktopMain/.../views/call/CallView.desktop.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt) | 308 | Desktop browser WebRTC via NanoWSD | +| `CallView.desktop.kt` | [`common/src/desktopMain/.../views/call/CallView.desktop.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/CallView.desktop.kt) | 400 | Desktop call view and state machine | +| `NativeCallEngine.kt` | [`common/src/desktopMain/.../views/call/NativeCallEngine.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/views/call/NativeCallEngine.kt) | 729 | Desktop native WebRTC engine (webrtc-java) | | `CallActivity.kt` | [`android/src/main/java/.../views/call/CallActivity.kt`](../../android/src/main/java/chat/simplex/app/views/call/CallActivity.kt) | 472 | Android call Activity, PiP, lock screen | | `CallService.kt` | [`android/src/main/java/.../CallService.kt`](../../android/src/main/java/chat/simplex/app/CallService.kt) | 207 | Android foreground service for calls | | `CallManager.kt` | [`common/src/commonMain/.../views/call/CallManager.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/views/call/CallManager.kt) | 119 | Call lifecycle management |