A lobby captures the microphone and the camera on the browser's devices
alone, with `createLocalMedia`, so that any backend can take them over:
`LocalMedia` hands out the same `LocalAudioMediaTrack` and
`LocalVideoMediaTrack` a call renders, with mute, device and facing
switches and the host's `VideoProcessor` run on the preview. Given to
`join` as `publish`, the tracks are moved rather than shared: the
publisher takes them over as it starts publishing, their lobby objects
are finished, and the LiveKit backend adopts the device tracks as its
own and the processor with its running output, so the join captures
nothing and initialises nothing again.
Also here, found on the way: a track already on the requested device is
left alone (LiveKit would restart it, always on Firefox, which echoes no
device id in `getConstraints()`), identical captures in flight are
coalesced (React's strict mode mounts twice), and every operation on a
processor takes its turn, since a stopped track tears its processor down
without waiting for the next track's init.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Port of edf1feae from johannes/rtc-remote-publish onto the SDK split,
where
the MatrixRTC logic lives in sdk/ rather than src/state/CallViewModel/.
Where the original lands:
- getLocalTransport's remote fallback: the LiveKit backend's new
findRemotePublishTransport (sdk/src/media-backend/livekit/
RemotePublishTransport.ts), which tries the candidates from the
js-sdk's
getRemoteLivekitPublishCandidates oldest first and keeps the first
whose
homeserver issues a token with the publish grant.
discoverLocalTransport
asks the backend for it after the host's fallback, with a snapshot of
the
session's memberships from RTCParticipation, and throws
MatrixRTCTransportMissingError where that finds nothing too. Transport
discovery stays backend neutral: the MSC4195 token check is the
backend's.
- LocalMember's "no delegation on a remote SFU" and the custom
transport's
server_name: LivekitBackend.prepareLocalTransport locates the
transport on
the homeserver it names (serverNameOfTransport) and skips the
delegation
probe where that is not our own.
- SessionBehaviors' membership.getTransportServerName: the same helper
in
LivekitBackend.mediaFor.
- decodeSFUJWTPayload, SFUJWTPayload.video.canPublish and the exported
isCsApiUnavailable: sdk/src/media-backend/livekit/auth/.
- DeveloperSettingsTab, the widget capabilities, the locale string and
the
js-sdk pin (9405287c, 44.0.0): applied as is. The error boundary
snapshots follow the string.
Tests: the LocalTransport cases go to RemotePublishTransport.test.ts and
LocalTransport.test.ts, the LocalMember remote case and the
SessionBehaviors
server-name case to LivekitBackend.test.ts, plus the memberships
snapshot in
RTCParticipation.test.ts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The publisher is driven through a real LocalParticipant whose device and
signalling calls are faked, so that LiveKit's own pending-publication
bookkeeping is part of what is tested: the cases ported from Element
Call's Publisher tests (one permission prompt for both devices, nothing
created while muted, the upstream held back until the member publishes,
a mute that lands while the track is still being created) plus the
publish, unpublish and destroy paths.
The local media runs on a fake connection map: publishing before and
after the connection exists, republishing on a new connection, mutes
remembered across connections, the first publish error kept, and the
output device applied to the current room and the ones to come.
The participation runs on a fake media backend: the join options, the
state machine through connecting, connected, reconnecting, failed and
left, publishing only while joined, the delayed leave delegation, the
member ids per mode, the key replay and the data channel's sender check.
The OpenID flow moves to src/livekit/auth/openID.ts with the SFU config
types in a sibling file, takes an options object, and replaces the
MatrixRTC mode with the role we take on the transport plus the session's
slot ID instead of a hard-coded m.call#ROOM.
This changes behaviour: publishers on a livekit_service_url transport
always use the legacy /sfu/get endpoint, so that subscribers can predict
their user:device participant identity, and subscribers always use the
default /get_token endpoint. The MatrixRTC mode no longer influences which
endpoint is used.
On the SDK this lands as sdk/src/media-backend/livekit/auth/ with the connection map, the discriminated connection state and the client-server API flow of the following commits of element-hq/element-call#4331, which rewrite the same files:
- Model ConnectionState as a discriminated union
- Show the whole transport in debug UI and logs (SDK part)
- Let the publishing Connection authenticate itself
- Reduce ConnectionManager to a ConnectionMap
- Use the unstable LiveKit transport type
- Identify transports by server name as well as URLs
- Get SFU tokens and delegate delayed leaves through the MSC4195 CS API
- Fall back to get_token as subscriber on widget API errors
- Share a connection between homeservers using the same JWT service
- Make the auth choice mode independent again
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
(cherry picked from commit adb380cd9ed108856e42d992fc03ced8c44f1e56)
A member authenticated through the legacy JWT endpoint publishes under a
`user:device` identity, while one authenticated through the default
endpoint publishes under a hashed identity. Both are listed in the
membership's backend identities, so match against the whole list rather
than only the primary one.
On the SDK this is RTCMember.backendIdentities, which the LiveKit backend matches media and data packets against; the harness keys its tiles by the membership id, which the backend identity no longer tells apart.
(cherry picked from commit c9aad2a2b3279782d101c9bc6c80af0417ccbb92)
The js-sdk at this commit also renamed MatrixRTCSession.leaveRoomSession to leave, which the SDK follows.
(cherry picked from commit 405348945febc1df10edf9c00748f1edd6b39e71)
The session speaks MatrixRTC and sees media through MediaBackend in
sdk/src/backend/api.ts. Everything LiveKit lives in
sdk/src/backend/livekit/,
the only place in the SDK that imports livekit-client; the import
boundary rule
now enforces that outside sdk/src/backend/ the LiveKit packages appear
in type
imports only.
The seam sits above the connection manager: one connection per transport
url
is the LiveKit multi-SFU strategy, not a MatrixRTC concept, so a
cascading SFU
or full mesh replaces the folder, not the session. A backend prepares
the local
transport the session discovered (the token exchange and the delegation
probe,
reported as canDelegateDelayedLeave), hands the delayed leave over per
delay
id, gives the local member one publish switch plus publish/unpublish,
builds a
remote member's media from its membership and registers that
membership's
transport for as long as the member lives, and reports its connections
for the
transport registry, which keys on the raw transport serialised with
sorted
keys. The backend's context carries no Matrix client (a backend picks
what it
calls in its own options), no membership list, and no js-sdk session:
media
keys arrive as a stream replayed on subscribe, and MatrixKeyProvider
subscribes
to it. The MatrixRTC mode no longer reaches LiveKit; it becomes a
tokenEndpoint
choice made where the default backend is built.
Two behaviour changes ride along. Publishing is one boolean, joined and
homeserver reachable, where before a sync hiccup after leave() resumed
the
upstream of tracks that were meant to stay silent. And
ResolvedConnection keeps
one ResolvedTransport object per fetch, so a state change no longer
reads as a
change of what was fetched.
MatrixRTCClientOptions gains backend?: MediaBackendFactory, defaulting
to
createLivekitBackend built from the client options.
authenticateWithTransport
takes tokenEndpoint instead of matrixRTCMode.
Unit tests for tracks$ and trackBySource$, an end-to-end check of isActive$ through a mute cycle (skipped on Chromium, whose fake microphone the SFU never hears), the architecture document brought in line with the track list, and the changed behaviour recorded in the plan.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
