diff --git a/.gitignore b/.gitignore index 6db1dab94..cc006912e 100644 --- a/.gitignore +++ b/.gitignore @@ -17,5 +17,4 @@ livekit-server # Dependency directories (remove the comment below to include it) # vendor/ -.idea/ bin/ diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 000000000..73f69e095 --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,8 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Datasource local storage ignored files +/dataSources/ +/dataSources.local.xml +# Editor-based HTTP Client requests +/httpRequests/ diff --git a/.idea/livekit-server.iml b/.idea/livekit-server.iml new file mode 100644 index 000000000..a858f8f8a --- /dev/null +++ b/.idea/livekit-server.iml @@ -0,0 +1,10 @@ + + + + + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 000000000..9c32c0523 --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/protoeditor.xml b/.idea/protoeditor.xml new file mode 100644 index 000000000..8af70d282 --- /dev/null +++ b/.idea/protoeditor.xml @@ -0,0 +1,17 @@ + + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 000000000..94a25f7f4 --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/.idea/watcherTasks.xml b/.idea/watcherTasks.xml new file mode 100644 index 000000000..cb1a78370 --- /dev/null +++ b/.idea/watcherTasks.xml @@ -0,0 +1,8 @@ + + + + + + + \ No newline at end of file diff --git a/docs/protocol.md b/docs/protocol.md index df9e19001..a35a25021 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -1,24 +1,57 @@ -# Signaling Protocol +# LiveKit RTC Protocol -LiveKit uses a websocket connection for signaling. Each client should start the RTC session by connecting to `/rtc`. +LiveKit uses a WebSocket connection for signaling. Each client should start the RTC session by connecting to `/rtc`. Once connected, LiveKit uses protocol buffers to exchange messages between the server and client. The messages are detailed in [rtc.proto](../proto/rtc.proto). Client always sends `SignalRequest` and will receive `SignalResponse` from the server. + +Like with other requests to LiveKit, the WebSocket connection needs to be [authenticated](authentication.md). ## Joining a room +When the WebSocket connection is established, server will first provide information about the joined room, and waits for client to perform the handshake. Flow: + +1. server sends a `JoinResponse`, which includes room information, the current participant's data, and information about other participants in the room. +2. client sends `offer`, and sets its local description. +3. server accepts connection and sends an `answer`. +4. ICE connectivity is established +5. server notifies other participants of the new participant +6. server subscribes new client to existing tracks in the room + ## Handling negotiations -Negotiation is how WebRTC makes changes to the current communication, including adding/removing tracks. In WebRTC, either side could initiate the change and it makes synchronization a challenge. +Negotiation is use to describe the offer/answer process in WebRTC. After the initial offer/answer process to establish the connection, negotiations are needed whenever track changes are made to the current session. Because WebRTC is a peer to peer protocol, negotiations can be initiated by either party as needed. -In LiveKit, we've added a layer of synchronization so that negotiations are more deterministic. The server becomes the place to control who should be issuing offers in a negotiation cycle. When the client wants to issue an offer to the server _after_ the initial connection is made, this is the flow +This creates a synchronization headache, as if both sides are initiating negotiations at the exact same time, it creates a race condition in which the peers would receive unexpected responses from the other side. This is called [glare](https://tools.ietf.org/agenda/82/slides/rtcweb-10.pdf). + +In LiveKit, we've added a layer of synchronization so that negotiations are more deterministic. The server is the authority determining who should be issuing offers in a negotiation cycle. When the client wants to issue an offer to the server, it needs to follow this flow. 1. client's onnegotiationneeded callback is triggered -2. client sends server a `NegotiationRequest` +2. client sends server a `negotiate` (`NegotiationRequest`) 3. server determines when it's ready for a new offer, and notifies client with `NegotiationResponse` -4. when client receives `NegotiationResponse`, it generates a new offer and sends it as an `offer` in the form of `SessionDescription`. +4. when client receives `negotiate` (`NegotiationResponse`), giving it permission to send an offer. Generates a new offer and sends it. 5. server accepts the offer and sends an `answer` 6. client calls `setRemoteDescription` with that session description. now the negotiation is complete and track changes have been applied. -## Subscribing to tracks +## Receiving subscribed tracks + +When the client is subscribed to other participants tracks, it needs to coordinate the metadata (sent in `ParticipantInfo.tracks`) with the WebRTC `ontrack` callback. For a audio or video track, `TrackInfo.sid` will match `MediaStreamTrack.id`. For a data track, `TrackInfo.sid` will match `RTCDataChannel.id.toString()`. ## Publishing tracks + +To publish a track, client needs to first notify server with track metadata before actually adding the track. + +1. client sends `add_track` with the `cid` field set to the client generated id for the track. (`MediaStreamTrack.id` or name of data track) +2. server adds it to a list of pending tracks and sends back `track_published` +3. client receives track_published, then: + a. for a audio/video track, client adds the `MediaStreamTrack` to its `PeerConnection` + b .for a data track, create the channel with `PeerConnection.createDataChannel` + +The above sequence will trigger `onnegotiationneeded` + +## Muting/Unmuting tracks + +Muting the track requires setting `MediaStreamTrack.enabled` to false, and then notifying the server with a `mute` message. + +Disabling the media track simply will not remove it from the stream, but to [send packets with 0 values periodically](https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamTrack/enabled). Because this will be a waste of bandwidth, LiveKit has optimizations to skip packet forwarding to other participants when the track is muted. + +Follow similar steps for unmuting diff --git a/proto/rtc.proto b/proto/rtc.proto index cf3bfd1b5..4d3e26ae8 100644 --- a/proto/rtc.proto +++ b/proto/rtc.proto @@ -7,7 +7,7 @@ import "model.proto"; message SignalRequest { oneof message { - // participant joining initially + // participant joining initially, and during negotiations SessionDescription offer = 1; // participant responding to server-issued offers SessionDescription answer = 2;