Added a new manager to handle all subscription needs. Implemented using reconciler pattern. The goals are:
improve subscription resilience by separating desired state and current state
reduce complexity of synchronous processing
better detect failures with the ability to trigger full reconnect
* add prometheus stats for rtt/jitter/packet loss
* add track source to metrics
* better packet loss bins
* add track type to metrics
* remove source from AnalyticsStat
* regenerate telemetry service fake
* compute loss from per stream packet count
Related to livekit/protocol#273
This PR adds:
- ParticipantResumed - for when ICE restart or migration had occurred
- TrackPublishRequested - when we initiate a publication
- TrackSubscribeRequested - when we initiate a subscription
- TrackMuted - publisher muted track
- TrackUnmuted - publisher unmuted track
- TrackPublish/TrackSubcribe events will indicate when those actions have been successful, to differentiate.
* Fix rtcp lost for downtrack used incorrect buffer factory
In buffer factory change(#1173), every pariticipant has its own
buffer factory, can't use publisher's bufferfactory to create
DownTrack
* clean code
* Cache RTPStats and seed on re-use
When a cached down track is re-used, RTPStats was not cached.
This caused sender reports getting out-of-sync with the remote side.
Cache RTPStats and seed it on re-use.
* staticcheck
* Start RTCP workers after peer connection connects
* Move more things into transport module
* Start RTCP workers only on connected
* Test needs PeerConnection() method
* adjust comment
* Prevent track subscriptions/adding receivers after close
With subscribe/unsubscribe queuing, a subscribe may be
attempted after a call to `RemoveAllSubscribers`.
So, renaming `RemoveAllSubscribers` to `InitiateClose`
and maintaining state that track is in the process of closing.
* Mime specific remove
* Remove unused error
* do not add receiver when closing
- Do not update jitter on padding only packet.
Padding only packet may not have proper timestamp.
If it does, it probably has the time stamp of the
last packet with payload. That will also affect
jitter calculation, i. e. wall clock time is moving,
but RTP time is the same.
- Do not send `onMaxLayer` changed on bind.
It was probably racing with update when max layer
is updated when adaptive stream is off. There is
no need to send that update as the default would
be OFF. It will be enabled when adaptive stream
subscription turns it on or when max layer is
set when down track bind happens and adaptive stream
is off.
* WIP commit
* Refactor media loss proxy
* Use DynacastQuality and MediaLossProxy from MediaTrack
* fix test
* Remove unused param
* Remove unused interfaces
* Move interface methods to local
* Split out DynacastManager
* have to add codec to dynacast manager
* RUnlock
* fix restart
* Adding API to force quality and also maintain closed state
* Address PR comments
With rapid changes to subscription settings, use of a goroutine
could end up processing dynacast needs for that subscriber in
a different order. So, record the susbcription needs of a subscriber
in the callback and process the data in a go routine.
* Keep track of pending subscriber operations.
This is required to determine if a receiver does not have
any subscription.
* correct spelling of queuing
* lock around hasPermission
* Move subscribe/unsubscribe queue to participant.
As subscribe/unsubscribe operation can come from both
local media track or remote media track, participant
needs to have it.
* Remove comment
* Stop reneg timer on close
* address comments
As there is a queue to send dynacast update, forcing
an update on unmute should be fine. That will send
the current state. If the subscribers change it,
an update will be sent as necessary.
This addresses the case of subscription changes happening
when the published track is muted. Dynacast updates are
not sent when publisher tarck is muted. If on unmute,
if subscribers do not have any changes, an update is missed
(i. e. the changes that happen when publisher track is muted
is not sent).
* Use a queue for add/remove subscribe operations.
If subscribe/unsubscribe happens very quickly, the subscription
state gets mixed up as things are keyed off of subscriberID.
Use a queue of subscribe operations and process it serially.
* set up callback for down track added
* move the queue on unexpected type
* move the queue if removeSubscirber does not have a subscribed track
* WIP commit
* Clean up
* spelling mistake
* Run subscribed track onBind in a go routine
* Address comments and more safety net
* Cache and restore forwarder state on resume
* conflicts
* mage generate
* WIP commit
* WIP commit
* Remove debug
* Revert to reduce diff
* Fix tests
* Determine spatial layer from track info quality if non-simulcast
* Adjust for invalid layer on no rid, previously that function was returning 0 for no rid case
* Fall back to top level width/height if there are no layers
* Use duration from RTPDeltaInfo
* Fixed unclean DownTrack close when removed before bound.
When a DownTrack is closed before it had a chance to be bound to a
transceiver, we'd skip close and leave it hanging. This is unlikely in
normal operations. However, it can be seen with permissions and
subscription APIs.
* remove remaining peerID references
* Support participant identity in permissions
It is harder for clients to update permissions by SID as remote
reconnecting means a new SID for that participant. Using participant
identity is a better option.
For now, participant SID is also supported. Internally, it will
get mapped to identity. Server code uses identity throughout after
doing any necessary conversion from SID -> Identity.
* Address comments
* Remove callbacks queue from sfu/DownTrack
- Connection stats callback was happening in connection stats go routine
- RTT update launches a goroutine on the receive side as it affects the
subscriber. So, no need to queue it.
- Changed two things
o Move close handler to goroutine. It is better that way as it touches
the subscriber as well
o Move max layer handling into a goroutine also so that the callback
does minimal work.
With this all the send side callback queues are removed.
* small clean up
* Prevent stats update if the deltas are empty
* increase force interval
* static check
* Change max delay to 30 seconds
* Restart the max subscribed quality timer on ICE restart.
Force an update after a restart to let clients apply dynacast settings.
Also, set the max expected layer to HIGH on restart so that stream
tracker starts up fast on a restart.
* fix test
cnderrauber