* Make connection quality not too optimistic.
With score normalization, the quality indicator showed good
under conditions which should have normally showed some badness.
So, a few things in this PR
- Do not normalize scores
- Pick the weakest link as the representative score (moving away from
averaging)
- For down track direction, when reporting delta stats, take the number
of packets sent actually. If there are holes in the feed (upstream
packet loss), down tracks should not be penalised for that loss.
State of things in connection quality feature
- Audio uses rtcscore-go (with a change to accommodate RED codec). This
follows the E-model.
- Camera uses rtcscore-go. No change here. NOTE: THe rtscore here is
purely based on bits per pixel per frame (bpf). This has the following
existing issues (no change, these were already there)
o Does not take packet loss, jitter, rtt into account
o Expected frame rate is not available. So, measured frame rate is
used as expected frame rate also. If expected frame rate were available,
the score could be reduced for lower frame rates.
- Screen share tracks: No change. This uses the very old simple loss
based thresholding for scoring. As the bit rate varies a lot based on
content and rtcscore video algorithm used for camera relies on
bits per pixel per frame, this could produce a very low value
(large width/height encoded in a small number of bits because of static content)
and hence a low score. So, the old loss based thresholding is used.
* clean up
* update rtcscore pointer
* fix tests
* log lines reformat
* WIP commit
* WIP commit
* update mute of receiver
* WIP commit
* WIP commit
* start adding tests
* take min score if quality matches
* start adding bytes based scoring
* clean up
* more clean up
* Use Fuse
* log quality drop
* clean up debug log
* - Use number of windows for wait to make things simpler
- track no layer expected case
- always update transition
- always call updateScore
When we unsubscribe from a speaker, SendSpeakerUpdates will drop updates
from that speaker. This has the side effect of dropping the "clearing"
message that we are sending as well.
Due to the order of events in MediaTrackReceiver and friends, SubscribedTrack
will be closed before the track is removed from RoomTrackManager.
Because of this, when a track is unpublished, it's possible to be subscribed
to the track as it's closing.
By introducing a closing state, we'd prevent accidental subscription to
closing tracks.
Not a good design. There is not an easy way to filter messages
before it hits media node. Without that, there is not a lot
of advantage.
And there are sequences that are not handled correctly in this
deleted implementation.
So, deleting code to prevent use.
UpdateSubscription had a shortcoming where when it couldn't find the
participant, it ignored the request.
This PR further removes the reliance of current publisher state from
subscribers.
- SubscribeToTrack only takes in a trackID
- Introduced RoomTrackManager to maintain all published tracks to a room
- Added TrackUnpublished event to clearly indicate when a track has been removed
- SubscribeRequested event no longer include information about the publisher
* Add Timer to detect dtls failure quickly
* Fix pc state check in timeout after ice
* More strict conditions to switch candidate type
* log for signal interuppt
* typo
Added a new manager to handle all subscription needs. Implemented using reconciler pattern. The goals are:
improve subscription resilience by separating desired state and current state
reduce complexity of synchronous processing
better detect failures with the ability to trigger full reconnect
* Initial commit of signal deduper.
Idea is protect against signal storm from misbehaving clients.
Design:
- SignalDeduper interface with one method to handle a SignalRequest and
return if dupe or not.
- Signal specific deduper. Could have made a single de-duper which could
handle all signal message types, but making it per type so that the
code is cleaner.
- Some module (like the router) can instantiate whatever signal types
it wants to de-dupe. When a signal message is received, that module
can run the signal message through the list of de-dupers and
potentially drop the message if any of the de-dupers declare that the
message is a dupe. Making it a list makes things a little bit
inefficient, but keeps things cleaner. Hopefully, not many de-dupers
will be needed so that the inefficiency is not pronounced.
* re-arrange comments
* helper function
* add ParticipantClosed
* Add option to issue full reconnect on a publication error.
Leaving the publication error timeout at 30 seconds as there
are some publications taking long. Also, there are cases
where the peer connection fails after 30 seconds. The peer
connection failure happens after publication error is detected.
But, 30 seconds is a good amount of time for publication to establish.
* prevent recursive lock
* Fix rtcp lost for downtrack used incorrect buffer factory
In buffer factory change(#1173), every pariticipant has its own
buffer factory, can't use publisher's bufferfactory to create
DownTrack
* clean code
There were some failures with missing media. The only thing I could
see between working and non-working case is when media forwarding
starts. So, delay media forwarding till peer connection is connected.
Also, add a subscribe op only if a subscribe/unsubscribe queuing is
successful. There was a recent change to not queue a subscribe when
the participant is closed/disconnected. This got the subscribe op
counter out of whack.
* Cache RTPStats and seed on re-use
When a cached down track is re-used, RTPStats was not cached.
This caused sender reports getting out-of-sync with the remote side.
Cache RTPStats and seed it on re-use.
* staticcheck
Seeing some supervisor error logs under two conditions
- Issuing a full reconnect - client should close this session and
form a new one. So, supervisor errors on the to be closed session
is not useful.
- Some times it takes a long time for publisher PC to establish.
If publish monitor timer stars when a pending track is added,
the time out fires before ICE/DTLS is established. So, include
a condition to start timer on publication monitor only after
peer connection is connected.
With migration in, once the local track is published, the
remote track should be closed. Add a flag to `RemovePublishedTrack`
to control the close behaviour. Invoke `Close` if specified.
Without, the remote track is not closed if it is waiting to resolve,
i. e. not yet attached. That remote track is left hanging.
* Remove VP9 from media engine set up.
* Remove vp9 from config sample
* Supervisor beginnings
Eventual goal is to have a reconciler which moves state from
actual -> desired. First step along the way is to observe/monitor.
The first step even in that is an initial implementation to get
feedback on the direction.
This PR is a start in that direction
- Concept of a supervisor at local participant level
- This supervisor will be responsible for periodically monitor
actual vs desired (this is the one which will eventually trigger
other things to reconcile, but for now it just logs on error)
- A new interface `OperationMonitor` which requires two methods
o Check() returns an error based on actual vs desired state.
o IsIdle() returns bool. Returns true if the monitor is idle.
- The supervisor maintains a list of monitors and does periodic check.
In the above framework, starting with list of
subscriptions/unsubscriptions. There is a new module
`SubscriptionMonitor` which checks subscription transitions.
A subscription transition is queued on subscribe/unsubscribe.
The transition can be satisfied when a subscribedTrack is added OR
removed. Error condition is when a transition is not satisfied for
10 seconds. Idle is when the transition queue is empty and
subscribedTrack is nil, i. e. the last transition would have been
unsubscribe and subscribed track removed (unsubscribe satisfied).
The idea is individual monitors can check on different things.
Some more things that I am thinking about are
- PublishedTrackMonitor - started when an add track happens,
satisfied when OnTrack happens, error if `OnTrack` does not
fire for a while and track is not muted, idle when there is
nothing pending.
- PublishedTrackStreamingMonitor - to ensure that a published track
is receiving media at the server (accounting for dynacast, mute, etc)
- SubscribedTrackStreamingMonitor - to ensure down track is sending
data unless muted.
* Remove debug
* Protect against early casting errors
* Adding PublicationMonitor
* Send connection type to telemetry
When connected, determine how the participant's primary connection is
connected and report it in ParticipantActive event.
* address feedback
* fixed case where prflx is reported instead of relay
* incorporate comments
* WIP commit
* WIP commit
* fix copy pasta
* setting PC with previous answer has to happen synchronously
* static check
* WIP commit
* WIP commit
* fixing transport tests
* fix tests and clean up
* minor renaming
* FIx test race
* log event when channel is full
* Export CloseSignalConnection
There are a few places where that close pattern is repeated.
Export it and use that function in other places directly.
* fix test
* Start RTCP workers after peer connection connects
* Move more things into transport module
* Start RTCP workers only on connected
* Test needs PeerConnection() method
* adjust comment
* Prevent track subscriptions/adding receivers after close
With subscribe/unsubscribe queuing, a subscribe may be
attempted after a call to `RemoveAllSubscribers`.
So, renaming `RemoveAllSubscribers` to `InitiateClose`
and maintaining state that track is in the process of closing.
* Mime specific remove
* Remove unused error
* do not add receiver when closing
