* Perform unsubscribe in parallel to avoid blocking
When unsubscribing from tracks, we flush a blank frame in order to prepare
the transceivers for re-use. This process is blocking for ~200ms. If
the unsubscribes are performed serially, it would prevent other subscribe
operation from continuing.
This PR parallelizes that operation, and ensures subsequent subscribe
operations could reuse the existing transceivers.
* also perform in parallel when uptrack close
* fix a few log fields
* Avoid reconnect loop for unsupported downtrack
If the client subscribes to a track which codec is unsupported by the
client, sfu will trigger negotiation failed and issue a full reconnect
after received client answer. If the client try to subscribe that track
then it will got full reconnect again. That will cause a infinite
reconnect loop until the client don't subscribe that track. This PR
will unsubscribe the error track for the client and send a
SubscriptionResponse that contain the reason to indicates the track's
codec is not supported to avoid the reconnect loop.
* Split probe controller from StreamAllocator.
With TWCC, there is a need to check for probe status
in a separate goroutine. So, probe specific stuff need
locking. Split out the probe controller to make that cleaner.
* remove defer
* Return max spatial layer from selectors.
With differing requirements of SVC and allowing overshoot in Simulcast,
selectors are best placed to indicate what is the max spatial layer when
they indicate a switch to max spatial layer.
* fix test
* prevent race
It is possible that publisher paces the media.
So, RTCP sender report from publisher could be ahead of
what is being fowarded by a good amount (have seen up to 2 seconds
ahead). Using the forwarded time stamp for RTCP sender report
in the down stream leads to jumps back and forth in the down track
RTCP sender report.
So, look at the publisher's RTCP sender report to check for it being
ahead and use the publisher rate as a guide.
Actually, was not filtering the not last sender report error before.
Previous PR did that. This PR restores the old no last sender report
filter. Both are filterable errors.
* Fix unwrap
An out-or-order packet wrapping back after a wrap around had already happened
was not using proper cycle ounter to calculate unerapped value.
* update mediatransportutil
1. Completely removing RTT and jitter from score calculation.
Need to do more work there.
a. Jitter is slow moving (RFC 3550 formula is designed that way).
But, we still get high values at times. Ideally, that should
penalise the score, but due to jitter buffer, effect may not be
too bad.
b. Need to smooth RTT. It is based on receiver report and if one
sample causes a high number, score could be penalised
(this was being used in down track direction only). One option
is to smooth it like the jitter formula above and try using it.
But, for now, disabling that also.
2. When receiving lesser number of packets (for example DTX), reduce the
weight of packet loss with a quadratic relationship to packet loss
ratio. Previously using a square root and it was potentially
weighting it too high. For example, if only 5 packets were received
due to DTX instead of 50, we were still giving 30% weight
(sqrt(0.1)). Now, it gets 1% weight. So, if one of those 5 packets
were lost (20% packet loss ratio), it still does not get much weight
as the number of packets is low.,
3. Slightly slower decrease in score (in EWMA)
4. When using RED, increase packet loss weight thresholds to be able to
take more loss before penalizing score.
Two things
- Somehow the publisher RTCP sender report time stamp goes back some
times. Log it differently. Also, use signed type for logging so
that negative is easy to see.
- On down track, because of silence frame injection on mute, the RTCP
sender report time stamp might be ahead of timestamp we will use
on unmute. If so, ensure that next timestamp is also not before
what was sent in RTCP sender report.
The PID controller seems to be working well. But, it is unclear where
it can be applied as some of the data shows significant jumps
(either caused by BT devices or possibly noise cancellation/cpu
constraint) and although PID controller is slowly pulling things
to expected sample rate, it could be a bit slow.
Unfortunately, cannot munge too much in a middle box.
However leaving the controller in there as it is doing its job
for cases where things slip slowly.
Changing things to log significant jumps (more than 200 ms away
from expected) at Infow level.
Also, recording drift and sample rate in RTP stats proto and string
representation.
* Handle time stamp increment across mute.
Two cases handled
1. Starting on mute could inject blank frame/padding packets.
These time stamps are randomly generated. So, when the publisher
unmutes, the time stamp was jumping ahead by only 1. Make it so
that they jump ahead by elapsed time since starting the blank frames/
padding packets.
2. When generating blank frames at the end of a down track, if
the track was muted at that time, the blank frame time stamps
could have been off (i. e. would have been pointing to time
after the last forwarded frame). Here also use current time
to adjust time stamp. Maybe, this could help in some cases where
we are seeing unflushed video buffer?
* remove unnecessary check
* address feedback and also maintain first synthesized time stamp
With short term measurements, the adjustment itself was causing
some oscillations and drift tend to settle at some small value
and oscillated around it due to push/pull affecting small window
measurement.
It was possible that the adjustment applied in the middle
of a frame resulting in the same frame having multiple time stamps.
That would have caused video to pause/jump.
Apply the offset only at the start of the frame so that all
packets of a frame get the same offset.
* Experimental flag to try time stamp adjustment to control drift.
There is a config to enable this.
Using a PID controller to try and keep the sample rate at expected
value. Need to be seen if this works well. Adjustment are limited
to 25 ms max at a time to ensure there are no large jumps.
And it is applied when doing RTCP sender report which happens
once in 5 seconds currently for both audio and video tracks.
A nice introduction to PID controllers - https://alphaville.github.io/qub/pid-101/#/
Implementation borrowed from - https://github.com/pms67/PID
A few things TODO
1. PID controller tuning is a process. Have picked values from test from
that implementation above. May not be the best. Need to try.
2. Can potentially run this more often. Rather than running it only when
running RTCP sender report (which is once in 5 seconds now), can
potentially run it every second and limit the amount of change to
something like 10 ms max.
* remove unused variable
* debug log a bit more
* Keep track of expected RTP time stamp and control drift.
- Use monotonic clock in RTCP Sender Report and packet times
- Keep the time stamp close to expected time stamp on layer/SSRC
switches
* clean up
* fix test compile
* more test compile failures
* anticipatory clean up
* further clean up
* add received sender report logging
* Keep track of expected RTP time stamp and control drift.
- Use monotonic clock in RTCP Sender Report and packet times
- Keep the time stamp close to expected time stamp on layer/SSRC
switches
* clean up
* fix test compile
* more test compile failures
data.
Without the check, it was getting tripped by publisher not publishing
any data. Both conditions returned nil, but in one case, the receiver
report should have been received, but no movement in number of packets.
* Run quality scorer when there are no streams.
In the down stream direction, receiver report is used for scoring.
If there are no receiver reports, it should go to `dry` state and report
poor quality.
Update scorer on dry condition only when update score has not happened
for longer than some multiple of update interval. Cannot update on every
interval when there are no streams as receiver report might be just
missed. Waiting for longer to ensure that report is definitely not
received.
* update last stats time
When current became unavailable, it was possible for
target to be set to opportunistic. Because of that,
the downgrade did not happen and PLI layer lock was
requested continuously.
* A coupke of stream allocator tweaks
- Do not overshoot on catch up. It so happens that during probe
the next higher layer is at some bit rate which is much lower
than normal bit rate for that layer. But, by the time the probe
ends, publisher has climbed up to normal bit rate.
So, the probe goal although achieved is not enough.
Allowing overshoot latches on the next layer which might be more
than the channel capacity.
- Use a collapse window to record values in case of a only one
or two changes in an evaluation window. Some times it happens
that the estimate falls once or twice and stays there. By collapsing
repeated values, it could be a long time before that fall in estimate
is processed. Introduce a collapse window and record duplicate value
if a value was not recorded for collapse window duration. This allows
delayed processing of those isolated falls in estimate.
* minor clean up
* add a probe max rate
* fix max
* use max of committed, expected for max limiting
* have to probe at goal
Raja Subramanian