* Notify max layer taking into account overshoot.
An attempt to handle case of FF stopping layer 0, but not layer 1.
When max subscribed layer is layer 0, server allows overshoot to layer 1
to ensure continued streaming when the channel is not congested.
But, dynacast could have reported maximum subscribed layer as layer 0.
This is a very simple attempt to address that by taking overshoot
into account. Needs testing if this works well or not.
NOTE: When subsriber/down track is unmuted, it will report
the max subscribed layer as the max required layer. In those cases,
if the client does not start layer 0, there will still be an issue.
IOW, server is not keeping track of client behaviour that the client
has stopped layer 0 and publishing only layer 1. Server is just
accommodating overshoot with this change.
* Fix a couple of tests and change reflect.DeepEqual -> require.Equal as
much as possible
* Cache RTPStats and seed on re-use
When a cached down track is re-used, RTPStats was not cached.
This caused sender reports getting out-of-sync with the remote side.
Cache RTPStats and seed it on re-use.
* staticcheck
* WIP commit
* Connection quality changes
- Fix Firefox showing poor quality
o The issue was that we were using max available layer and
calculating quality. The rationale being that even if
server sends dynacast messages, client may not implement
dynacast and still stream all layers. But, with Firefox
(maybe a Firefox bug), it sends some small amount of
data on layer 2 even when that layer is disabled.
Guessing it is probing (or actually we might be using
some small value for high layers as Firefox cannot turn off
layers). That higher layer gets used in quality calculation.
As the bit rate on that layer is extremely low, it yields low
score.
Fixed by considering the max expected layer. That is of most
interest. Yes, clients may ignore dynacast and stream all layers,
but, max expected is the one of interest. So, look for
quality in the max expected layer and not max available layer.
- Lots of clean up around connection quality stuff
o Use a dynamic scaling thing to ensure that we do not get bitten
by absolute values. Calculate best possible scenario score and
map that to maximum MOS score. This will ensure that different
codecs, different settings do not mess up the scoring. For example,
a client might use 1 Mbps for 720p, but a different client could
use 2 Mbps for 720p. As an SFU/infrastructure middlebox, we do
not have control over quality at those rates. We can only ensure
that streaming happens smoothly at those rates. So, in that
example, for client 1, 1 Mbps will map to MOS 5.0 and for client 2,
2 Mbps will map to MOS 5.0. Any impairments after that will
reflect in the score.
o Penalise for missing target layer by one level for one layer missed.
o Move tests to connection quality directory. The participant test
was not super useful.
* Add missed file
* Remove debug code
* use more constants and initialise normalisation factor
* rtcscore pointer
* WIP commit
* Clean up
* spelling mistake
* Run subscribed track onBind in a go routine
* Address comments and more safety net
* Cache and restore forwarder state on resume
* conflicts
* mage generate
* Allow overshooting maximum when there are no bandwidth constraints.
Some clients prioritize sending higher layer of video (for e.g.
Firefox). They may get into a state where congestion does not allow
sending lower layers.
That combined with adaptive streaming capping the max layer at a
certain level, it is possible to get into a state where video is
not streamed.
To make this experience better, allow overshoot beyond max layers
in case of optimal (i. e. no congestion) allocation.
NOTE: This means that the video could freeze when there is congestion
even if `AllowPause` is not disabled as in congested state, we do
not overshoot the max layer.
* log about allowing overshoot
* Inject silence opus frames on mute.
If not, under certain circumstances, residual noise
seems to trigger comfort noise generation at the decoder
which is not all that comfortable.
* inject silence only when muting
* Add comment
* augment comment
* Delete blank line
* Adjust TS offset on blank frames
* Remove debug
* Do not modify `lastTS` as it affects timestamp on next switch.
* Trying more stuff for DTX
* - Use a go routine to send blank frames
- Use duration instead of number of frames and calculate number of
frames
* augment comment
* Remove debug
* Return a chan from writeBlankFrameRTP
* Use a long duration for mute
* add comment
* Incorporate suggestion from Jie
Should be okay as that is usually only three elements max.
Stream tracker + manager send available layers. As stream trackers
run in different go routines, need to think some more about chances
of out-of-order operations. But, making a copy in forwarder so
that it is not referncing moving data.
Also, when setting up provisional allocation, make a copy
(that was the intention, i. e. a provisional allocation should have
stable data to work with).
* Inject silence opus frames on mute.
If not, under certain circumstances, residual noise
seems to trigger comfort noise generation at the decoder
which is not all that comfortable.
* inject silence only when muting
* Add comment
* augment comment
* Delete blank line
* Adjust TS offset on blank frames
* Remove debug
* Do not modify `lastTS` as it affects timestamp on next switch.
* use duration to calculate number of blank frames
* log errors in blank frames write path
Found another reason for potential stalling on layer switch up.
When switching up, it is possible that the key frame of the packet
at higher layer arrives very close to a packet in the currently
forwarded layer.
An inversion happens when higher layer packet arrives earlier
(as timestamped at receicver), but gets scheduled for forwarding
later (as seen by forwarder).
Because of the order of processing goroutines,
it is possible to have negative diff when trying to adjust
RTP timestamps for layer switch. A negative diff results in
large jump in RTP timestamp. Client stalls, sends PLI three seconds
later (3 seconds is a Chrome thing, not sure about others), waits
for another key frame and starts again. In the mean time, the
video is frozen.
* Handle an edge in layer lock.
A very edge case
- Available layer: [0, 1, 2], but bitrate is not yet available.
We set it to layer 2 awaiting measurement.
- Measurement for layers 0 and 1 come through.
- Still no key frame for layer 2.
- Finalize layers runs and sees that bitrate is available for 0 and 1.
It finalizes layer 1.
- Layer 1 key frame comes (because we asked key frame of layer 2,
publisher sends key frame for all layers). Locks to layer 1.
- No more events happen to switch to layer 2.
Changes
-------
- Move bit rate measurement to StreamTrackerManager. Allows re-use
in relay.
- Make bit rate availability (from zero -> non-zero) an event
and let it flow through the stream allocation flow so that we
always have an event when bit rate measurement becomes available.
This gets rid of finalization which I was unhappy with it anyway as
it was polling every second.
- Removing REMB stuff from buffer. We do not use it.
It is incorrect anyway. REMB should be ay peer connection level.
* Fix test
* fix test
* Simplify allocate
* Simplify/clean up
* Key frames
- Keep track of key frame stats
- Split out PLI from down track used for purpose of layer locking.
This will give us a good picture of down stream issues forcing a PLI.
- Use key frame requester whenever there is a layer lock required.
Not just the first key frame. With the synchronous thing, the counter
was just ridiculously high like 150 or something because of all
the initial padding packets. Also, use RTT in key frame requester.
* send first PLI before waiting
* Turn off key frame requester when disabled
* simplify
* Random clean up
* Split out StreamTrackerManager for re-use
* Reset the correct tracker
* use generation counter to exit coroutine
* start only for video and when enabled
* Add RemoveAllTrackers method
* Add a resync API to sfu.DownTrack
Also passing in logger with context into sfu package. More to do here
with proper logging context in all modules, but this is a start
* Remove debug code
* fix tests
* Prevent multiple resume notifications on track
When returning previous allocation as is, reset change to none.
Ideally, would like to have change as a separate return and not in last
allocation. But, also prefer to have last allocation state. For now,
resetting change.
* log bandwidth estimate only on decrease
* Fix forwarding status deduction
- When muted OR when there are no available layers, declare optimal
- When target layer is the maximum it can achieve taking available
layers into account even if they are not the maximum subscribed layer,
it is still optimal as there is nothing better available.
* Fix and add more tests for forwarding status
* WIP commit
* deficient handling
* Add missing ProvisionalAllocatePrepare
* adjust state on track removal
* Increase test timeout
* - Add comments about cooperative routines
- Take down transition if available in cooperative scheme
- Use layer comparison when taking down transition. Because of when the
bitrate is measured, it is not always guaranteed bandwidthDelta is -ve
when moving down.
- Do not add track to stream allocator till bind.
* make comment better
* a bit more clear comments
* Use OnBind on subscribed track
* Stream Allocator Try 3
Making an intermediate PR to do
- Special treatment for screen share tracks
- When allocating all tracks,
o try to stream all tracks by starting with the lowest layer
o multi-pass across tracks to get a more even distribution
Not yet done:
-------------
In deficient state,
o Allocate a specific track on a change
o Steal from other tracks
* Correct sense of managed track
* have to range to copy
* generate
* fix VideoLayers compare
* Use t.simulcasted
cnderrauber