When the publisher stops publishing, the individual receivers would close
attached DownTracks first before notifying MediaTrackReceiver callbacks.
This means #1454 does not fix the issue entirely since there is still a window
when we can subscribe to a closing track.
A couple of other bits
1. Use request layer for sending PLI on bind and connected.
2. When adjusting for overshoot, do not adjust target unless current is
at max. If not, it could get stuck in a lower layer in the following
scenario
a. Overshoot to layer 2
b. Max layer is 1, start sending PLI
c. Get key frame for layer 0, adjust for overshoot as we have
something at a layer lower than max.
d. Adjust for overshoot.
e. Setting target to max means that current and target are equal
and no further adjustment happens.
* Do not overshoot when layer is locked.
One more challenging case. When current layer is already locked,
should not set up for overshoot.
* set target to current
Missed this in the last commit. Sorry.
The case of
- locking to a layer
- that layer stops
- re-allocation
This should trigger a key frame request to the max available layer.
So, have to set the request layer to max available.
Addressing edge case where a layer stopped before bitrate could be
measured. Purely bit rate based change deduction missed this as
the before and after did not have bit rates.
Use available layers to look for changes, especially currently
forwarding layer going away.
Also, simplifying bits. Only in the optimal allocation path,
these things are required. When congested, bitrate is always needed.
So, for optimal path, just look at available layer changes and adjust.
Don't need to look for bitrate based layer changes. Clean up that code.
* WIP commit
* Send stream state paused only when it is paused due to bandwidth limitation.
When stream is resumed after a stream is paused, an active update is
sent. Note that this means if there are intervening events like
mute/unmute between pause and resume, resume will be sent.
* WIP commit
* fix compile
* WIP commit
* fixing tests
* clean up exempted layers
* clean up unused stuff
* correct comment
* Don't need ops queue as order is not important now
* static check
* kick off allocation when callbacks are set up, calling from receiver means callbacks may not be set up
Even if an add track has been queued and can be used immediately
when the previous incarnation unpublishes, send the unpublished
callback as the track was technically unpublished and republished.
The re-publish will pick the same track SID when the pending track
is queued as it will get the SID from an existing published track.
* WIP commit
* Send stream state paused only when it is paused due to bandwidth limitation.
When stream is resumed after a stream is paused, an active update is
sent. Note that this means if there are intervening events like
mute/unmute between pause and resume, resume will be sent.
When downgrading (could be due to overshoot or opportunistically
locking to a higher layer), need to check if the max layer notification
needs to be done so that dynacast has the right max layer for the
participant corresponding to this downtrack.
* Re-initialise reference layer on a resume.
It is possible for a down track to resume from seeded state.
But, storing reference layer means it is storing something that
is dependent on up track. On a resume, if the up track does not
produce the reference layer, RTCP sender reports will not be sent.
So, remove that up track dependency and re-initialise the reference
layer on a resume.
This could affect A/V sync, but it should not be bad.
But, need to observe the effects.
* min difference of 1
* Notes on wht to do
- Should targetLayers be altered while doing opportunistic locking
- Should targetLayers be altered in any other path than stream allocator path?
- Lock to layer as long as it is <= opportunistic layer
- When not congested, opportunistic can be highest
- When congested, opportunistic could be nil or lowest if paused is not allowed
- When muting, can we hold on to current layers (or keep it as previous) and
restore on unmute.
- Store current/target in forwarder state and restore on seeding
- Watch for looking for targetLayers, etc. when looking to insert padding
packets. There may be an assumption about restarting on key frame and hence
okay to insert padding when target layers are invalid. This may not be true
any more when doing opportunistic forwarding.
- Can we distinguish between publisher mute or dynacast (i. e. publisher side
stopping) vs subscriber mute and do something useful? Publisher side mute
could mean continuity in sequence numbers on a restart (might be able to
catch it with opportunistic forwarding). But, there is the challenge of
unmute from publisher via signalling channel vs media. If media is arriving,
should subscribers do opportunistic forwarding before publisher mute state
update happens?
- Maybe introduce a mode where forwarding continues to a frame end (of course
with a time limit just in case the end of frame packet is lost) and then
insert silence/padding packets?
- Ensure that audio blank frame insertion does not suffer from frame boundary
issues.
* pub/sub mute separate + more notes on things to check
* WIP commit, more notes
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* clean up
* slightly better comments
* Do not stop on unmute
* do not inject blank frames when pub muted
* do not forward on audio publisher mute
* Use purely RR based RTT.
With normalization of NTP time stamp to local time,
don't need to keep track of NTP time of publisher + local time of
when a report is sent. RTT calculations can happen with RR only.
Also, do not log errors when RTT cannot be calculated due to
no last SR. This can happen if the receiver sends an RR before
it receives an SR. As SFU is doing SRs once in 5 seconds, it is
possible some RRs happen before the first SR.
* use error type
* correct error name
* Use local time base for NTP in RTCP Sender Report for downtracks.
More details in comments in code.
* Remove debug
* RTCPSenderReportInfo -> RTCPSenderReportDataExt
* Get rid of sender report data pointer checks
From logs, can see when reference layer is out of order, it is
out of order by a small amount most of the time (saw a maximum
of 5 ms in a sampling of logs). When using arrival time in those
case, some times the offset comes out to 10 ms or so. In most
of the cases, the time based diff is a lot higher (by several ms).
Just use the default of +1 diff on switch layer in case of
out-of-order time stamp. That will allow playback to move
forward and keep the timing close to actual frame time.
* some additional logging
* Do not use local time stamp when sending RTCP Sender Report
As local time does not take into account the transmission delay
of publisher side sender report, using local time to calculate
offset is not accurate.
Calculate NTP time stamp based on difference in RTP time.
Notes in code about some shortcomings of this, but should
get better RTT numbers. I think RTT numbers were bloated because of
using local time stamp.
* WIP commit
* comment
* clean up
* remove unused stuff
* cleaner comment
* remove unused stuff
* remove unused stuff
* more comments
* TrackSender method to handle RTCP sender report data
* fix test
* push rtcp sender report data to down tracks
* Need payload type for codec id mapping in relay protocol
* rename variable a bit
* FPS based stream tracker tweaks
- Cleaning up code
- Two tweaks
o A layer is declared active on receiving first packet (when starting fresh).
But, if there are no frames after that (no packets after girst packet or
there is only one frame), layer would not have been declared stopped as
the previous version waited for second frame. Now, if there are no more
frames in eval interval, declare the layer stopped.
o When frame rate goes to 0, reset FPS calculator. Otherwise, layer starting
after a long time will have frames spaced apart too far which would result
in very low frame rate. Reset the calculator and let it pick up after the
the layer restarts
- Also changing from lowest FPS -> estimated FPS and update up slowly and down fast.
There are cases where frames are to far apart result in really low FPS. Seems to
happen with NLC kind of cases where bandwidth is changed rapidly and the estimator
on browser probably gets a bit confused and starts/stops layers a bit erratically.
So, update estimate periodically to ensure eval interval is tracking current rate.
* fix factor
* spelling fix
* Split stream tracker impl from base
* slight re-arrangement of code
* fps based stream tracker
* MinFPS config
* switch back to packet based tracker
* use video config by default to handle sources without type
There were some failures with missing media. The only thing I could
see between working and non-working case is when media forwarding
starts. So, delay media forwarding till peer connection is connected.
Also, add a subscribe op only if a subscribe/unsubscribe queuing is
successful. There was a recent change to not queue a subscribe when
the participant is closed/disconnected. This got the subscribe op
counter out of whack.
When switching from local -> remote or remote -> local,
the forwarder state is cached and restored after the switch
to ensure continuity in sequence number /time stamp.
But, if the forwarder had not started before the switch,
the sequence number always starts at 1 because of seeding.
So, do not see unless forwarder was started before the switch.
David Zhao