* Connectino quality misc changes
1. Call scorer.Update() with nil stat when no data available so that
scorer can synthesise window with proper window time.
2. Substract out loss in interval to account for packets not sent at
all.
3. Fix `packetsNotFound` variable in `getIntervalStats`. I remember this
working at some point. Not sure if I fat fingered in another PR and
deleted the increment line.
4. Logging a bit more when no packets expected. Those can get noisy
especially when track is muted. But, seeing some unexplained
instances of no packets leading to quality drop. So, temporary logging
to get a bit more information.
* correct spelling
* Limit packet score minimum to 0.0
* Make connection quality not too optimistic.
With score normalization, the quality indicator showed good
under conditions which should have normally showed some badness.
So, a few things in this PR
- Do not normalize scores
- Pick the weakest link as the representative score (moving away from
averaging)
- For down track direction, when reporting delta stats, take the number
of packets sent actually. If there are holes in the feed (upstream
packet loss), down tracks should not be penalised for that loss.
State of things in connection quality feature
- Audio uses rtcscore-go (with a change to accommodate RED codec). This
follows the E-model.
- Camera uses rtcscore-go. No change here. NOTE: THe rtscore here is
purely based on bits per pixel per frame (bpf). This has the following
existing issues (no change, these were already there)
o Does not take packet loss, jitter, rtt into account
o Expected frame rate is not available. So, measured frame rate is
used as expected frame rate also. If expected frame rate were available,
the score could be reduced for lower frame rates.
- Screen share tracks: No change. This uses the very old simple loss
based thresholding for scoring. As the bit rate varies a lot based on
content and rtcscore video algorithm used for camera relies on
bits per pixel per frame, this could produce a very low value
(large width/height encoded in a small number of bits because of static content)
and hence a low score. So, the old loss based thresholding is used.
* clean up
* update rtcscore pointer
* fix tests
* log lines reformat
* WIP commit
* WIP commit
* update mute of receiver
* WIP commit
* WIP commit
* start adding tests
* take min score if quality matches
* start adding bytes based scoring
* clean up
* more clean up
* Use Fuse
* log quality drop
* Periodically report bitrate to down track.
For connection quality based on bitrate for down tracks,
the measured rate should be used. That is to ensure that
down track quality measurement does not get affected by
publisher side changes negatively (or positively).
Report the optimal bit rate to connection quality scorer
every second so that scorer has a continuously updating
picture of the stream and can compare the actual bit rate
against expected optimal bitrate more reliably.
Doing it at time like allocation, the bitrate may not be
accurate (or may not even be available). So, a periodic update
is necessary.
* add transition at allocation times
* clean up debug log
* - Use number of windows for wait to make things simpler
- track no layer expected case
- always update transition
- always call updateScore
* Make connection quality not too optimistic.
With score normalization, the quality indicator showed good
under conditions which should have normally showed some badness.
So, a few things in this PR
- Do not normalize scores
- Pick the weakest link as the representative score (moving away from
averaging)
- For down track direction, when reporting delta stats, take the number
of packets sent actually. If there are holes in the feed (upstream
packet loss), down tracks should not be penalised for that loss.
State of things in connection quality feature
- Audio uses rtcscore-go (with a change to accommodate RED codec). This
follows the E-model.
- Camera uses rtcscore-go. No change here. NOTE: THe rtscore here is
purely based on bits per pixel per frame (bpf). This has the following
existing issues (no change, these were already there)
o Does not take packet loss, jitter, rtt into account
o Expected frame rate is not available. So, measured frame rate is
used as expected frame rate also. If expected frame rate were available,
the score could be reduced for lower frame rates.
- Screen share tracks: No change. This uses the very old simple loss
based thresholding for scoring. As the bit rate varies a lot based on
content and rtcscore video algorithm used for camera relies on
bits per pixel per frame, this could produce a very low value
(large width/height encoded in a small number of bits because of static content)
and hence a low score. So, the old loss based thresholding is used.
* clean up
* update rtcscore pointer
* fix tests
* log lines reformat
* WIP commit
* WIP commit
* update mute of receiver
* WIP commit
* WIP commit
* start adding tests
* take min score if quality matches
* start adding bytes based scoring
* clean up
* more clean up
* Use Fuse
* log quality drop
* clean up debug log
* - Use number of windows for wait to make things simpler
- track no layer expected case
- always update transition
- always call updateScore
In the following order, got the wrong layer
- Max layer is 0, max published is 0, request layer is 0
- Current locks to 0.
- Max changes to 1. Nothing changes as 1 is not published yet.
- Max published changes to 1.
- As curernt layer is valid, available and locked to request layer, it
was kept. But, it should have checked if the request layer changed
and updated accordingly.
In the following scenario, PLI layer lock got stuck at the wrong layer
- target is at 2 to allow overshoot
- current gets to 1, but can't get higher because publisher is not
publishing higher layer
- max layer changed to 0
Because of adjusting for overshoot only when current == target, it never
happened and layer lock PLI kept asking for layer 0. Although, key
frames were received, switch did not happen.
Always check for overshoot adjustment possibility against current layer.
* Change lock scope of access to RTCP sender report data.
Forwarder calls back to get time stamp offset.
Holding buffer lock is a much bigger scoped lock.
Reduce lock scope and cache latest sender report under its own lock.
And use that cache when calculating time stamp offset.
* move sr cache to stream tracker manager for re-use in relay
* cache before spread
When the publisher stops publishing, the individual receivers would close
attached DownTracks first before notifying MediaTrackReceiver callbacks.
This means #1454 does not fix the issue entirely since there is still a window
when we can subscribe to a closing track.
A couple of other bits
1. Use request layer for sending PLI on bind and connected.
2. When adjusting for overshoot, do not adjust target unless current is
at max. If not, it could get stuck in a lower layer in the following
scenario
a. Overshoot to layer 2
b. Max layer is 1, start sending PLI
c. Get key frame for layer 0, adjust for overshoot as we have
something at a layer lower than max.
d. Adjust for overshoot.
e. Setting target to max means that current and target are equal
and no further adjustment happens.
* Do not overshoot when layer is locked.
One more challenging case. When current layer is already locked,
should not set up for overshoot.
* set target to current
Missed this in the last commit. Sorry.
The case of
- locking to a layer
- that layer stops
- re-allocation
This should trigger a key frame request to the max available layer.
So, have to set the request layer to max available.
Addressing edge case where a layer stopped before bitrate could be
measured. Purely bit rate based change deduction missed this as
the before and after did not have bit rates.
Use available layers to look for changes, especially currently
forwarding layer going away.
Also, simplifying bits. Only in the optimal allocation path,
these things are required. When congested, bitrate is always needed.
So, for optimal path, just look at available layer changes and adjust.
Don't need to look for bitrate based layer changes. Clean up that code.
* WIP commit
* Send stream state paused only when it is paused due to bandwidth limitation.
When stream is resumed after a stream is paused, an active update is
sent. Note that this means if there are intervening events like
mute/unmute between pause and resume, resume will be sent.
* WIP commit
* fix compile
* WIP commit
* fixing tests
* clean up exempted layers
* clean up unused stuff
* correct comment
* Don't need ops queue as order is not important now
* static check
* kick off allocation when callbacks are set up, calling from receiver means callbacks may not be set up
Even if an add track has been queued and can be used immediately
when the previous incarnation unpublishes, send the unpublished
callback as the track was technically unpublished and republished.
The re-publish will pick the same track SID when the pending track
is queued as it will get the SID from an existing published track.
* WIP commit
* Send stream state paused only when it is paused due to bandwidth limitation.
When stream is resumed after a stream is paused, an active update is
sent. Note that this means if there are intervening events like
mute/unmute between pause and resume, resume will be sent.
When downgrading (could be due to overshoot or opportunistically
locking to a higher layer), need to check if the max layer notification
needs to be done so that dynacast has the right max layer for the
participant corresponding to this downtrack.
* Re-initialise reference layer on a resume.
It is possible for a down track to resume from seeded state.
But, storing reference layer means it is storing something that
is dependent on up track. On a resume, if the up track does not
produce the reference layer, RTCP sender reports will not be sent.
So, remove that up track dependency and re-initialise the reference
layer on a resume.
This could affect A/V sync, but it should not be bad.
But, need to observe the effects.
* min difference of 1
* Notes on wht to do
- Should targetLayers be altered while doing opportunistic locking
- Should targetLayers be altered in any other path than stream allocator path?
- Lock to layer as long as it is <= opportunistic layer
- When not congested, opportunistic can be highest
- When congested, opportunistic could be nil or lowest if paused is not allowed
- When muting, can we hold on to current layers (or keep it as previous) and
restore on unmute.
- Store current/target in forwarder state and restore on seeding
- Watch for looking for targetLayers, etc. when looking to insert padding
packets. There may be an assumption about restarting on key frame and hence
okay to insert padding when target layers are invalid. This may not be true
any more when doing opportunistic forwarding.
- Can we distinguish between publisher mute or dynacast (i. e. publisher side
stopping) vs subscriber mute and do something useful? Publisher side mute
could mean continuity in sequence numbers on a restart (might be able to
catch it with opportunistic forwarding). But, there is the challenge of
unmute from publisher via signalling channel vs media. If media is arriving,
should subscribers do opportunistic forwarding before publisher mute state
update happens?
- Maybe introduce a mode where forwarding continues to a frame end (of course
with a time limit just in case the end of frame packet is lost) and then
insert silence/padding packets?
- Ensure that audio blank frame insertion does not suffer from frame boundary
issues.
* pub/sub mute separate + more notes on things to check
* WIP commit, more notes
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* clean up
* slightly better comments
* Do not stop on unmute
* do not inject blank frames when pub muted
* do not forward on audio publisher mute
* Use purely RR based RTT.
With normalization of NTP time stamp to local time,
don't need to keep track of NTP time of publisher + local time of
when a report is sent. RTT calculations can happen with RR only.
Also, do not log errors when RTT cannot be calculated due to
no last SR. This can happen if the receiver sends an RR before
it receives an SR. As SFU is doing SRs once in 5 seconds, it is
possible some RRs happen before the first SR.
* use error type
* correct error name
* Use local time base for NTP in RTCP Sender Report for downtracks.
More details in comments in code.
* Remove debug
* RTCPSenderReportInfo -> RTCPSenderReportDataExt
* Get rid of sender report data pointer checks
From logs, can see when reference layer is out of order, it is
out of order by a small amount most of the time (saw a maximum
of 5 ms in a sampling of logs). When using arrival time in those
case, some times the offset comes out to 10 ms or so. In most
of the cases, the time based diff is a lot higher (by several ms).
Just use the default of +1 diff on switch layer in case of
out-of-order time stamp. That will allow playback to move
forward and keep the timing close to actual frame time.
* some additional logging
* Do not use local time stamp when sending RTCP Sender Report
As local time does not take into account the transmission delay
of publisher side sender report, using local time to calculate
offset is not accurate.
Calculate NTP time stamp based on difference in RTP time.
Notes in code about some shortcomings of this, but should
get better RTT numbers. I think RTT numbers were bloated because of
using local time stamp.
* WIP commit
* comment
* clean up
* remove unused stuff
* cleaner comment
* remove unused stuff
* remove unused stuff
* more comments
* TrackSender method to handle RTCP sender report data
* fix test
* push rtcp sender report data to down tracks
* Need payload type for codec id mapping in relay protocol
* rename variable a bit
* FPS based stream tracker tweaks
- Cleaning up code
- Two tweaks
o A layer is declared active on receiving first packet (when starting fresh).
But, if there are no frames after that (no packets after girst packet or
there is only one frame), layer would not have been declared stopped as
the previous version waited for second frame. Now, if there are no more
frames in eval interval, declare the layer stopped.
o When frame rate goes to 0, reset FPS calculator. Otherwise, layer starting
after a long time will have frames spaced apart too far which would result
in very low frame rate. Reset the calculator and let it pick up after the
the layer restarts
- Also changing from lowest FPS -> estimated FPS and update up slowly and down fast.
There are cases where frames are to far apart result in really low FPS. Seems to
happen with NLC kind of cases where bandwidth is changed rapidly and the estimator
on browser probably gets a bit confused and starts/stops layers a bit erratically.
So, update estimate periodically to ensure eval interval is tracking current rate.
* fix factor
* spelling fix
Raja Subramanian