* Do server PLI when sync is required.
A few changes
- Run key frame requester goroutine always. Runs every 200 ms which is
not bad.
- Post a key frame request when server knows it needs one, like after an
allocation. This ensures that the initial request is not delayed.
- Periodic check will ensure PLI for cases like all frame chains of a
dependency descriptor being broken.
* simplify
A few things
- Log PLI requests from client.
- Pass in marker to RTP munger as SVC can insert marker.
- Adjusting first packet time should be aware of SVC as there is single
stream in SVC
The probing + munging has not been set up to drop packets that follow
a gap. Dropping such a packet leads to padding packet sequence numbers
overlapping with regular packets.
This change does two things though.
- The not relevant packet will still not be sent over the wire. That could
create holes in the sequence number leading to NACKs
- Would the hole cause decode issues? Unclear as making this condition is hard.
Simulating it is not showing issues, but that may not be producing the bad
sequence if any.
Will look at the ability to drop a packet after a gap later.
Seeing a time stamp jump that I am not able to explain.
Basically, it looks like the time stamp doubles at some
point. There is no code which doubles the timestamp.
Can understand an erroneous roll over/wrap around, but
doubling is very strange.
So, logging only audio packets. Will disable as soon
as I have some smaples from canary.
* More fine grained filtering NACKs after a key frame.
There are applications with periodic key frame.
So, a packet lost before a key frame will not be retransmitted.
But, decoder could wait (jitter buffer, play out time) and cause
a stutter.
Idea behind disabling NACKs after key frame was another knob to
throttle retransmission bit rate. But, with spaced out retransmissions
and max retransmissions per sequence number, there are throttles.
This would provide more throttling, but affects some applications.
So, disabling filtering NACKs after a key frame.
Introducing another flag to disallow layers. This would still be quite
useful, i. e. under congestion the stream allocator would move the
target lower. But, because of congestion, higher layer would have lost
a bunch of packets. Client would NACK those. Retransmitting those higher
layer packets would congest the channel more. The new flag (default
enabled) would disallow higher layers retransmission. This was happening
before this change also, just splitting out the flag for more control.
* split flag
* Use 32-bit time stamp to get reference time stamp on a switch.
With relay and dyncast and migration, it is possible that different
layers of a simulcast get out of sync in terms of extended type,
i. e. layer 0 could keep running and its timestamp could have
wrapped around and bumped the extended timestamp. But, another layer
could start and stop.
One possible solution is sending the extended timestamp across relay.
But, that breaks down during migration if publisher has started afresh.
Subscriber could still be using extended range.
So, use 32-bit timestamp to infer reference timestamp and patch it with
expected extended time stamp to derive the extended reference.
* use calculated value
* make it test friendly
Streaming could start after 16-bits has rolled over. So, have to add
that base back to what is received in receiver report.
Otherwise, it looks like there are not packets received in window
leading to poor quality.
Need to pass in the correct time. Previously streaming start was
determined by another delta snap shot which as removed for efficiency.
Did not realise that we were passing in zero time for stats.
Also, revert of the change (the part which did not re-pause) from this
PR (https://github.com/livekit/livekit/pull/2037). That change affects
other paths. The edge it was trying to fix is more rare. Need to think
about a way which covers all cases.
* Fix time stamp adjustment when starting with dmummy packets.
- Populated extended values in ExtPacket on dummy packet.
- Have to pass reference time stamp offset to first packet time
adjustment.
* display participant version info
When restoring state, reference layer could change before this change.
That meant the time stamp base would change and cause jumps.
But, the solution in this change to store the reference layer state
and restoring it has a different issue. It is possible that the reference is
layer 2 (HIGH) for example. On a migration when the down track has
to re-attach and resume to a moved up stream track, it is possible that
layer 2 is not published due to bandwidth constraint after publisher
migrates to new node. In that case, the stream cannot be resumed as
time stamp adjustment cannot be calculated.
An option is to set referenceSpatialLayer always at layer 0 (LOW).
But, that also has a couple of issues
- Browsers like FF have shown issues with layer mapping.
- Layer 0 is lowest bit rate. So, it will have RTCP at lower frequency.
That could introduce a slight latency in stream start as we need
RTCP sender report to calculate the time stamp.
Open to ideas on how to handle this better.
* Use VP9 Key frame detection from Galene.
With ffmpeg generated VP9 file with single layer
and publishing via Go SDK, the key picture determination
outlined at https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#page-13
under the F bit explanation does not work. It declares kay frame for
pretty much all frames. Unclear if ffmpeg generated bitstream has issues
or if that procedure in the above document does not work for single
layer case.
Using the bit stream explained here
https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf
(pages 28, 62, 63) implemented in Galene.
That is more expensive as it has to parse more, but works in all cases.
* Add AV1-TODo
* add some TODOs
As expectedTS is tied to first packet and first packet adjustment
may not have happened, refTS being ahead is not a bad thing.
In one example,
- first packet was late
- a layer switch happened around 110ms later
- in that time, 190ms worth of media was forwarded
- but first packet adjustment did not happen yet
- so at that layer switch, expected was behind
- choosing ref at that switch is the right thing
A publisher mute is when the reference can fall behind because
of replaceTrack(nil). On a subscriber mute, should not jump ahead
to expected because publisher could still be lagging and behind.
Also consolidate logging.
* Remove parked layer feature.
Not worth the added complexity.
Several reasons
- Not seeing black frames on pub mute always.
- If they are there, it can consume more than 30kbps if the parked layer
is high res. That is wasted bandwidth downstream when pub is muted.
- On resume, client some time sends PLI and that triggers a key frame
request.
But, leaving the separate `PubMuted` flag in forwarder in case we can
use it for better handling.
* need the request spatial
* Add control of playout delay
Add config to enable playout delay. The delay will be limited by
[min,max] in the config option and calculated by upstream & downstream
RTT.
* check protocol version to enable playout delay
* Move config to room, limit playout-delay update interval, solve comments
* Remove adaptive playout-delay
* Remove unused config
1. When re-allocating for a track in DEFICIENT state, try to use
available headroom to accommodate change before trying to steal
bits from other tracks.
2. If the changing track gives back bits (because of muting or
moving to a lower layer subscription), use the returned bits
to try and boost deficient track(s).
* Plug a couple of holes in stream transitions.
1. Missed negative sign meant stealing bits from other tracks was not
working.
2. When a track change (mute, unmute, subscription change) cannot be
allocated, explicitly pause so that stream state update happens.
Refactor stream state update a bit to make it a bit cleaner.
* correct comment
* Add debug log for RTCP sender report.
Temporary to collect more data. Hitting scenarios under congestion
where the sender report gets off sync. Need some data to pore through
and understand and implement changes.
* Debugw
* Check for request layer lock only in the goroutine
* check before sending PLI
* max layer notifier worker
* test cleanup
* clean up
* do notification in the callback
* WIP commit
* WIP commit
* WIP commit
* Some clean up
- Removed a chatty debug log
- some spelling, punctuation correction in comments
- missed an `Abs` in check, add it.
* Add ability to roll back video layer selection.
Not currently useful, but it is possible to do things like not
applying a layer switch if the switch point time stamp is too far back.
Add ability to roll back a layer switch and invoke rollback if
a packet was selected for forwarding, but a subsequent error or decision
to drop the packet can rollback layer switch if that was the switching
packet.
In current code, the paths where a packet can be dropped after selection
does not happen at switch points. So, it was okay to apply the selection
unconditionally. But, adding the call to rollback in the current code
also in all paths where packet is dropped after selection for consistent
code flow.
* separate switch for temporal layer
* Push track quality to poor on a bandwidth constrained pause.
* add tests
* scale distance by divisor
* fix test distance to desired
* wait longer for subscription manager to reconcile
* Don't update dependency info if unordered packet received
* Trace all active svc chains for downtrack
* Try to keep lower decode target decodable
* remove comments
* Test case
* clean code
* solve comments
If ref is coming in slow (due to pacing), it is possible that
expected is ahead. Pulling next too far towards expected causes
warps in a subsequent report. Keep switches closer to ref.
Raja Subramanian