* Remove `Head` field from `ExtPacket` structure.
Although we do not intend to, but if packets get out-of-order
in the forwarding path (maybe reading in multiple goroutines
or using some worker pool to distribute packets), the `Head`
indicator could lead to wrong behaviour. It is possible that
at the receiver, the order is
- Seq Num N, Head = true
- N + 1, Head = true
If the forwarding path sees `N + 1` first, the Head flag
when it sees `N` packet is incorrect and will lead to incorrect
behaviour.
The alternative check is very simple. So, remove `Head` flag.
* Remove unused field
* Use seq num offsets cache instead of missing seq num map.
Map operations can be costly. Use a fixed size array to store offsets.
4096 sequence numbers should be more than 16 seconds for 720p video
which should be plenty to look up offset of out-of-order packets.
Packets out-of-order more than that will probably be useless anyway.
* Move offset cache population to only when we are forwarding
* add some debug logs
* Remove debug
* Remove callbacks queue from sfu/DownTrack
- Connection stats callback was happening in connection stats go routine
- RTT update launches a goroutine on the receive side as it affects the
subscriber. So, no need to queue it.
- Changed two things
o Move close handler to goroutine. It is better that way as it touches
the subscriber as well
o Move max layer handling into a goroutine also so that the callback
does minimal work.
With this all the send side callback queues are removed.
* small clean up
* Do not post close callback in ops queue if not started.
Ops queue is started in `Bind()`. If `Close()` is called
when bind did not happen (because the underlying peer connection
closed before bind), the close callback does not run.
Check if ops queue is running before posting close callback
into the queue.
Not pretty, but covers this case. Need to think about it more.
* correct check
* Use delta stats throughout and avoid calculating deltas in telemetry
* Fix a few things after testing
* Remove debug
* Fix tests
* delete instead of setting to nil
* Point to the latest protocol
Found another reason for potential stalling on layer switch up.
When switching up, it is possible that the key frame of the packet
at higher layer arrives very close to a packet in the currently
forwarded layer.
An inversion happens when higher layer packet arrives earlier
(as timestamped at receicver), but gets scheduled for forwarding
later (as seen by forwarder).
Because of the order of processing goroutines,
it is possible to have negative diff when trying to adjust
RTP timestamps for layer switch. A negative diff results in
large jump in RTP timestamp. Client stalls, sends PLI three seconds
later (3 seconds is a Chrome thing, not sure about others), waits
for another key frame and starts again. In the mean time, the
video is frozen.
* Handle an edge in layer lock.
A very edge case
- Available layer: [0, 1, 2], but bitrate is not yet available.
We set it to layer 2 awaiting measurement.
- Measurement for layers 0 and 1 come through.
- Still no key frame for layer 2.
- Finalize layers runs and sees that bitrate is available for 0 and 1.
It finalizes layer 1.
- Layer 1 key frame comes (because we asked key frame of layer 2,
publisher sends key frame for all layers). Locks to layer 1.
- No more events happen to switch to layer 2.
Changes
-------
- Move bit rate measurement to StreamTrackerManager. Allows re-use
in relay.
- Make bit rate availability (from zero -> non-zero) an event
and let it flow through the stream allocation flow so that we
always have an event when bit rate measurement becomes available.
This gets rid of finalization which I was unhappy with it anyway as
it was polling every second.
- Removing REMB stuff from buffer. We do not use it.
It is incorrect anyway. REMB should be ay peer connection level.
* Fix test
* fix test
* Simplify allocate
* Simplify/clean up
This still does not address root cause of large loss, but at least
does not display crazy thing like packets = 0, but packet rate is 45/s.
Also, RLock in ToString() as there are bits of structure used in
stringification.
* fix keyFrameRequestGeneration not exit after close
* change to bound check
* Refactor stop key frame requester into a function
* Remove redundant key frame requester stop
Co-authored-by: boks1971 <raja.gobi@tutanota.com>
Raja Subramanian