* Log potential sequence number de-sync in receicer reports.
Seeing some cases of a roll over being missed. That ends up
as largish range to search in an interval and reports missing packets
in the packet metadata cache.
Logging some details.
* just log in one place
Ideally, can remove the nil return when there are too many packets
as we have more information with extended sequence numbers, but
logging duration first to understand what is happening better.
When converting from RED -> Opus, if there is a loss, SFU recovers
that loss if it can using a subsequent redundant packet. That path
was not setting the extended sequence number properly.
Also, ensuring use of monotonic clock for first packet time adjustment
also.
* Cap expected packets to padding diff.
On the receiver, no longer using packet metadata cache to calculate
interval stats. An optimisation to get rid of packet metadata cache
on receiver side.
Because of that, padding packets in an interval could be more than
expected packets. As padding packets is just a counter, out-of-order
padding packets will make the diff look larger than expected packets
in a window. Cap the expected to 0.
NOTE: This makes it so that the count is not accurate in a window,
but that is okay occasionally. It will affect reported stats and quality
calculations, but it should be rare. For example, if 30 packets were
received in a window and 60 out-of-order padding packets were received,
it would reported as 0 packets were received. One option is to not
increment padding packets when they are out-of-order, but that will mess
up overall stats. Will make that change if we see this happen a lot.
* log unexpected padding packets
* Log resync at Infow.
Seeing potentially large sequence number jumps on a resync.
And it seems to happen on a lot of subscribe/unsubscribe.
Logging at Infow to understand better.
Probably need to find a way to avoid resync. But, logging for now to
check if I can catch one.
* Remove resync and log large sequence number jumps
Sending a single PLI on connected & bound meant that the upstream
throttler may not have sent it and down stream does not have a key frame
to lock onto. Caused some e2e test failures due to limited time of
track.
There are cases that experience the signalling channel timeout
and disconnect and there are no logs of what the state of ICE at all.
Log ICE candidates when closing transport so that there is some
visibility in those cases.
n
Logging expected WS close at Infow to understand reasons for closure.
Moving "read from ws" to Debugw as it happens when signalling closes.
Also filter out a data channel abort chunk log as it shows a bunch of
errors, but those are expected though.
* Use marshal + unmarshal to ensure unmarshallable fields are not copied.
Need to ensure that config structs/fields are marshallable.
There was a use of a = b copy of struct and some of the embeded structs
had locks and copying was not good.
* update protocol
* Update deps
Happens when converting quality in subscibed settings to layer.
Looks like it can happen only if the provided quality is OFF.
Don't know of any client that does that. Anyhow, prevent out-of-range
access which causea a panic.
Switching to using session specific TURN credentials instead of shared
credentials per Room. Also eliminates need to load Room from Redis
during TURN authentication
It's been reported that "ghost" participants, those that did not terminate
cleanly, hang around the room for too long after they disappear.
Evaluating our timeouts a bit, it seems that we are really conservative
in waiting for participants to disconnect. This PR cuts down the disconnect
timeout from 50s to 20s, a 30s reduction.
* Use bit map.
Also, duplicate packet detection is impoetant for dropping padding
only packets at the publisher side itself. In the last PR, mentioned
that it is only for stats.
* clean up
* Update deps
* Reduce packet meta data cache - part 1
Packet meta data cache takes a good amount of space.
That cache is 8K entries deep and each entry is 8 bytes.
So, that takes 64KB per RTP stream.
It is mostly needed for down stream to line up with receiver reports.
So, removing cache from up stream (RTPStatsReceiver) as part 1.
Will look at optimising the down stream in part 2.
* Remove caching from RTPStatsReceiver
* clean up a bit more
* maintain history and fix test
* Throttle packet errors/warns a bit.
In very bad network conditions, it is possible that packets
arrive out-of-order, have choppy behaviour.
Use some counters and temper logs.
* slight change in comment
Using time from outside make anachronous samples in expected
distance/bit rate measurement. So, have to let the time be
snap shotted in scorer lock scope.
Streaming could start after 16-bits has rolled over. So, have to add
that base back to what is received in receiver report.
Otherwise, it looks like there are not packets received in window
leading to poor quality.
Need to pass in the correct time. Previously streaming start was
determined by another delta snap shot which as removed for efficiency.
Did not realise that we were passing in zero time for stats.
Also, revert of the change (the part which did not re-pause) from this
PR (https://github.com/livekit/livekit/pull/2037). That change affects
other paths. The edge it was trying to fix is more rare. Need to think
about a way which covers all cases.
* Split RTPStats into receiver and sender.
For receiver, short types are input and need to calculate extended type.
For sender (subscriber), it can operate only in extended type.
This makes the subscriber side a little simpler and should make it more
efficient as it can do simple comparisons in extended type space.
There was also an issue with subscriber using shorter type and
calculating extended type. When subscriber starts after the publisher
has already rolled over in sequence number OR timestamp, when
subsequent publisher side sender reports are used to adjust subscriber
time stamps, they were out of whack. Using extended type on subscriber
does not face that.
* fix test
* extended types from sequencer
* log
* Fix time stamp adjustment when starting with dmummy packets.
- Populated extended values in ExtPacket on dummy packet.
- Have to pass reference time stamp offset to first packet time
adjustment.
* display participant version info
* Sequencer small optimisations
1. Use range map to exclude padding only packets. Should take lesser
space as we are not using slice to hold pointer to actual data.
2. Avoid `time.Now()` when adding each packet. Just use the arrival time
as it should be close enough. `time.Now()` was showing up in
profile.
* remove debug
* correct comment
