* test: verify upstream and downstream connection stats
Adds TestConnectionStats integration test where two clients connect,
each publishes audio + video, and the test asserts that both
publisher-side (LocalMediaTrack.GetTrackStats) and subscriber-side
(DownTrack.GetTrackStats) report non-zero packets and bytes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sfu: add DownTrack.OnStatsUpdate hook and use it in stats test
Adds a public OnStatsUpdate setter on DownTrack mirroring the existing
pattern on WebRTCReceiver. The new callback fires alongside the
configured DownTrackListener (production path is unaffected) and is
intended for tests/observers to validate the AnalyticsStat data flowing
through the listener.
Augments TestConnectionStats to:
- hook WebRTCReceiver.OnStatsUpdate for each published track and assert
the captured AnalyticsStat has non-zero packets/bytes (upstream).
- hook the new DownTrack.OnStatsUpdate for each subscribed track and
make the same assertion (downstream).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Media loss proxy is not use, so it is okay, but was an unintentional
delete in https://github.com/livekit/livekit/pull/3252/changes
Was checking code due to a report of how Chrome does RTCP reports in
147+ breaking a few services. Don't think that affects LK, but found
this while reading code.
* Legacy TrackInfo.Simulcast flag.
When AddTrack did not send SimulcastCodecs, the legacy `Simulcast` flag
was not set. Fix it by setting the flag when a second layer is
published.
* staticcheck
* use the existing PrimaryReceiver function
- prevent some escape to heap
- avoid copying by using a ring buffer for receiver reports (probably
should remove this as this is for debugging only and data so far has
shown clients sending bad data and nothing more.)
* Use Muted in TrackInfo to propagated published track muted.
When the track is muted as a receiver is created, the receiver
potentially was not getting the muted property. That would result in
quality scorer expecting packets.
Use TrackInfo consistently for mute and apply the mute on start up of a
receiver.
* update mute of subscriptions
* Update go deps
Generated by renovateBot
* update api usage
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: David Zhao <dz@livekit.io>
For a participant migrating out, the track could be resumed on a
different node, but ending on the migrating out node. So, `flush` should
be used to indicate if track is going to be resumed.
Removing some logs which have not been useful in terms of insights other
than saying that there are a bunch of packets missing. Going to start
looking at gaps in terms of time if the inter-packet gap is too high.
Also, using logging these events as first 20 and then every 200.
A bunch of edges to note here
RED packet does not have sequence number for redundant blocks. It only
has timestamp offset compared to the primary payload. The receivers are
supposed to use just timestamp to sequence the payload and decode.
But, when converting from RED -> Opus, the packets extracted from RED
packet should be assigned a sequence number before they can be
forwarded. The simple rule is, if packet N contains X redundant
payloads, they are assigned sequence number of N - X to N - 1.
However there are cases like the following sequence (with 1 packet
redundancy)
- Seq num 10, timestamp 2000, forwarded
- Seq num 11 is lost
- Seq num 12 has a redundant payload. Seq num 12 has timestamp of 4000.
Ideally would expect the redundant payload to have a timestamp offset
of 1000, so the redundant payload can be mapped to sequence number 11
and timestamp 3000 (4000 - 1000). But, in the problematic case, it has
an offset of 3000 resulting in sequence number 11 and timestamp of
1000 causing an inversion with packet at sequence number 10.
Unclear if this a publisher issue, i. e. packing RED wrong or if this is
some expected behaviour with DTX. i. e. the DTX packets are not included
in redundant payload. For example, the sequence
- Seq num 10 -> DTX
- Seq num 11 -> DTX -> lost
- Seq num 12 -> Regular packet and include sequence num 9 as that is the
last regular packet.
Anyhow, detect this condition and drop the time inverted packet.
Note however this handles only inversion against the highest sent packet
sequence number and timestamp. So, some old packet inverted with some
other old packet getting forwarded will get through. That has been the
case always though and detecting that would be expensive and
complicated.
At least for egress, will also look at adding a check for inversion so
that it can catch it before sending it down the gstreamer pipeline. As
the egress uses a jitter buffer with ordered sequence number emits, it
will be simpler to detect timestamp going back when sequence number is
moving forward (of course the mute/dtx challenege is there).
* Log time inversion between incoming packets
Log of timestamp inversion within a red packet did not show anything.
Log across packets. Not dropping till there is more evidence of the
cause.
* save
* comment
* Guard against timestamp inversion in RED -> Opus conversion.
Seeing timestamp inversion (sequence number is +1, but timestamp is
-960, i.e. 20ms) in the RED -> Opus conversion path. Not able to spot
any bugs in code. So, logging details upon detection and also dropping
the packet. If not dropped, downstream components like Egress treat it
as a big timestamp jump (because sequence number is moving forward) and
try to adjust pts which ends up causing drops.
* do not log time reversal at the start
* typo
* Add debug for receiver restart.
Have a suspicion that something is deadlocking between restart receiver
and buffer bind during replay. Adding debug to get a better picture of
state of receiver restart.
* consistent logging
* Optimise some bits in rtpstats_receiver
RTPStatsReceiver.Update is one of the high CPU bits. Taking some
suggestions from Cursor. Makes the `Update` function verbose though :-(
* zap.Inline logging fields
* rename
* Set up audio config in audio level module when config is updated.
It is possible to get audio config after bind (bind is where the audio
level module is created) for remote tracks. So, split out setting audio
level config in audio level module and invoke it when config is updated.
* coderabbit review
* prevent divide-by-0
* not active before config
* Do not increase max expected layer on track info update.
When max expected layer increases, the corresponding trackers are reset
so that first packets from those layers can trigger a layer detected
change enabling quick detection of layer start.
A track info update changing max to what is in track info could set the
max expected to be higher without resetting the tracker. And that would
cause dynacast induced max layer change to miss tracker reset too.
Sequence
- dynacast sets max expected to 0
- track info update sets it to 2
- dynacast sets it to 1 --> this should have reset tracker on layer 1,
but because it is less than current max (2), it is skipped.
* thank you CodeRabbit
* force update on start
* Reducing some info level logs.
Also, relaxing the check for runaway RTCP receiver report to allow for
rollover to catch up if it is not too far away.
* set logger
With audio simulcast codecs, it is possible that the clock rate of the
primary codec is different from the secondary codec. If a subscriber
binds to the secondary codec, the clock rate should be set correctly. Do
it at bind time.
Raja Subramanian