* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
In single peer connection mode, when the server answers a subscriber's
offer, configureSenderAudio set the sender codec preferences from the
server MediaEngine's payload types. The answer could therefore advertise
Opus on a payload type the offerer never offered (server PT 111 vs
offered PT 109). Chrome tolerates this; Firefox decodes 0 samples
(silence) -- packets are received but never decoded. The forwarded RTP
already uses the offered PT, so only the answer SDP was inconsistent.
This regressed in v1.12.0 once the single-PC MediaEngine became a union
of publish+subscribe codecs.
Parse the remote offer's audio rtpmap and remap the sender audio codec
preferences to echo the offered payload types (RFC 3264 6.1) before
SetCodecPreferences.
Fixes#4599
Co-authored-by: laosun <14806343+cnvipstar@users.noreply.github.com>
Previously it was anchored to participant transitioning to `ACTIVE` if
the add track request happened before that. But, that has a few issues
1.`ACTIVE` is for primary peer connection which could be subscriber peer
connection.
2. `ACTIVE` also include data channel establishment.
Switch to first connected time of publisher peer connection for that to
get a more accurate measure of track publish time.
* Add prom metrics for peer connectino state.
By direction (PUBLISHER vs SUBSCRIBER) and state ("started" ->
"connected"). This gives a way to track peer connections failing to
finish establishment.
The RTC active count can be useful for primary peer connection, but not
for non-primary. This counter can be used to track any and can generally
be used to understand success/failure rate of peer connection
establishment.
* add a couple of more states
* clean up and avoid duplicate reporting fully established
* staticcheck
Reverting
- https://github.com/livekit/livekit/pull/4521
- https://github.com/livekit/livekit/pull/4525
There are TWCC feedback packets that are larger than MTU. Seems to
happen under a couple of conditions
1. Bad client data, i. e. severely out-of-order packets, bad sequence
numbers, etc.
2. On an ICE restart - this is rare, but it seemed to be flaky network
with some packets arriving and some not and causing a lot of gaps.
Either case, not much to do. If fargmentation/re-assembly back to
publisher works, the feedback will make it through. If not, feedbacks
will be missed and clients have to work with some missing data which is
not unexpected and the protocol is designed to handle.
However, filed pion/interceptor issue just in case - https://github.com/pion/interceptor/issues/416
* Don't require media sections when joining
Client except browser (rust/libwebrtc is known) could have problem
to fire ontrack event when reuses extra media section to subscribe
track, so disable this feature in server side and let client determine
if extra media sections are needed.
* lint
* Log details of RTCP packets.
Seeing large (> MTU) packets on publisher peer connection RTCP. The
four types there are
- RTCP Receiver Reports
- NACK
- TWCC
- PLI
Can't think of what would be blowing up in size.
RTCP Receiver Report and PLI are fixed in size
NACKs vary, but the limit is 100 NACKs which should fit in 400 bytes
even if all of them are spread apart in the sequence number space.
TWCC varies, but a feedback packet is sent every 100ms or when it holds
100 packets. So, that also should not be too big.
Logging packet details to understand this better.
* revert debug
* Log large packets receive/send.
Seeing cases of servers reporting need for segmentation/re-assembly of
packets. So, logging packet receive/send for RTP/RTCP to check if
anything is seeing more than 1400 byte packets.
* log downtrack RTCP too
* Fix publish-only limitations being incorrectly applied receive-side in a single PC
* `StaticConfigurations` disabled some codecs for publish only, which worked in dual PC
* In single PC, the server incorrectly disabled these codecs in both directions
* Dual PC mode is unchanged; single PC handles per-direction filtering correctly
* Filter recv-side codecs to publish list in single-PC SDP answer
* Confirm H264 is present in offer in test
* Close peer connection unconditionally to unblock set local/remote
description operations.
Have been chasing a leak where participants have a lot of connectivity
issues and analysed a goref with Claude. Output below.
Jo Turk quickly patched sctp for reported issue -
https://github.com/pion/sctp/pull/465.
This PR moves the peer connection close to before waiting for events
queue to be drained as event queue could be blocked on
`SetLocal/RemoteDescription` hanging.
The scenario is a bit far-fetched as a lot of things have to happen, but
it does point to a scenario where things could hang. Remains to be seen
if this helps. Note that closing the peer connection early could mean
the contained objects (like data channels) could all be closed as part
of the peer connection close. But, still keeping the explicit clean up
path (which should effectively become no-op) to minimise changes.
------------------------------------------------------------------
The wedge is in pion/sctp's blocking-write gate, called synchronously from inside the PC's operations queue. Five things have to be true at the same time, and on this build they all are:
1. SCTPTransport.Start is synchronous in the SetRemoteDescription op
The stuck stack:
PeerConnection.SetRemoteDescription.func2 (peerconnection.go:1363)
→ startRTP → startSCTP
→ SCTPTransport.Start (sctptransport.go:141)
→ DataChannel.open (datachannel.go:178)
→ datachannel.Dial → Client → Stream.WriteSCTP
→ Association.sendPayloadData (association.go:3141) ← blocks here
SCTPTransport.Start synchronously sends the DCEP "OPEN" for each pre-negotiated channel. The operations.start goroutine runs SetRemoteDescription's logic; it does not return until Start does.
2. The wait has no deadline
Stream.WriteSCTP (stream.go:289) calls sendPayloadData(s.writeDeadline, ...). s.writeDeadline is the default zero-value deadline.Deadline — never armed, because DataChannel.Dial doesn't call Stream.SetWriteDeadline. So the <-ctx.Done() arm of the wait select can
never fire.
3. EnableDataChannelBlockWrite(true) puts SCTP into a serialized-write gate
At livekit-server/pkg/rtc/transport.go:362 livekit calls se.EnableDataChannelBlockWrite(true). That flips the sendPayloadData path to:
// association.go:3138-3148
if a.blockWrite {
for a.writePending {
a.lock.Unlock()
select {
case <-ctx.Done(): // never (no deadline)
case <-a.writeNotify: // only fires when writeLoop fully drains pendingQueue
}
a.lock.Lock()
}
a.writePending = true
}
4. writeNotify only fires after the writeLoop drains everything
The only place notifyBlockWritable is called is gatherOutbound (association.go:3085-3088), and only when len(chunks) > 0 && a.pendingQueue.size() == 0 — i.e., the writeLoop actually managed to move all pending chunks to inflight. If cwnd is full and SACKs stop
arriving, the writeLoop wakes up, sees zero room, sends nothing, and writePending stays true.
5. There is no association-level abort timer for data writes
At association.go:764:
assoc.t3RTX = newRTXTimer(timerT3RTX, assoc, noMaxRetrans, rtoMax)
noMaxRetrans means the retransmission timer never gives up. INIT has maxInitRetrans, but data does not. There is no equivalent of TCP's tcp_retries2 → ETIMEDOUT → ABORT. So once the path is dead post-handshake, t3RTX keeps firing into the void and the association
never transitions out of established on its own.
What it takes to wake it up
Only an external close: somebody has to terminate the underlying DTLS conn (which makes Association.readLoop's netConn.Read fail, which closes closeWriteLoopCh, which lets timerLoop exit). But — and this is the kicker — readLoop's defer at association.go:976-996
closes everything except it does not call notifyBlockWritable. So even if readLoop unwinds, any goroutine parked on <-a.writeNotify stays parked unless it was watching ctx (which here it isn't).
So the trigger sequence on this pod was almost certainly:
1. Peer establishes ICE+DTLS+SCTP, association goes established.
2. Peer disappears (ICE silently fails, NAT rebinding, OS sleep, kill -9, etc.).
3. The first DCEP-OPEN for one of livekit's pre-negotiated channels is queued; cwnd never opens because no SACKs return.
4. writePending is now true for the lifetime of the process, with no deadline, no ctx, no kill.
5. The PC's operations queue is wedged, SetRemoteDescription never returns, livekit-server's handleRemoteOfferReceived event handler is parked, the participant is never torn down, and the SCTP timerLoop pins the entire participant graph in memory until OOM-kill.
Realistic fixes (in order of how clean they are)
1. Upstream: in pion/sctp, broadcast notifyBlockWritable() (or close writeNotify) inside readLoop's defer cleanup, so a closed association unblocks any pending writers. This is the right fix.
2. livekit-server: wrap pc.SetRemoteDescription(...) with a timeout, and on timeout call pc.Close() — Close ultimately tears down the DTLS conn, which lets readLoop exit (point 1 still needs to be true for the writer goroutine to actually unblock, though).
3. Workaround: call stream.SetWriteDeadline(...) on the SCTP stream before issuing the DCEP open, so the ctx arm of the select can fire. Requires reaching past webrtc.DataChannel though.
4. Heaviest hammer: don't pre-negotiate the data channels inline with SetRemoteDescription — open them lazily after PC reaches connected so a stuck open never blocks signaling.
Without (1), even (2) leaves the writer goroutine itself parked forever — but at least the PC and its participant-side state would be released; only the SCTP goroutine subtree (much smaller) would leak.
* revert probe stop change
* handle nil offer
Some e2e is failing due to subscriptions happening late and the expected
order of m-lines is different. Not a hard failure, but logging more to
make seeing this easie.
* Resolve RTX pair via OnTrack also.
In simulcast probing path, the interceptor chain is not invoked
for primary stream. Not sure if this is a recent change. Due to this,
the RTX pair does not get resolved.
Use the onTrack callback to resolve the pair.
* remove debug
* Use published track for model access in data down track.
No need to pass in the model.
* implementation type assertion
* remove redundant log key
* one more type
* Close interface method
* clean up redundant log key
* remove unused method
* some interface changes to support data tracks in replay
* correct mock signature
After adding more fields in
https://github.com/livekit/livekit/pull/4105/files, it was not even
logging. Access to one of the added fields must have ended up waiting on
a lock and blocked.
Unfotunately, the deadlock fix in https://github.com/pion/ice/pull/840
did not address the peer connection close hang.
Splitting the logs so that the base log still happens. Ordering after
looking at the code and guessing what could still log to see if we get
more of the logs and learn more about the state and which lock ends up
the first blocking one.
* Add checks for participant and sub-components close.
Looks like there might be some memory leak with participant sessions not
getting closed properly. Adding checks (to be cleaned up later) to see
if there is a consistent place where things might hang.
* init with right type
* Remove unnecessary goroutine, thank you @milos-lk
* clean up
Correct triple-d spelling of "address" field in transport logs.
I’m not sure whether this was intentional, but I noticed it
while creating Grafana queries and filters. This matters because
anyone filtering logs using the correct spelling may
unintentionally miss relevant data, leading to incomplete or
misleading analysis.
Pion does not protect the stats getter and using it after close could
cause nil de-reference. Do a couple of things
1. Stop timer that access peer connection stats before closing peer
connection.
2. Do not access stats if peer connection is already closed
Currently, the signal requests are counted on media side and signal
responses are counted on controller side. This does not provide the
granularity to check how many response messages each media node is
sending.
Seeing some cases where track subscriptions are slow under load. This
would be good to see if the media node is doing a lot of signal response
messages.
* Set publisher codec preferences after setting remote description
Munging SDP prior to setting remote description was becoming problematic
in single peer connection mode. In that mode, it is possible that a
subscribe track m-section is added which sets the fmtp of H.265 to a
value that is different from when that client publishes. That gets
locked in as negotiated codecs when pion processes remote description.
Later when the client publishes H.265, the H.265 does only partial
match. So, if we munge offer and send it to SetRemoteDescription, the
H.265 does only a partial match due to different fmtp line and that gets
put at the end of the list. So, the answer does not enforce the
preferred codec. Changing pion to put partial match up front is more
risky given other projects. So, switch codec preferences to after remote
description is set and directly operate on transceiver which is a better
place to make these changes without munging SDP.
This fixes the case of
- firefox joins first
- Chrome preferring H.265 joining next. This causes a subscribe track
m-section (for firefox's tracks) to be created first. So, the
preferred codec munging was not working. Works after this change.
* clean up
* mage generate
* test
* clean up
* Do not send both asb-send-time and twcc.
In single peer connection mode, both extensions are set on the media
engine and both would be negotiated. Unfortunately, pion/webrtc does
not yet support RTPSender.SetParameters() which would allow setting
specific header extensions for the sender. So, check for TWCC enabled
and use it. If not, do abs-send-time if that is enabled.
* check BWE type
* comment
* Handle migration better in single peer connection case.
Have to avoid creating a sender for publish tracks.
Otherwise, the sender messes up the answer SDP because
it forced to inactive and that gets used by type match when setting
remote description offer.
Create sender only for m-lines which was used to send.
* remove unnecessary RemoveTrack
Was breaking migration in cases where there was inactive transceivers
because of direction check.
This change will break single peer connection case, but can look at that
one later.
* WIP
* check using protocol version
* revert
* clean up
* sdp cid argument
* WIP
* WIP
* test
* clean up
* clean up
* fixes
* clean up
* clean up
* clean up
* conditional checks
* tests for both dual and single peer connection
* test
* test
* test
* type check
* test
* todo
* munges
* combined config
* populate mid
* limit to receive only
* clean up
* clean up
* clean up
* older test
* clean up
* alternative audio codec
* dtx
* don't need to copy
* Anunay feedback
* use the available peer connection
* publisher check
* WIP
* WIP
* WIP
* no mid
* media sections requirement
* mage generate
* WIP
* WIP
* set data channel receive size for test
* handle early media better
* WIP
* do not do ICERestart if no subscriber
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* start up subscriber RTCP worker
* WIP
* WIP
* clean up
* clean up
* flag to indicate use of single peer connection
* remove unused interface method
* clean up
* clean up
* Jie feedback #1
* deps
* do not access subscriber in one shot mode
* more places for one shot mode
* more one shot fixes
* deps
* deps
* test
* Populate SDP cid in track info when available.
- Adding SDP cid to TrackInfo. Browsers like FF uses a different stream
id for AddTrack and actual SDP offer. So, have to look up using both
on server side. To make it easier, store both (only if different) in
TrackInfo.
- Use a codec in TrackInfo for audio also. There is some discussion
around doing simulcast codec for audio so that something like PSTN can
use G.711 without any transcoding. So, just keep it consistent between
audio and video.
- Populate SDP cid when SDP offer is received. It could populate a
pending track or an already published track if the new offer is for a
back up codec where the primary codec is already published.
- Passed around parsed offer to more places to avoid parsing multiple
times.
- Clean up MediaTrack interface a bit and remove unneeded methods.
* WIP
* WIP
* deps
* stream allocator mime aware
* clean up
* populate SDP cid before munging
* interface methods
