* Close peer connection unconditionally to unblock set local/remote
description operations.
Have been chasing a leak where participants have a lot of connectivity
issues and analysed a goref with Claude. Output below.
Jo Turk quickly patched sctp for reported issue -
https://github.com/pion/sctp/pull/465.
This PR moves the peer connection close to before waiting for events
queue to be drained as event queue could be blocked on
`SetLocal/RemoteDescription` hanging.
The scenario is a bit far-fetched as a lot of things have to happen, but
it does point to a scenario where things could hang. Remains to be seen
if this helps. Note that closing the peer connection early could mean
the contained objects (like data channels) could all be closed as part
of the peer connection close. But, still keeping the explicit clean up
path (which should effectively become no-op) to minimise changes.
------------------------------------------------------------------
The wedge is in pion/sctp's blocking-write gate, called synchronously from inside the PC's operations queue. Five things have to be true at the same time, and on this build they all are:
1. SCTPTransport.Start is synchronous in the SetRemoteDescription op
The stuck stack:
PeerConnection.SetRemoteDescription.func2 (peerconnection.go:1363)
→ startRTP → startSCTP
→ SCTPTransport.Start (sctptransport.go:141)
→ DataChannel.open (datachannel.go:178)
→ datachannel.Dial → Client → Stream.WriteSCTP
→ Association.sendPayloadData (association.go:3141) ← blocks here
SCTPTransport.Start synchronously sends the DCEP "OPEN" for each pre-negotiated channel. The operations.start goroutine runs SetRemoteDescription's logic; it does not return until Start does.
2. The wait has no deadline
Stream.WriteSCTP (stream.go:289) calls sendPayloadData(s.writeDeadline, ...). s.writeDeadline is the default zero-value deadline.Deadline — never armed, because DataChannel.Dial doesn't call Stream.SetWriteDeadline. So the <-ctx.Done() arm of the wait select can
never fire.
3. EnableDataChannelBlockWrite(true) puts SCTP into a serialized-write gate
At livekit-server/pkg/rtc/transport.go:362 livekit calls se.EnableDataChannelBlockWrite(true). That flips the sendPayloadData path to:
// association.go:3138-3148
if a.blockWrite {
for a.writePending {
a.lock.Unlock()
select {
case <-ctx.Done(): // never (no deadline)
case <-a.writeNotify: // only fires when writeLoop fully drains pendingQueue
}
a.lock.Lock()
}
a.writePending = true
}
4. writeNotify only fires after the writeLoop drains everything
The only place notifyBlockWritable is called is gatherOutbound (association.go:3085-3088), and only when len(chunks) > 0 && a.pendingQueue.size() == 0 — i.e., the writeLoop actually managed to move all pending chunks to inflight. If cwnd is full and SACKs stop
arriving, the writeLoop wakes up, sees zero room, sends nothing, and writePending stays true.
5. There is no association-level abort timer for data writes
At association.go:764:
assoc.t3RTX = newRTXTimer(timerT3RTX, assoc, noMaxRetrans, rtoMax)
noMaxRetrans means the retransmission timer never gives up. INIT has maxInitRetrans, but data does not. There is no equivalent of TCP's tcp_retries2 → ETIMEDOUT → ABORT. So once the path is dead post-handshake, t3RTX keeps firing into the void and the association
never transitions out of established on its own.
What it takes to wake it up
Only an external close: somebody has to terminate the underlying DTLS conn (which makes Association.readLoop's netConn.Read fail, which closes closeWriteLoopCh, which lets timerLoop exit). But — and this is the kicker — readLoop's defer at association.go:976-996
closes everything except it does not call notifyBlockWritable. So even if readLoop unwinds, any goroutine parked on <-a.writeNotify stays parked unless it was watching ctx (which here it isn't).
So the trigger sequence on this pod was almost certainly:
1. Peer establishes ICE+DTLS+SCTP, association goes established.
2. Peer disappears (ICE silently fails, NAT rebinding, OS sleep, kill -9, etc.).
3. The first DCEP-OPEN for one of livekit's pre-negotiated channels is queued; cwnd never opens because no SACKs return.
4. writePending is now true for the lifetime of the process, with no deadline, no ctx, no kill.
5. The PC's operations queue is wedged, SetRemoteDescription never returns, livekit-server's handleRemoteOfferReceived event handler is parked, the participant is never torn down, and the SCTP timerLoop pins the entire participant graph in memory until OOM-kill.
Realistic fixes (in order of how clean they are)
1. Upstream: in pion/sctp, broadcast notifyBlockWritable() (or close writeNotify) inside readLoop's defer cleanup, so a closed association unblocks any pending writers. This is the right fix.
2. livekit-server: wrap pc.SetRemoteDescription(...) with a timeout, and on timeout call pc.Close() — Close ultimately tears down the DTLS conn, which lets readLoop exit (point 1 still needs to be true for the writer goroutine to actually unblock, though).
3. Workaround: call stream.SetWriteDeadline(...) on the SCTP stream before issuing the DCEP open, so the ctx arm of the select can fire. Requires reaching past webrtc.DataChannel though.
4. Heaviest hammer: don't pre-negotiate the data channels inline with SetRemoteDescription — open them lazily after PC reaches connected so a stuck open never blocks signaling.
Without (1), even (2) leaves the writer goroutine itself parked forever — but at least the PC and its participant-side state would be released; only the SCTP goroutine subtree (much smaller) would leak.
* revert probe stop change
* handle nil offer
* Use Muted in TrackInfo to propagated published track muted.
When the track is muted as a receiver is created, the receiver
potentially was not getting the muted property. That would result in
quality scorer expecting packets.
Use TrackInfo consistently for mute and apply the mute on start up of a
receiver.
* update mute of subscriptions
* fix: ensure num_participants is accurate in webhook events (#4265)
Three fixes for stale/incorrect num_participants in webhook payloads:
1. Move participant map insertion before MarkDirty in join path so
updateProto() counts the new participant.
2. Use fresh room.ToProto() for participant_joined webhook instead of
a stale snapshot captured at session start.
3. Remove direct NumParticipants-- in leave path (inconsistent with
updateProto's IsDependent check), force immediate proto update,
and wait for completion before triggering onClose callbacks.
* fix: use ToProtoConsistent for webhook events instead of forcing immediate updates
Some e2e is failing due to subscriptions happening late and the expected
order of m-lines is different. Not a hard failure, but logging more to
make seeing this easie.
Subscription can switch between remote track and local track or
vice-versa. When that happens, closing the subscribed track of one or
the other asynchronously means the re-subscribe could race with
subscribed track closing.
Keeping the case of `isExpectedToResume` sync to prevent the race.
Would be good to support multiple subscribed tracks per subscription.
So, when subscribed track closes, subscription manager can check and
close the correct subscribed track. But, it gets complex to clearly
determine if a subccription is pending or not and other events. So,
keeping it sync.
When a subscriber disconnects, observer closures registered on the
publisher's TrackChangedNotifier and TrackRemovedNotifier were never
removed. These closures capture the SubscriptionManager, which holds
the ParticipantImpl, preventing the entire participant object graph
(PCTransport, SDPs, RTP stats, DownTracks) from being garbage collected.
In rooms with many participants that disconnect and reconnect frequently,
this causes unbounded memory growth proportional to the number of
disconnect events. The leaked memory is not recoverable while the room
remains open.
Clear notifiers in both handleSubscribedTrackClose (individual
subscription teardown) and SubscriptionManager.Close (full participant
teardown), matching the existing cleanup in handleSourceTrackRemoved.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Close both peer connections to aid migration.
In single peer connection case, that would close publisher peer
connection.
@cnderrauber I don't remember why we only closed subscriber peer
connection. I am thinking it is okay to close both (or the publisher
peer connection in single peer connection mode). Please let me know if I
am missing something.
* log change only
And match design to RTP header extension, i. e. the padding for
extensions is not at per extension level (which was the case before),
but has been changed to padding the aggregate of all extensions in this
PR.
This allows for abstracting away how the stop is implemented - default implementation stays the same - the existing OSS egress launcher just calls the existing Stop method on the client.
Had made a change in remote participant case to not have telemetry
listener as telemetry does not apply to remote participant. But, that
listener ended up getting used for subscriber and became a null
listener. Use the listener of the subscriber participant for subscribed
tracks.
Release to Docker / docker (push) Failing after 3m42s
* Key telemetry stats work using combination of roomID, participantID
With forwarded participant, the same participantID can existing in two
rooms.
NOTE: This does not yet allow a participant session to report its
events/track stats into multiple rooms. That would require regitering
multiple listeners (from rooms a participant is forwarded to).
* missed file
* data channel stats
* PR comments + pass in room name so that telemetry events have proper room name also
* Do not increase max expected layer on track info update.
When max expected layer increases, the corresponding trackers are reset
so that first packets from those layers can trigger a layer detected
change enabling quick detection of layer start.
A track info update changing max to what is in track info could set the
max expected to be higher without resetting the tracker. And that would
cause dynacast induced max layer change to miss tracker reset too.
Sequence
- dynacast sets max expected to 0
- track info update sets it to 2
- dynacast sets it to 1 --> this should have reset tracker on layer 1,
but because it is less than current max (2), it is skipped.
* thank you CodeRabbit
* force update on start
To allow using "participant closing" log entry for calculating things
like session duration by paricipant kind or some other client SDK based
attribute.
* Reducing some info level logs.
Also, relaxing the check for runaway RTCP receiver report to allow for
rollover to catch up if it is not too far away.
* set logger
Paul Wells