* Log large packets receive/send.
Seeing cases of servers reporting need for segmentation/re-assembly of
packets. So, logging packet receive/send for RTP/RTCP to check if
anything is seeing more than 1400 byte packets.
* log downtrack RTCP too
This patch updates the check for auto creating rooms to also
consider the RoomCreate grant per token instead of just the
global config option.
With this patch, applications can decide on their own whether
users or which users can auto create rooms. This allows
applications that rely on auto creation (saving an API call)
to co-exist with those who might want to mint tokens for
subscribe-only users.
Specifically LaSuite Meet relies on the auto create behavior,
however enabling the global config option would make a
MatrixRTC deployment vulnerable to abuse, as users on remote
homeservers get tokens in order to subscribe.
* add AssignmentHook to AssignJob; propagate websocket write errors
- Replace the `url *string` parameter on `Worker.AssignJob` with a
middleware-style `AssignmentHook` so callers can intercept the
`JobAssignment` send (e.g. to set Url, or to gate hedged attempts so
only one assignment is written).
- Remove the `sendRequest` helper. Inline `WriteServerMessage` and
propagate the error: `AssignJob` returns immediately on a failed
availability or assignment write, leaving the job out of
`runningJobs`; `TerminateJob` still updates local bookkeeping when
the wire write fails but surfaces the write error to the caller.
* tidy
* Add TURN permission handler.
- Turn off permissions to private/link local/multicast and internal IPs
- Add a list of CIDRs that can be used for more things to deny
permission to.
* unused
* add config for allowing private IPs, used in testing
* add a TTL to user name and use it to auth
* allow list for restricted peer CIDRs
* test: verify upstream and downstream connection stats
Adds TestConnectionStats integration test where two clients connect,
each publishes audio + video, and the test asserts that both
publisher-side (LocalMediaTrack.GetTrackStats) and subscriber-side
(DownTrack.GetTrackStats) report non-zero packets and bytes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sfu: add DownTrack.OnStatsUpdate hook and use it in stats test
Adds a public OnStatsUpdate setter on DownTrack mirroring the existing
pattern on WebRTCReceiver. The new callback fires alongside the
configured DownTrackListener (production path is unaffected) and is
intended for tests/observers to validate the AnalyticsStat data flowing
through the listener.
Augments TestConnectionStats to:
- hook WebRTCReceiver.OnStatsUpdate for each published track and assert
the captured AnalyticsStat has non-zero packets/bytes (upstream).
- hook the new DownTrack.OnStatsUpdate for each subscribed track and
make the same assertion (downstream).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Media loss proxy is not use, so it is okay, but was an unintentional
delete in https://github.com/livekit/livekit/pull/3252/changes
Was checking code due to a report of how Chrome does RTCP reports in
147+ breaking a few services. Don't think that affects LK, but found
this while reading code.
* Fix publish-only limitations being incorrectly applied receive-side in a single PC
* `StaticConfigurations` disabled some codecs for publish only, which worked in dual PC
* In single PC, the server incorrectly disabled these codecs in both directions
* Dual PC mode is unchanged; single PC handles per-direction filtering correctly
* Filter recv-side codecs to publish list in single-PC SDP answer
* Confirm H264 is present in offer in test
* Legacy TrackInfo.Simulcast flag.
When AddTrack did not send SimulcastCodecs, the legacy `Simulcast` flag
was not set. Fix it by setting the flag when a second layer is
published.
* staticcheck
* use the existing PrimaryReceiver function
- prevent some escape to heap
- avoid copying by using a ring buffer for receiver reports (probably
should remove this as this is for debugging only and data so far has
shown clients sending bad data and nothing more.)
* Close peer connection unconditionally to unblock set local/remote
description operations.
Have been chasing a leak where participants have a lot of connectivity
issues and analysed a goref with Claude. Output below.
Jo Turk quickly patched sctp for reported issue -
https://github.com/pion/sctp/pull/465.
This PR moves the peer connection close to before waiting for events
queue to be drained as event queue could be blocked on
`SetLocal/RemoteDescription` hanging.
The scenario is a bit far-fetched as a lot of things have to happen, but
it does point to a scenario where things could hang. Remains to be seen
if this helps. Note that closing the peer connection early could mean
the contained objects (like data channels) could all be closed as part
of the peer connection close. But, still keeping the explicit clean up
path (which should effectively become no-op) to minimise changes.
------------------------------------------------------------------
The wedge is in pion/sctp's blocking-write gate, called synchronously from inside the PC's operations queue. Five things have to be true at the same time, and on this build they all are:
1. SCTPTransport.Start is synchronous in the SetRemoteDescription op
The stuck stack:
PeerConnection.SetRemoteDescription.func2 (peerconnection.go:1363)
→ startRTP → startSCTP
→ SCTPTransport.Start (sctptransport.go:141)
→ DataChannel.open (datachannel.go:178)
→ datachannel.Dial → Client → Stream.WriteSCTP
→ Association.sendPayloadData (association.go:3141) ← blocks here
SCTPTransport.Start synchronously sends the DCEP "OPEN" for each pre-negotiated channel. The operations.start goroutine runs SetRemoteDescription's logic; it does not return until Start does.
2. The wait has no deadline
Stream.WriteSCTP (stream.go:289) calls sendPayloadData(s.writeDeadline, ...). s.writeDeadline is the default zero-value deadline.Deadline — never armed, because DataChannel.Dial doesn't call Stream.SetWriteDeadline. So the <-ctx.Done() arm of the wait select can
never fire.
3. EnableDataChannelBlockWrite(true) puts SCTP into a serialized-write gate
At livekit-server/pkg/rtc/transport.go:362 livekit calls se.EnableDataChannelBlockWrite(true). That flips the sendPayloadData path to:
// association.go:3138-3148
if a.blockWrite {
for a.writePending {
a.lock.Unlock()
select {
case <-ctx.Done(): // never (no deadline)
case <-a.writeNotify: // only fires when writeLoop fully drains pendingQueue
}
a.lock.Lock()
}
a.writePending = true
}
4. writeNotify only fires after the writeLoop drains everything
The only place notifyBlockWritable is called is gatherOutbound (association.go:3085-3088), and only when len(chunks) > 0 && a.pendingQueue.size() == 0 — i.e., the writeLoop actually managed to move all pending chunks to inflight. If cwnd is full and SACKs stop
arriving, the writeLoop wakes up, sees zero room, sends nothing, and writePending stays true.
5. There is no association-level abort timer for data writes
At association.go:764:
assoc.t3RTX = newRTXTimer(timerT3RTX, assoc, noMaxRetrans, rtoMax)
noMaxRetrans means the retransmission timer never gives up. INIT has maxInitRetrans, but data does not. There is no equivalent of TCP's tcp_retries2 → ETIMEDOUT → ABORT. So once the path is dead post-handshake, t3RTX keeps firing into the void and the association
never transitions out of established on its own.
What it takes to wake it up
Only an external close: somebody has to terminate the underlying DTLS conn (which makes Association.readLoop's netConn.Read fail, which closes closeWriteLoopCh, which lets timerLoop exit). But — and this is the kicker — readLoop's defer at association.go:976-996
closes everything except it does not call notifyBlockWritable. So even if readLoop unwinds, any goroutine parked on <-a.writeNotify stays parked unless it was watching ctx (which here it isn't).
So the trigger sequence on this pod was almost certainly:
1. Peer establishes ICE+DTLS+SCTP, association goes established.
2. Peer disappears (ICE silently fails, NAT rebinding, OS sleep, kill -9, etc.).
3. The first DCEP-OPEN for one of livekit's pre-negotiated channels is queued; cwnd never opens because no SACKs return.
4. writePending is now true for the lifetime of the process, with no deadline, no ctx, no kill.
5. The PC's operations queue is wedged, SetRemoteDescription never returns, livekit-server's handleRemoteOfferReceived event handler is parked, the participant is never torn down, and the SCTP timerLoop pins the entire participant graph in memory until OOM-kill.
Realistic fixes (in order of how clean they are)
1. Upstream: in pion/sctp, broadcast notifyBlockWritable() (or close writeNotify) inside readLoop's defer cleanup, so a closed association unblocks any pending writers. This is the right fix.
2. livekit-server: wrap pc.SetRemoteDescription(...) with a timeout, and on timeout call pc.Close() — Close ultimately tears down the DTLS conn, which lets readLoop exit (point 1 still needs to be true for the writer goroutine to actually unblock, though).
3. Workaround: call stream.SetWriteDeadline(...) on the SCTP stream before issuing the DCEP open, so the ctx arm of the select can fire. Requires reaching past webrtc.DataChannel though.
4. Heaviest hammer: don't pre-negotiate the data channels inline with SetRemoteDescription — open them lazily after PC reaches connected so a stuck open never blocks signaling.
Without (1), even (2) leaves the writer goroutine itself parked forever — but at least the PC and its participant-side state would be released; only the SCTP goroutine subtree (much smaller) would leak.
* revert probe stop change
* handle nil offer
`iceServersForParticipant` builds UDP TURN URLs by interpolating the
node IP directly into a format string:
fmt.Sprintf("turn:%s:%d?transport=udp", ip, port)
When `NodeIP.V6` is set, `ToStringSlice()` includes the bare IPv6
address, producing URLs like:
turn:2a05:d014:ee4:1201:7039:38c:f652:a252:443?transport=udp
RFC 3986 §3.2.2 requires IPv6 addresses in URIs to be enclosed in
square brackets. Without them the port is ambiguous and WebRTC clients
(e.g. libdatachannel) reject the URL with "Invalid ICE server port".
Use `net.JoinHostPort` which handles bracketing for IPv6 and is a
no-op for IPv4, producing well-formed URLs:
turn:[2a05:d014:ee4:1201:7039:38c:f652:a252]:443?transport=udp
turn:1.2.3.4:443?transport=udp
* Use Muted in TrackInfo to propagated published track muted.
When the track is muted as a receiver is created, the receiver
potentially was not getting the muted property. That would result in
quality scorer expecting packets.
Use TrackInfo consistently for mute and apply the mute on start up of a
receiver.
* update mute of subscriptions
* fix: ensure num_participants is accurate in webhook events (#4265)
Three fixes for stale/incorrect num_participants in webhook payloads:
1. Move participant map insertion before MarkDirty in join path so
updateProto() counts the new participant.
2. Use fresh room.ToProto() for participant_joined webhook instead of
a stale snapshot captured at session start.
3. Remove direct NumParticipants-- in leave path (inconsistent with
updateProto's IsDependent check), force immediate proto update,
and wait for completion before triggering onClose callbacks.
* fix: use ToProtoConsistent for webhook events instead of forcing immediate updates
* Update go deps
Generated by renovateBot
* update api usage
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: David Zhao <dz@livekit.io>
Raja Subramanian