* rtc: apply simulcast rids from SDP for WHIP (one-shot) publishers
In one-shot signalling mode HandleRemoteDescription set the remote offer
directly and never called OnSetRemoteDescriptionOffer, so
updateRidsFromSDP did not run for WHIP participants. Every simulcast layer
then mapped to spatial layer 0 and all but the first were rejected with
'duplicate layer'.
* rtc: test that a one-shot simulcast offer applies rids to the pending track
* rtc: use h/q rids in one-shot simulcast test, cover single-layer offer
Name the simulcast test layers medium/h and low/q per convention, and add
TestOneShotSingleLayerOfferUnchanged to check a one-shot offer without rids
still yields a single layer with rid "" at spatial layer 0.
* rtc: reclaim abandoned pending tracks
A pending track is freed only when its first media arrives or the
participant leaves. A client that calls AddTrack and then gives up
before any media flows leaves the slot held for the whole session.
After 20 such slots, every later AddTrack is rejected.
Reclaim these slots when the cap is hit, if no media arrived and the
slot is older than the publish timeout. Keep the supervisor monitor in
step for queued tracks that share a track id. Add a publish_timeout
config value, default 30s.
Fixes: #4970
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: fix publish_timeout comment to not name an internal constant
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: release supervisor-timed-out pending tracks at the limit
Replace the createdAt sweep with the supervisor's publish timeout. The
supervisor counts time only while the publisher is connected and the
track is unmuted, and never reports a live track, so a successor queued
behind a live track is not released.
Restart the publish timer when a queued publish reaches the front of
the queue, so it does not inherit the clock of the track ahead of it.
Write the publication error under the write lock, since it is now read
from another goroutine.
Drop the publish_timeout config value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: skip stale supervisor timeouts and keep live monitors
TimedOutPublications re-checks the monitor, because the cached error is
cleared only on the next tick. A track that published since is not
reported, so its queued successor is not released.
RemovePublication keeps a monitor that still holds a published track.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: drop late media for released pending tracks
Media or an offer with no matching pending track falls back to the first
pending track of the same type. After a release, late media for the
released track could take another pending track's identity.
Record the released track's signal and SDP client ids, the 32 most
recent, and drop media or offers for them. An AddTrack that reuses the
client id clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: clear all ids of a released track when it is re-added
A re-add cleared only the signal client id, so a Firefox republish that
keeps the old SDP id had its offer and media dropped. Keep the ids of
each released track together, and clear the whole track when a new
AddTrack reuses its signal or simulcast codec client id. The limit now
counts released tracks, and a track ages out with all of its ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
GetRecentlyDisconnectedParticipants walked the participant update cache
with Get, which moves each entry to the front of the LRU. Walking from
newest to oldest left the cache in reverse order, so after a reconnect
the next updates evicted the most recently updated participants first.
In a room where the subscriber has tracked 128 or more participants, a
participant that had just disconnected was then dropped from the cache
and a later reconnect did not report it as disconnected, and the
out-of-order version check was lost for recently updated participants.
Use Peek, which reads an entry without changing its recency.
A subscribe that found the reconcile queue full was dropped and waited
for the 3 s ticker. Flag the overflow, and have the worker reconcile
all subscriptions after its next queued item.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: keep one subscription when first subscribes race
SubscribeToTrack and SubscribeToDataTrack looked up the subscription,
then created one and stored it under a separate lock. Two concurrent
first subscribes, or a subscribe and a settings update, could each
store their own entry. The replaced entry could still add the down
track, which nothing owned after that, and the settings could land on
the replaced entry.
Find or create the subscription and set it as desired under one lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: keep early settings and a racing data track resubscribe
A settings or options update before the first subscribe creates an
entry that is not desired. Cleanup removed it on the next pass, so the
subscribe created a new entry without the settings. Keep such an entry
for notFoundTimeout.
After a data track unsubscribe, the entry was deleted even if a
subscribe set it as desired during the removal. Delete it only if it
is still not desired, else reconcile it again.
Cleanup deletes only the entry it checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: use the fast debounce for the first negotiation
A new transport set lastNegotiate to its creation time. A non-forced
Negotiate soon after creation then waited for the 150 ms debounce
instead of 10 ms, although nothing was negotiated yet. Start from the
zero time so that the first offer uses the fast debounce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the first negotiation condition without a deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check that the first offer goes out before a debounced one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test: give mock participants a connection quality
A real participant never returns a nil connection quality. The mock
did, which can crash code that does not expect nil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: close test rooms when each test ends
Most room tests left their rooms open, so each room's workers kept
running for the rest of the package.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
selectAlternativeVideoCodec walked the enabled codecs and returned the
first one with a video/ prefix. video/rtx and video/flexfec-03 both have
that prefix, so when the requested codec was not enabled the track could
fall back to RTX. RTX is a retransmission stream, not a real codec, so
the track got registered as video/rtx and subscribers saw a blank tile.
Skip RTX and FEC entries so the fallback is always a valid video codec.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
send_side_bwe_pacer accepted any string and silently ran no-queue. leaky-bucket
is a defined PacerBehavior with a full LeakyBucket implementation, but the
selection switch in NewPCTransport only handled pass-through and no-queue, so
both leaky-bucket and outright typos fell through to the default with no
diagnostic.
leaky-bucket is deliberately left unwired here. LeakyBucket paces to a bitrate
supplied via Pacer.SetBitrate, and no production code calls SetBitrate or
SetInterval -- Base no-ops both and the only caller in the tree is the
test-only capturingPacer. Wiring it as-is would pace at a static bitrate and
ignore the bandwidth estimate, which is worse than the fallback.
Instead, make the existing behaviour visible: CongestionControlConfig.Validate
rejects unrecognised values at startup, the fallback logs a warning naming the
requested pacer, and config-sample.yaml documents send_side_bwe_pacer and
use_send_side_bwe for the first time.
Refs #4724
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* rtc: record a room's departure when the participant leaves it
RemoveParticipant removed the participant from the room, closed it, and
only then recorded leftAt. CloseIfEmpty runs every second, so a tick
landing during the close saw an empty room with a stale or zero
departure time and closed it at once: with no departure recorded it
compares against the room's creation time and empty_timeout, and with an
earlier departure it measures the grace period from that older
timestamp. Either way departure_timeout is skipped.
Record the departure where the participant is removed, under the lock
that already covers it.
* rtc: drop the removal wait from the departure timeout test
The close signal already pins the interleaving, and the room's close
state is decided by the CloseIfEmpty call itself.
* rtc: split EndSession out of MoveToRoom
A participant's session in a room can end without the participant
closing: today on a move, and next on a room re-key, where a room ended
by the sweeper while participants were joining restarts under a new room
id with the same participants in it.
EndSession is the room-independent half of MoveToRoom: report the
published tracks unpublished, report the session end, run the caller's
leave under the ending session's telemetry guard, then take a fresh
guard and reset the deferred resolvers for the next session. The guard
matters: ParticipantLeft releases it and the telemetry service refuses
to open a stats worker for a released guard, so without a fresh one the
next session's join would silently get no worker.
MoveToRoom keeps the subscriber teardown and the onClose fan-out, passed
in as the leave.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rtc: rename EndSession to LeaveSession, add JoinSession
EndSession read as terminal, but the participant survives it and joins
another session, so name the pair for what happens to the participant:
LeaveSession and JoinSession.
JoinSession is the other half that review pointed out was missing from
the API: the caller reports the join, then the published tracks are
reported published again under the new session, pairing the unpublish
LeaveSession reported. Without it the re-publish lived only in the
caller, with nothing in OSS proving the two halves balance.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
participants.
While migrating, the old node will have the recently disconnected
participants in the migrating out participant's cache and needs to sent
out to the participant from the migrating in node via the new WebSocket.
This method can be used to exchange that data between the two nodes.
Keeping it as `recently` disconnected and leaving the semantics upto how
recent for implementation.
There are reports of disconnected participant update missing -
https://github.com/livekit/client-sdk-js/pull/2096. That results in
clients retaining disconnected participant(s) in its roster.
Two changes
1. Look for participant disconnects a little farther back than last
signal time as client app might have lost visibility earlier.
2. Look at all participants within the relaxed window above. The update
cache is LRU. So, in theory can stop when an entry is before the time
threshold above. But, theoretically it is possible that `updatedAt`
time and LRU queuing time can flip ordering although very unlikely.
I think we should do one of the following to make it more robust, but
both require client side changes
1. Send full participant list in `ReconnectResponse`. Clients can use
that to drop participants not in the list.
2. Add a sequence number to all signal responses (maybe signal requests
as well) and have client send up the last response it saw in
`SyncState` message. Server can then replay all the messages after
that. This also requires a message cache on server side.
* dynacast: re-send subscribed qualities after every publisher answer
The SFU answers publisher offers without the simulcast pause markers
(SetIgnoreRidPauseForRecv), so applying the answer re-enables in the
browser the layers that dynacast had paused. Dynacast only notifies
changes, so those layers stayed on until the subscribed quality of the
track changed again.
Send the publisher its committed subscribed qualities again 1 s after
each answer, once the client has applied it: livekit-client handles the
next signal message without waiting for the answer to be applied.
Fixes https://github.com/livekit/livekit/issues/4906
* dynacast: resend once per burst of publisher answers, only for paused layers
In SPC, every subscription change of a participant goes through a new publisher offer and answer, so answers come in bursts when a room is busy. Each answer scheduled its own resend of the subscribed qualities, sending the same update several times in a row.
Thist change keeps one pending resend per participant instead. It fires 2 s after the last answer, and never later than 8 s after the first answer of the burst, so a steady stream of answers cannot postpone it forever. It also skips the resend for tracks whose committed qualities are all HIGH: with every layer enabled, an answer has nothing paused to re-enable. We can save those messages.
* agent: release JobTerminate handler when job ends before registration
JobRequest registered the job's JobTerminate handler after AssignJob
returned, so an ended UpdateJobStatus or worker disconnect handled in
between deregistered nothing and the handler leaked. AssignJob also sent
the assignment before recording the job as running, so an immediate
ended update was dropped.
Record the job before sending the assignment, and after registering the
handler recheck under h.mu that the job is still running and its worker
still registered, deregistering otherwise.
Fixes#4901
* replace utils.CloneProto with proto.CloneOf
* agent: wait on worker readiness and poll for released handlers in test
* Increase receiver loadbalance threshold when batch io enabled
With batch IO the send syscall moves to the BatchConn flush goroutine, so WriteRTP only
translates the packet and enqueues it. Increash the threshold to reduce
* lint
* Do not request media sections for transceivers that cannot send
getNumUnmatchedTransceivers counted every local transceiver without a mid.
A transceiver whose track was removed before its section was negotiated
(direction recvonly, no sender) or that was stopped (inactive) can never be
matched to the recvonly section the remote adds in response to a
MediaSectionsRequirement, so it stayed "outstanding" forever: every answer
asked the remote for a section for it again, the remote appended another
batch of transceivers each round, and its offer grew by a fixed amount per
negotiation until it exceeded limit.signal_message_size_limit and the
signalling connection was closed (full reconnect, then the cycle restarts).
Count only transceivers that have something to send. A released transceiver
that AddTrack reuses later becomes sendrecv and is counted again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Address review: blank lines between test functions, assert transceiver reuse
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* test: stop registering the global DefaultConfig with the logger
InitLoggerFromConfig hands the pointer to zaputil.ComponentLeveler as its
level resolver, and protocol a879e94 gave logger.Config a
ResolveComponentLevel that takes c.lock. The two test packages passed
&config.DefaultConfig.Logging, so every first-time component level
resolution locked a mutex inside the global DefaultConfig while
config.NewConfig marshalled that same global from another goroutine:
Read at 0x39baa08 yaml.Marshal -> pkg/config/config.go:633 (NewConfig)
Write at 0x39baa08 ComponentLeveler.resolve -> zaputil/leveler.go:117
That tripped the race detector in pkg/rtc TestPreferMediaCodecForPublisher
and broke CI on master.
Register a test-local LoggingConfig instead, so the logger never touches
the global that NewConfig reads. DefaultConfig.Logging only ever sets
PionLevel, so behavior is unchanged. A plain struct copy is not an option
here: logger.Config holds a mutex and vet's copylocks would reject it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: make RTCClient callbacks safe to set while the client runs
The scenario tests assign c2.OnDataReceived after waitUntilConnected, so
the assignment raced the data channel goroutine already reading the field
in handleDataMessage:
Read test/client.(*RTCClient).handleDataMessage client.go:1146
Write test.scenarioDataPublish.func1 scenarios.go:160
This is what failed TestMultinodeDataPublishing on master.
Replace the three exported callback fields with atomic pointers behind
SetOnConnected/SetOnDataReceived/SetOnDataUnlabeledReceived. OnConnected
and OnDataUnlabeledReceived have no writers today, but they are read from
the same background goroutines and would race the moment one appeared, so
all three move together rather than leaving a split API on one struct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
SubscriptionManager tracks data-track subscriptions but exposed no accessor
for the active ones, so callers could see subscribed and published media tracks
and published data tracks, but not subscribed data tracks. Add
GetSubscribedDataTracks(), returning the bound data down-tracks, mirroring
GetSubscribedTracks() for media, so a participant's full track set can be
accounted for. Regenerated the LocalParticipant fake.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: send connection quality to participants that subscribe after their first update
Added tracking for participants' connection quality updates to ensure all subscribed participants receive necessary updates, even if they were added after the last quality announcement
* rtc: record sent connection quality only after a successful send
A failed SendConnectionQualityUpdate must not mark the participant as informed, otherwise the update is never retried while qualities stay stable. Also simplify the untold-subscription check.
* fix: hold signal messages until the ReconnectResponse goes out
Clients take the ReconnectResponse as the first message on a resumed or
migrated in signal connection, anything ahead of it is dropped. Hold
messages back until it has been written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: keep queued participant updates on a path that always flushes
Queue only while the participant is not ready, that queue is always
drained by the join or reconnect response. Log a dropped SDP, it leaves
the negotiation waiting until the state machine recovers it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: flush queued updates whenever the connection opens
Queue participant updates while the handshake is pending again, and give
the signaller a hook that fires when the connection opens, on an explicit
open and on the handshake window expiring, so the queue always drains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: drive the handshake window off a timer and read the gate atomically
The window ran only when something asked whether the handshake was
pending, so a connection with nothing else to send held its queue.
Reading the gate under the lock the flush takes closes the race where an
update queued just after a flush.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: ignore a handshake timeout from a wait that has ended
Stop cannot cancel a timeout that is already running, so tag each wait
and let a timeout act only on its own. Count opens atomically in the
test, the timer fires on its own goroutine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* report a reason on room-ended telemetry
Room.Close already took a ParticipantCloseReason, but OnClose was func()
with no arguments, so the reason was structurally dropped before it could
reach RoomEnded.
Add types.RoomCloseReason and carry it through Close -> OnClose -> RoomEnded,
where it lands on both the analytics event and the room_finished webhook.
Room.Close now takes the room reason and derives the participant reason from
it, so the two can never disagree. ToParticipantCloseReason maps each reason
to exactly what its call site passed before, so no participant-facing
behaviour changes; a table test pins that mapping.
* fix(test): update webhook test for RoomCloseReason
test/ was outside the packages checked before pushing, so this Close call
site was missed. Also assert the reason reaches the room_finished webhook,
which the unit test cannot cover since it runs with a nil notifier.
In migration cases, dummy receiver trackInfo is used by relay tracks to
set up the receivers and those need the proper track info.
Also check for proper receiver when adding a migrated track.
TestUnsubscribe checked that the changed-notifier observer was gone as
soon as the unsubscribe had settled, but setDesired leaves the
RemoveObserver call to a goroutine of its own and nothing the test waits
on orders against it, so CI caught the assertion running first.
TestSubscribe has the same defect on the unsubscribed callback, which
unmarkSubscribedTo delivers with a bare go while the subscribed one is
called inline.
Wait for both, each with a message of its own, so that a leak that is
real still says which of them broke.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
pion no longer starts the repair stream reader when a custom BufferFactory
is set, so the mid/rid/rsid extensions were never observed and simulcast RTX
streams were never paired with their primary streams.
Extract the extensions on the buffer write path instead. Migrated publishers
send no extensions at all, so pair those from SimTracks.
Adds an integration test covering both paths, and moves the vnet setup it
shares with the downtrack test into pkg/testutils/vnettest.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Reduce locking in media track + telemetry listener on move participant.
Telemetry listener was not getting it from new room on room move.
* update comment
* fix signal bytes reporting
* room aware telemetry listener in media track
* data track telemetry listener
SendData API messages do not have sender ID or sequence number. So, they
were not cached and hence excluded from the reliable caching feature
which is meant to provide reliability of data channel messages between
the time other participants see participant as ACTIVE (which happens on
ICE connected) and data channel being open (DTLS done + data channels
opened).
Add a small cache for that and flush those messages on data channel
establishment.
* Check for ICE connection before closing participant on signal close.
Only close the participant if ICE has not connected by the time signal
source is closed. If ICE had connected, candidates have been exchanged
and link was established. So, it should be resumable.
Waiting for DTLS closed the participant sometimes in the windowa after
ICE connection, but before DTLS finishes and that unnecessarily closed
the participant forcing a full reconnect.
* variable name
David Chen