* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover a couple of more cases on data track runt packet handling.
* Guard data track header parser against extensions-size integer wraparound.
Widen the extensions-size arithmetic to int so a 0xFFFF wire value no
longer wraps in uint16, and reject any packet whose computed hdrSize
exceeds the buffer before slicing the payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix publish track count on migration in.
https://github.com/livekit/livekit/pull/4707 addressed the case of
publish tracks overcounting due to synthesised track publish on migrate
in. But, it introduced an issue where published tracks count could go
negative because unpublish subtracted the counter irrespective of the
track actually migrated in or not.
Fix it by keeping track of local publish.
Also, the older code was skipping publisher track count increase if the
synthesised publish was handled first. Address it by checking if the
track is actually new (i. e. fresh local publish) when the track was
already created in the migrate in path.
* fix pub time for tracks published after migration
* test
* prevent multiple track egresses
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
onMediaLossUpdate notified the participant handler directly, which only
sends a leave request with resume action. handleConnectionFailed that
actually switches the ICE preference to TCP/TLS was never called on
this path, so the client reconnected over UDP again and the fallback
kept firing every 30-60s without ever migrating.
Fixeslivekit/livekit#4702
* log high stream start latency.
There is something wrong in measurement as audio is showing high p99
latency. Must be misattributing samples. So, logging for high latency to
understand this better.
* use correct variable
* time since create
* Fix getRefLayerRTPTimestamp off-by-one that can panic on max layer index.
Reject ref/target layers with >= len(refInfos) so layer==len is an error instead of an out-of-range index.
* Remove historical comment from ref-layer bounds test.
Keep the regression coverage without referencing the old bounds check in source.
* Register h264 main profile if enabled explicitly
We don't support the h264 main profile for compatibility,
user can enabled it by set fmtp explicitly in codec config
to enable it if want to use it in special scenario.
* go mod
- Count a publish attempt on a migrating in tarck as there is no
AddTrack for that.
- Add cancel publish only if the participant connection is canceled
- Do not add publish counter for synthetic publish attempts which
happens for migrating in tracks. It will be counted on migrating in
node when the track is actually published, i. e. negotiated/packets
flowing.
* Do not call telemetry listener under pending track lock.
Fix the TrackPublishRequested call of telemetry listener.
Audited other callbacks to ensure that it is not under lock.
* missed some paths of recording it, thanks Devin
* Record subscribe stream start time in prometheus.
Adjust for mutes, i. e. take the last unmute time as the start point and
calculate time till the first byte is sent.
* close the tiny window of race
* Prevent long tail sample when publisher glitches.
Thanks to @milos-lk for this.
Publisher restarting would have reset the layer and would have caused a
sample with very high stream start time. We only need to capture when we
do a dummy start or when the state is seeded to a different node upon
migration.
* reduce a diff
* test
* changed the wrong thing, thank you Devin
* Tests for down stream packet push.
A recent issue (padding bit in RTP header) surfaced a gap which slipped
through due to lack of tests. Changes in pion/rtp were not adopted
properly.
So, adding some tests (thank you Claude for the heavy lifting) to test
the down stream packet path using the whole pion chain.
Split out some interfaces so it is easier to have it all in one place
and create fakes.
Will help adding more tests, for example include the upstream path also
in the integration test. May have to create more interfaces and make
things testable, but this is a start.
* missed file
* rtx specific test
Addresses https://github.com/livekit/livekit/issues/4689
We were probably missing a couple of bits with this
1. Not affected for regular traffic like from browsers as it does not
add padding, but special clients were affected.
2. Probe packets were probably using wrong last byte as pion/rtp would
have overwrriten with 0 because the header.PaddingSize for the newer
versions were not set. That could have affected bandwidth estimation
catch up.
Following up on this PR (https://github.com/livekit/livekit/pull/4678 -
great catch btw, thank you), propagating the spreader threshold to RED
receivers also.
Removing the test as it was a simple case and that would not have caught
the original issue.
In single-PC and one-shot signalling modes the subscriber PCTransport is
never created, so t.subscriber is nil. Every other TransportManager method
that touches the subscriber nil-checks it first; HandleAnswer did not, so a
client sending an SDP answer in those modes crashed the process with a nil
pointer dereference. Guard it and log, matching AddICECandidate/Negotiate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Was doing it on participant close. But, that misaligns `rtc_init`
(number of attempts) a bit if the participant sessions are long.
Record it as soon as RTC connects so that it is more time aligned with
`rtc_init`.
* fix: don't undercount data channel bitrate over idle gaps
BitrateCalculator divided drained bytes by wall-clock time, including
idle gaps between sparse writes. A fast channel fed sparsely (e.g. 1 KB
every second, acked in a few ms) therefore reported its low offered load
(~8 kbps) instead of its drain capacity (~800 kbps). In the unreliable
writer that shrinks targetLatencyLimit and drops bursts on a channel
that is actually keeping up.
Measure the rate over backlogged ("busy") time only, so idle time no
longer dilutes the estimate. Backlog is detected from the buffer
occupancy just before a write (bufferedAmount - bytesWritten): since no
bytes are added between writes the buffer only drains, so a non-zero
pre-write level means it never emptied and the interval was genuinely
busy. Gating on the post-write buffered amount would be wrong -- it
always includes the just-enqueued (not yet SACKed) bytes and is ~never
zero, collapsing the estimate back to wall-clock. When no backlog is
ever observed the calculator reports no estimate (ok=false) instead of a
confidently-low number.
Introduce BitrateMode:
- BusyOnly (writer default): busy denominator, all drained bytes counted.
- ExcludeIdleDrain: also drops bytes drained across non-backlogged
intervals for a clean drain-capacity estimate free of idle
contamination.
- WallClock: elapsed-time denominator, all bytes; for the test-client
reader which has no send buffer to observe.
Call sites keep the conservative behaviour (writers BusyOnly, reader
WallClock). Tests exercise both writer modes and the mixed idle-then-
backlog window where they diverge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Trim comments in data channel bitrate calculator
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make BitrateMode.String a method on the enum
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix goroutine leak from orphaned signal relay streams
signalService.RelaySignal blocks on the first `<-stream.Channel()` waiting for the StartSession message. psrpc's streamHandler.handleOpenRequest only closes the stream after the handler returns, so if a stream is opened but the client goes away before sending StartSession, the channel is never fed and never closed, and this goroutine blocks forever. Under mass reconnects this leaks one goroutine (and its retained objects) per orphaned stream; they only clear on process restart.
Wrap the initial receive in a select that also returns when the stream context is cancelled or after config.SignalRelay.RetryTimeout, so an orphaned stream returns before Hijack() and psrpc closes it.
Signed-off-by: SKaterinenko <skaterinenko@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Some data track related tests are failing due to setting a target
latency of 100ms. Would be good to understand if taget latency or min
buffer is causing the drops.
* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
