SendData API messages do not have sender ID or sequence number. So, they
were not cached and hence excluded from the reliable caching feature
which is meant to provide reliability of data channel messages between
the time other participants see participant as ACTIVE (which happens on
ICE connected) and data channel being open (DTLS done + data channels
opened).
Add a small cache for that and flush those messages on data channel
establishment.
* Check for ICE connection before closing participant on signal close.
Only close the participant if ICE has not connected by the time signal
source is closed. If ICE had connected, candidates have been exchanged
and link was established. So, it should be resumable.
Waiting for DTLS closed the participant sometimes in the windowa after
ICE connection, but before DTLS finishes and that unnecessarily closed
the participant forcing a full reconnect.
* variable name
* Flush pending signal responses before closing the web socket.
When the request direction of a signalling connection goes away, the
web socket was closed right away. Responses that the participant had
already sent were dropped.
This loses the leave request on migration. The media node writes
leave(RESUME) and closes the signalling connection just after. The
close won the race, so the client saw a plain web socket close with
code 1000 and never got the leave. It then did a full reconnect
instead of a resume.
Now the response pump is signalled first and drains what is pending,
then the web socket is closed. The producer closes the response
source after its last write, so draining till the source is closed is
a complete flush. A deadline bounds the case where the source stays
open.
The web socket is still closed on all paths, so the ping worker does
not leak.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Give the response pump a margin over the drain deadline.
Both waits used the same timeout and started at about the same time.
So when the drain ran to its deadline, the outer wait could give up at
the same moment and close the web socket while the pump was still
writing. That write failed and the message was lost. It also logged a
timeout even though nothing was stuck.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- Renovate
config:recommended (config:base is deprecated) and matchPackageNames globs instead of the deprecated matchPackagePrefixes.
Vulnerability alerts get a fast path: 2-day quarantine, no concurrency/hourly/schedule limits.
Go modules are no longer grouped into one "go deps" PR — each gets its own, so a bad bump can be reverted alone. The pion modules stay grouped as a documented exception: they're co-released and interdependent, so individual PRs wouldn't build.
First-party github.com/livekit/** skips the 2-week quarantine.
go.mod's go directive is no longer an update target — the build toolchain is pinned in the Dockerfile instead.
Dockerfile deps get pinDigests; the golang image is ungrouped with separateMinorPatch so a patch and a minor bump are each separately approvable.
Custom manager to bump the builder image's -alpineA.B suffix together with its digest, which the stock docker manager holds fixed.
- Pinning
Both Dockerfiles pin golang and alpine by digest alongside the readable tag.
GOTOOLCHAIN=local so a go.mod bump fails loudly instead of silently downloading a different toolchain.
apk upgrade in the runtime stage — a digest pin plus the 2-week quarantine would otherwise ship base-package CVEs Alpine has already fixed. This relies on a cold layer cache, which holds today because the release workflow configures no buildx cache; there's a comment saying so.
Workflows resolve the Go version from the Dockerfile via .github/scripts/go-version.sh, so tests, releases and images share one toolchain.
- Tools
All four code generators now come from the module graph, and tools/tools.go (the pre-Go-1.24 blank-import idiom) is replaced by go.mod tool directives:
tool how why
goimports go tool lives in x/tools — its own module is the one being selected
gotestfmt go tool zero dependencies, nothing to skew
wire go run pins x/tools v0.24.1; building it in our graph changes its output
counterfeiter go run unchanged, matches its //go:generate directives
The wire distinction is load-bearing. Building wire inside our module raises it from the x/tools v0.24.1 it pins to our v0.48.0, and that module version difference changes what it generates: it falls back to v/v2/v3 instead of deriving real identifiers from the type. wire_gen.go is regenerated here to match the in-module build — a cosmetic rename of 9 lines, with no other change to the generated code.
golangci-lint deliberately keeps its action rather than becoming a tool: it pins its own x/tools (v0.44.0 vs our v0.48.0) for the analyzers it bundles, adding it to go.mod would double our go.mod/go.sum (158→338 / 441→889 lines), and the action supplies caching, only-new-issues and PR annotations that invoking a binary can't. Its version stays manual by request.
* Redact stream keys in UpdateStream API log fields
UpdateStream add/remove urls were logged raw into the API request log,
including rtmp stream keys and mux/twitch shorthand keys. Redact them
with utils.RedactStreamKey before appending log fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump protocol for query-value redaction fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Flush the downtrack sequencer on stream restart (Resync, ReceiverRestart,
codec change) so NACK retransmissions can't use metadata that no longer
matches the resynced bucket. Add a defensive bounds guard on the RTX and
forward payload slicing.
Cap the PacketHistory and FrameIntegrityChecker catch-up loops to the ring
size so a large sequence/frame-number jump can't drive a big per-packet
iteration count.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-NACK-packet goroutine spawn in DownTrack.handleRTCP with a
single long-lived worker that coalesces pending NACKs into one retransmit
pass. Pending sequence numbers are capped so a high NACK arrival rate cannot
grow goroutine count or memory unboundedly.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bound the size of HTTP request bodies on the main API listener so large
messages cannot exhaust memory. Configurable via limit.max_api_request_body_size
(defaults to 10 MiB, 0 disables).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Limit number of pending tracks per participant.
Prevents just a signalling connection adding tracks without actually
publishing them growing a large number.
* add to supervisor only if pending track is accepted
With https://github.com/livekit/livekit/pull/4706, there was a case of
some downstream component taking a long time while lock was held. While
the underlying cause of holding a lock while doing callback was removed
in that PR, to catch such cases, some publish side metric anomaly would
be useful to monitor and alert on.
Adding a publish time record for pending tracks on participant close.
That would inflate the publish time for participants not being able to
publish and can be alerted on as it will spike up the value at node
level and at cluster level if multiple nodes have the issue.
* Close web socket connections in all paths.
There was a leak of WebSocket pingWorker if the initial response write
errored as it did not close the WebSocket connection.
* graceful close
* Add configurable read-message size limit on signalling WebSockets
Set a read limit on both the client-facing (/rtc) and agent worker
WebSocket connections so an oversized frame is rejected by the transport
before being buffered. The limits are operator-tunable via
signal_message_size_limit and agent_signal_message_size_limit, both
defaulting to 2 MiB (0 disables).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add tests for signalling WebSocket read-message size limit
Cover the configurable signal_message_size_limit added in the prior
commit:
- config: assert both limits default to 2 MiB and that a YAML override
(including 0 to disable) is parsed correctly.
- full-path integration: a real client connects to /rtc on a single-node
server and an oversized frame is rejected by the transport with a 1009
close; a 0 limit leaves the connection unbounded and signalling
proceeds.
Adds setupSingleNodeTestWithConfig so a single-node server can be started
with config overrides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound decompressed size of signalling WebSocket messages
conn.SetReadLimit only accounts for the compressed bytes read off the
wire, and the client-facing /rtc upgrader negotiates permessage-deflate,
so a small compressed frame could still expand into a much larger buffer
once inflated. Enforce the same limit on the decompressed message by
reading through NextReader + io.LimitReader in WSSignalConnection instead
of the unbounded ReadMessage.
The transport-level SetReadLimit is kept as the cheap wire-level guard;
the new check is the decompressed-size backstop.
Adds NextReader to the WebsocketClient interface (regenerated fake) and a
unit test plus permessage-deflate integration tests covering the
amplification case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover a couple of more cases on data track runt packet handling.
* Guard data track header parser against extensions-size integer wraparound.
Widen the extensions-size arithmetic to int so a 0xFFFF wire value no
longer wraps in uint16, and reject any packet whose computed hdrSize
exceeds the buffer before slicing the payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix publish track count on migration in.
https://github.com/livekit/livekit/pull/4707 addressed the case of
publish tracks overcounting due to synthesised track publish on migrate
in. But, it introduced an issue where published tracks count could go
negative because unpublish subtracted the counter irrespective of the
track actually migrated in or not.
Fix it by keeping track of local publish.
Also, the older code was skipping publisher track count increase if the
synthesised publish was handled first. Address it by checking if the
track is actually new (i. e. fresh local publish) when the track was
already created in the migrate in path.
* fix pub time for tracks published after migration
* test
* prevent multiple track egresses
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
onMediaLossUpdate notified the participant handler directly, which only
sends a leave request with resume action. handleConnectionFailed that
actually switches the ICE preference to TCP/TLS was never called on
this path, so the client reconnected over UDP again and the fallback
kept firing every 30-60s without ever migrating.
Fixeslivekit/livekit#4702
* log high stream start latency.
There is something wrong in measurement as audio is showing high p99
latency. Must be misattributing samples. So, logging for high latency to
understand this better.
* use correct variable
* time since create
* Fix getRefLayerRTPTimestamp off-by-one that can panic on max layer index.
Reject ref/target layers with >= len(refInfos) so layer==len is an error instead of an out-of-range index.
* Remove historical comment from ref-layer bounds test.
Keep the regression coverage without referencing the old bounds check in source.
* Register h264 main profile if enabled explicitly
We don't support the h264 main profile for compatibility,
user can enabled it by set fmtp explicitly in codec config
to enable it if want to use it in special scenario.
* go mod
- Count a publish attempt on a migrating in tarck as there is no
AddTrack for that.
- Add cancel publish only if the participant connection is canceled
- Do not add publish counter for synthetic publish attempts which
happens for migrating in tracks. It will be counted on migrating in
node when the track is actually published, i. e. negotiated/packets
flowing.
* Do not call telemetry listener under pending track lock.
Fix the TrackPublishRequested call of telemetry listener.
Audited other callbacks to ensure that it is not under lock.
* missed some paths of recording it, thanks Devin
* Record subscribe stream start time in prometheus.
Adjust for mutes, i. e. take the last unmute time as the start point and
calculate time till the first byte is sent.
* close the tiny window of race
* Prevent long tail sample when publisher glitches.
Thanks to @milos-lk for this.
Publisher restarting would have reset the layer and would have caused a
sample with very high stream start time. We only need to capture when we
do a dummy start or when the state is seeded to a different node upon
migration.
* reduce a diff
* test
* changed the wrong thing, thank you Devin
* Tests for down stream packet push.
A recent issue (padding bit in RTP header) surfaced a gap which slipped
through due to lack of tests. Changes in pion/rtp were not adopted
properly.
So, adding some tests (thank you Claude for the heavy lifting) to test
the down stream packet path using the whole pion chain.
Split out some interfaces so it is easier to have it all in one place
and create fakes.
Will help adding more tests, for example include the upstream path also
in the integration test. May have to create more interfaces and make
things testable, but this is a start.
* missed file
* rtx specific test
Genseric