* utils: make Median generic, overflow-safe, and add comprehensive
* tests: fix staticcheck unused variable warning in changenotifier_test.go
* tests: switch from assert to require for consistency with existing tests
* trigger ci rerun
In migration cases, dummy receiver trackInfo is used by relay tracks to
set up the receivers and those need the proper track info.
Also check for proper receiver when adding a migrated track.
TestUnsubscribe checked that the changed-notifier observer was gone as
soon as the unsubscribe had settled, but setDesired leaves the
RemoveObserver call to a goroutine of its own and nothing the test waits
on orders against it, so CI caught the assertion running first.
TestSubscribe has the same defect on the unsubscribed callback, which
unmarkSubscribedTo delivers with a bare go while the subscribed one is
called inline.
Wait for both, each with a message of its own, so that a leak that is
real still says which of them broke.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Let the docker-backed service tests be skipped with a flag.
TestMain called log.Fatalf when it could not reach a docker daemon, so
the whole package refused to run without one, including every test in it
that needs no container at all.
Record why docker is unavailable instead, and gate the tests that want a
container on it. A run asks to go without them with -docker=false;
otherwise a missing daemon still fails the package, so an unreachable
daemon stays a broken build rather than a run that quietly covers less
than the last one did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pion no longer starts the repair stream reader when a custom BufferFactory
is set, so the mid/rid/rsid extensions were never observed and simulcast RTX
streams were never paired with their primary streams.
Extract the extensions on the buffer write path instead. Migrated publishers
send no extensions at all, so pair those from SimTracks.
Adds an integration test covering both paths, and moves the vnet setup it
shares with the downtrack test into pkg/testutils/vnettest.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
VP9 SVC sends each spatial layer as its own encoded frame,
so a picture carries one trailer per layer, but only the
top layer's last packet has the RTP marker bit set.
Fix https://github.com/livekit/egress/issues/1347
* Reduce locking in media track + telemetry listener on move participant.
Telemetry listener was not getting it from new room on room move.
* update comment
* fix signal bytes reporting
* room aware telemetry listener in media track
* data track telemetry listener
SendData API messages do not have sender ID or sequence number. So, they
were not cached and hence excluded from the reliable caching feature
which is meant to provide reliability of data channel messages between
the time other participants see participant as ACTIVE (which happens on
ICE connected) and data channel being open (DTLS done + data channels
opened).
Add a small cache for that and flush those messages on data channel
establishment.
* Check for ICE connection before closing participant on signal close.
Only close the participant if ICE has not connected by the time signal
source is closed. If ICE had connected, candidates have been exchanged
and link was established. So, it should be resumable.
Waiting for DTLS closed the participant sometimes in the windowa after
ICE connection, but before DTLS finishes and that unnecessarily closed
the participant forcing a full reconnect.
* variable name
* Flush pending signal responses before closing the web socket.
When the request direction of a signalling connection goes away, the
web socket was closed right away. Responses that the participant had
already sent were dropped.
This loses the leave request on migration. The media node writes
leave(RESUME) and closes the signalling connection just after. The
close won the race, so the client saw a plain web socket close with
code 1000 and never got the leave. It then did a full reconnect
instead of a resume.
Now the response pump is signalled first and drains what is pending,
then the web socket is closed. The producer closes the response
source after its last write, so draining till the source is closed is
a complete flush. A deadline bounds the case where the source stays
open.
The web socket is still closed on all paths, so the ping worker does
not leak.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Give the response pump a margin over the drain deadline.
Both waits used the same timeout and started at about the same time.
So when the drain ran to its deadline, the outer wait could give up at
the same moment and close the web socket while the pump was still
writing. That write failed and the message was lost. It also logged a
timeout even though nothing was stuck.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- Renovate
config:recommended (config:base is deprecated) and matchPackageNames globs instead of the deprecated matchPackagePrefixes.
Vulnerability alerts get a fast path: 2-day quarantine, no concurrency/hourly/schedule limits.
Go modules are no longer grouped into one "go deps" PR — each gets its own, so a bad bump can be reverted alone. The pion modules stay grouped as a documented exception: they're co-released and interdependent, so individual PRs wouldn't build.
First-party github.com/livekit/** skips the 2-week quarantine.
go.mod's go directive is no longer an update target — the build toolchain is pinned in the Dockerfile instead.
Dockerfile deps get pinDigests; the golang image is ungrouped with separateMinorPatch so a patch and a minor bump are each separately approvable.
Custom manager to bump the builder image's -alpineA.B suffix together with its digest, which the stock docker manager holds fixed.
- Pinning
Both Dockerfiles pin golang and alpine by digest alongside the readable tag.
GOTOOLCHAIN=local so a go.mod bump fails loudly instead of silently downloading a different toolchain.
apk upgrade in the runtime stage — a digest pin plus the 2-week quarantine would otherwise ship base-package CVEs Alpine has already fixed. This relies on a cold layer cache, which holds today because the release workflow configures no buildx cache; there's a comment saying so.
Workflows resolve the Go version from the Dockerfile via .github/scripts/go-version.sh, so tests, releases and images share one toolchain.
- Tools
All four code generators now come from the module graph, and tools/tools.go (the pre-Go-1.24 blank-import idiom) is replaced by go.mod tool directives:
tool how why
goimports go tool lives in x/tools — its own module is the one being selected
gotestfmt go tool zero dependencies, nothing to skew
wire go run pins x/tools v0.24.1; building it in our graph changes its output
counterfeiter go run unchanged, matches its //go:generate directives
The wire distinction is load-bearing. Building wire inside our module raises it from the x/tools v0.24.1 it pins to our v0.48.0, and that module version difference changes what it generates: it falls back to v/v2/v3 instead of deriving real identifiers from the type. wire_gen.go is regenerated here to match the in-module build — a cosmetic rename of 9 lines, with no other change to the generated code.
golangci-lint deliberately keeps its action rather than becoming a tool: it pins its own x/tools (v0.44.0 vs our v0.48.0) for the analyzers it bundles, adding it to go.mod would double our go.mod/go.sum (158→338 / 441→889 lines), and the action supplies caching, only-new-issues and PR annotations that invoking a binary can't. Its version stays manual by request.
* Redact stream keys in UpdateStream API log fields
UpdateStream add/remove urls were logged raw into the API request log,
including rtmp stream keys and mux/twitch shorthand keys. Redact them
with utils.RedactStreamKey before appending log fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump protocol for query-value redaction fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Flush the downtrack sequencer on stream restart (Resync, ReceiverRestart,
codec change) so NACK retransmissions can't use metadata that no longer
matches the resynced bucket. Add a defensive bounds guard on the RTX and
forward payload slicing.
Cap the PacketHistory and FrameIntegrityChecker catch-up loops to the ring
size so a large sequence/frame-number jump can't drive a big per-packet
iteration count.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-NACK-packet goroutine spawn in DownTrack.handleRTCP with a
single long-lived worker that coalesces pending NACKs into one retransmit
pass. Pending sequence numbers are capped so a high NACK arrival rate cannot
grow goroutine count or memory unboundedly.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bound the size of HTTP request bodies on the main API listener so large
messages cannot exhaust memory. Configurable via limit.max_api_request_body_size
(defaults to 10 MiB, 0 disables).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Limit number of pending tracks per participant.
Prevents just a signalling connection adding tracks without actually
publishing them growing a large number.
* add to supervisor only if pending track is accepted
With https://github.com/livekit/livekit/pull/4706, there was a case of
some downstream component taking a long time while lock was held. While
the underlying cause of holding a lock while doing callback was removed
in that PR, to catch such cases, some publish side metric anomaly would
be useful to monitor and alert on.
Adding a publish time record for pending tracks on participant close.
That would inflate the publish time for participants not being able to
publish and can be alerted on as it will spike up the value at node
level and at cluster level if multiple nodes have the issue.
* Close web socket connections in all paths.
There was a leak of WebSocket pingWorker if the initial response write
errored as it did not close the WebSocket connection.
* graceful close
* Add configurable read-message size limit on signalling WebSockets
Set a read limit on both the client-facing (/rtc) and agent worker
WebSocket connections so an oversized frame is rejected by the transport
before being buffered. The limits are operator-tunable via
signal_message_size_limit and agent_signal_message_size_limit, both
defaulting to 2 MiB (0 disables).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add tests for signalling WebSocket read-message size limit
Cover the configurable signal_message_size_limit added in the prior
commit:
- config: assert both limits default to 2 MiB and that a YAML override
(including 0 to disable) is parsed correctly.
- full-path integration: a real client connects to /rtc on a single-node
server and an oversized frame is rejected by the transport with a 1009
close; a 0 limit leaves the connection unbounded and signalling
proceeds.
Adds setupSingleNodeTestWithConfig so a single-node server can be started
with config overrides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound decompressed size of signalling WebSocket messages
conn.SetReadLimit only accounts for the compressed bytes read off the
wire, and the client-facing /rtc upgrader negotiates permessage-deflate,
so a small compressed frame could still expand into a much larger buffer
once inflated. Enforce the same limit on the decompressed message by
reading through NextReader + io.LimitReader in WSSignalConnection instead
of the unbounded ReadMessage.
The transport-level SetReadLimit is kept as the cheap wire-level guard;
the new check is the decompressed-size backstop.
Adds NextReader to the WebsocketClient interface (regenerated fake) and a
unit test plus permessage-deflate integration tests covering the
amplification case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover a couple of more cases on data track runt packet handling.
* Guard data track header parser against extensions-size integer wraparound.
Widen the extensions-size arithmetic to int so a 0xFFFF wire value no
longer wraps in uint16, and reject any packet whose computed hdrSize
exceeds the buffer before slicing the payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix publish track count on migration in.
https://github.com/livekit/livekit/pull/4707 addressed the case of
publish tracks overcounting due to synthesised track publish on migrate
in. But, it introduced an issue where published tracks count could go
negative because unpublish subtracted the counter irrespective of the
track actually migrated in or not.
Fix it by keeping track of local publish.
Also, the older code was skipping publisher track count increase if the
synthesised publish was handled first. Address it by checking if the
track is actually new (i. e. fresh local publish) when the track was
already created in the migrate in path.
* fix pub time for tracks published after migration
* test
* prevent multiple track egresses
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
onMediaLossUpdate notified the participant handler directly, which only
sends a leave request with resume action. handleConnectionFailed that
actually switches the ICE preference to TCP/TLS was never called on
this path, so the client reconnected over UDP again and the fallback
kept firing every 30-60s without ever migrating.
Fixeslivekit/livekit#4702
* log high stream start latency.
There is something wrong in measurement as audio is showing high p99
latency. Must be misattributing samples. So, logging for high latency to
understand this better.
* use correct variable
* time since create
* Fix getRefLayerRTPTimestamp off-by-one that can panic on max layer index.
Reject ref/target layers with >= len(refInfos) so layer==len is an error instead of an out-of-range index.
* Remove historical comment from ref-layer bounds test.
Keep the regression coverage without referencing the old bounds check in source.
* Register h264 main profile if enabled explicitly
We don't support the h264 main profile for compatibility,
user can enabled it by set fmtp explicitly in codec config
to enable it if want to use it in special scenario.
* go mod
Sanjay P