Let the docker-backed service tests be skipped with a flag.
TestMain called log.Fatalf when it could not reach a docker daemon, so
the whole package refused to run without one, including every test in it
that needs no container at all.
Record why docker is unavailable instead, and gate the tests that want a
container on it. A run asks to go without them with -docker=false;
otherwise a missing daemon still fails the package, so an unreachable
daemon stays a broken build rather than a run that quietly covers less
than the last one did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Flush pending signal responses before closing the web socket.
When the request direction of a signalling connection goes away, the
web socket was closed right away. Responses that the participant had
already sent were dropped.
This loses the leave request on migration. The media node writes
leave(RESUME) and closes the signalling connection just after. The
close won the race, so the client saw a plain web socket close with
code 1000 and never got the leave. It then did a full reconnect
instead of a resume.
Now the response pump is signalled first and drains what is pending,
then the web socket is closed. The producer closes the response
source after its last write, so draining till the source is closed is
a complete flush. A deadline bounds the case where the source stays
open.
The web socket is still closed on all paths, so the ping worker does
not leak.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Give the response pump a margin over the drain deadline.
Both waits used the same timeout and started at about the same time.
So when the drain ran to its deadline, the outer wait could give up at
the same moment and close the web socket while the pump was still
writing. That write failed and the message was lost. It also logged a
timeout even though nothing was stuck.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- Renovate
config:recommended (config:base is deprecated) and matchPackageNames globs instead of the deprecated matchPackagePrefixes.
Vulnerability alerts get a fast path: 2-day quarantine, no concurrency/hourly/schedule limits.
Go modules are no longer grouped into one "go deps" PR — each gets its own, so a bad bump can be reverted alone. The pion modules stay grouped as a documented exception: they're co-released and interdependent, so individual PRs wouldn't build.
First-party github.com/livekit/** skips the 2-week quarantine.
go.mod's go directive is no longer an update target — the build toolchain is pinned in the Dockerfile instead.
Dockerfile deps get pinDigests; the golang image is ungrouped with separateMinorPatch so a patch and a minor bump are each separately approvable.
Custom manager to bump the builder image's -alpineA.B suffix together with its digest, which the stock docker manager holds fixed.
- Pinning
Both Dockerfiles pin golang and alpine by digest alongside the readable tag.
GOTOOLCHAIN=local so a go.mod bump fails loudly instead of silently downloading a different toolchain.
apk upgrade in the runtime stage — a digest pin plus the 2-week quarantine would otherwise ship base-package CVEs Alpine has already fixed. This relies on a cold layer cache, which holds today because the release workflow configures no buildx cache; there's a comment saying so.
Workflows resolve the Go version from the Dockerfile via .github/scripts/go-version.sh, so tests, releases and images share one toolchain.
- Tools
All four code generators now come from the module graph, and tools/tools.go (the pre-Go-1.24 blank-import idiom) is replaced by go.mod tool directives:
tool how why
goimports go tool lives in x/tools — its own module is the one being selected
gotestfmt go tool zero dependencies, nothing to skew
wire go run pins x/tools v0.24.1; building it in our graph changes its output
counterfeiter go run unchanged, matches its //go:generate directives
The wire distinction is load-bearing. Building wire inside our module raises it from the x/tools v0.24.1 it pins to our v0.48.0, and that module version difference changes what it generates: it falls back to v/v2/v3 instead of deriving real identifiers from the type. wire_gen.go is regenerated here to match the in-module build — a cosmetic rename of 9 lines, with no other change to the generated code.
golangci-lint deliberately keeps its action rather than becoming a tool: it pins its own x/tools (v0.44.0 vs our v0.48.0) for the analyzers it bundles, adding it to go.mod would double our go.mod/go.sum (158→338 / 441→889 lines), and the action supplies caching, only-new-issues and PR annotations that invoking a binary can't. Its version stays manual by request.
* Redact stream keys in UpdateStream API log fields
UpdateStream add/remove urls were logged raw into the API request log,
including rtmp stream keys and mux/twitch shorthand keys. Redact them
with utils.RedactStreamKey before appending log fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump protocol for query-value redaction fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Bound the size of HTTP request bodies on the main API listener so large
messages cannot exhaust memory. Configurable via limit.max_api_request_body_size
(defaults to 10 MiB, 0 disables).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Close web socket connections in all paths.
There was a leak of WebSocket pingWorker if the initial response write
errored as it did not close the WebSocket connection.
* graceful close
* Add configurable read-message size limit on signalling WebSockets
Set a read limit on both the client-facing (/rtc) and agent worker
WebSocket connections so an oversized frame is rejected by the transport
before being buffered. The limits are operator-tunable via
signal_message_size_limit and agent_signal_message_size_limit, both
defaulting to 2 MiB (0 disables).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add tests for signalling WebSocket read-message size limit
Cover the configurable signal_message_size_limit added in the prior
commit:
- config: assert both limits default to 2 MiB and that a YAML override
(including 0 to disable) is parsed correctly.
- full-path integration: a real client connects to /rtc on a single-node
server and an oversized frame is rejected by the transport with a 1009
close; a 0 limit leaves the connection unbounded and signalling
proceeds.
Adds setupSingleNodeTestWithConfig so a single-node server can be started
with config overrides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound decompressed size of signalling WebSocket messages
conn.SetReadLimit only accounts for the compressed bytes read off the
wire, and the client-facing /rtc upgrader negotiates permessage-deflate,
so a small compressed frame could still expand into a much larger buffer
once inflated. Enforce the same limit on the decompressed message by
reading through NextReader + io.LimitReader in WSSignalConnection instead
of the unbounded ReadMessage.
The transport-level SetReadLimit is kept as the cheap wire-level guard;
the new check is the decompressed-size backstop.
Adds NextReader to the WebsocketClient interface (regenerated fake) and a
unit test plus permessage-deflate integration tests covering the
amplification case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix goroutine leak from orphaned signal relay streams
signalService.RelaySignal blocks on the first `<-stream.Channel()` waiting for the StartSession message. psrpc's streamHandler.handleOpenRequest only closes the stream after the handler returns, so if a stream is opened but the client goes away before sending StartSession, the channel is never fed and never closed, and this goroutine blocks forever. Under mass reconnects this leaks one goroutine (and its retained objects) per orphaned stream; they only clear on process restart.
Wrap the initial receive in a select that also returns when the stream context is cancelled or after config.SignalRelay.RetryTimeout, so an orphaned stream returns before Hijack() and psrpc closes it.
Signed-off-by: SKaterinenko <skaterinenko@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
The WHIP connection-notify loop kept issuing RPCs after the participant
had left the room. Guard sendConnectionNotify against a closed
participant (returning ErrParticipantNotFound) and treat that error as a
clean loop exit. Also reorder DeleteSession so the participant is removed
before its WHIP OnClose entry is cleared.
Adds tests covering loop termination on participant leave, context
cancellation, and the closed-participant guard.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* deps
* atomic
* sctp at 1.9.5
* remove proto clone
---------
Co-authored-by: Jacob Gelman <3182119+ladvoc@users.noreply.github.com>
* Prometheus metric for join latency.
Also including a couple of other failures in the signal connection path
and moving the signal connected to after all that.
Not doing counters for the new signal failure paths. I should not have
done for the other two I added a little while ago also (
validation failure and start participant failure) as those are not
scalable to keep adding to node stats. Will probably remove those two
from node stats later. Can add those counters if they are useful.
* deprecate signal failed counters
* service: enforce metadata size limit in CreateRoom, bump default to 512 KiB
CreateRoom previously accepted any metadata size; only UpdateRoomMetadata
rejected oversized payloads. Mirror the same CheckMetadataSize check at
the CreateRoom API boundary so both entrypoints are bounded.
Default MaxMetadataSize moves from 64000 to 512 * 1024 to match the
practical needs of customers using room metadata for richer state. The
limit remains configurable via the existing limits.max_metadata_size knob.
* service: split room vs. participant metadata limit, enforce on join + agent dispatch
LimitConfig.MaxMetadataSize was shared between room metadata and
participant metadata. Last commit's bump to 512 KiB lifted both ceilings;
this restores the participant ceiling to 64 KB and introduces a separate
MaxRoomMetadataSize (default 512 KiB) for room metadata.
Additional enforcement:
- RoomManager.StartSession rejects joins whose JWT-grants metadata or
attributes exceed the participant/attributes limits. The check was
missing entirely from this path.
- AgentDispatchService.CreateDispatch and the embedded
CreateRoomRequest.Agents path now validate metadata and attributes
against the common 64 KB ceilings (previously unbounded).
NewAgentDispatchService gains a LimitConfig parameter; the two wire_gen
callsites are updated.
* service: collapse metadata size limit to single 512 KiB knob
Reverts the LimitConfig split introduced in the previous commit:
MaxRoomMetadataSize, CheckRoomMetadataSize, and the max_room_metadata_size
yaml key are removed. MaxMetadataSize moves back to 512 * 1024 and gates
all metadata uniformly — room (CreateRoom, UpdateRoomMetadata), participant
(UpdateParticipant, signal UpdateMetadata, JWT grants on join), and agent
dispatch (CreateDispatch + embedded RoomAgentDispatch).
MaxAttributesSize stays at 64 KB and continues to gate participant and
agent-dispatch attributes separately.
Test cases consolidated under the single knob.
* kb -> kib
There are several places the participant can drop off after initiating a
connection attempt. Count those places as cancellation including when
participant is closed due to specific reasons.
Cancels should be discounted when determining RTC/ICE connectivity
success/failure percentage.
* agent: thread simulation flag from dispatch to job
Reads simulation from AgentDispatch / RoomAgentDispatch and copies it
onto Job in agent.LaunchJob and the inline room-agent path so workers
see the flag.
Stacked on top of livekit/protocol#1629.
* agent: replace simulation bool with attributes map
Threads the renamed attributes map (was bool simulation) from dispatch
to job and bumps the protocol pseudo-version.
* deps
* Update go deps to v4
Generated by renovateBot
* update dockertest to v4
* fix
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: David Zhao <dz@livekit.io>
* Metrics for participant active, i. e. fully established.
- Egress stub for v2 API
- Fix the participant canceled counter 🤦
- Add active counter -> this is increment when a participant becomes
active, i. e. primary peer connection established. Can be used to
monitor node wise connection establishment issues.
- Add singnalling validation fail counter.
With this, we have
- signalling validation fail
- signalling failed --> this is when the `startSession` fails
- signalling connected -> signalling is succesful and can send back
joinResponse to client
on media connection side
- rtc_init -> start
- rtc_connected -> participant session created (joined)
- rtc_active -> primay peer connection established
- rtc_canceled -> could not proceed with RTC connection due to not being
able to resume.
* signalling counters deps
* revert pion/webrtc to 4.2.12 to get SCTP without interleaving
* go back to pion/webrtc 4.2.11 and sctp 1.9.5
* telemetry: split webhook-processed hook registration out of NewTelemetryService
NewTelemetryService used to register a notifier processed-hook on the inner
*telemetryService directly. That made it impossible for downstream wrappers
(e.g. cloud's TelemetryService that overrides Webhook to fan out to a v3
observability pipeline) to intercept webhook events without double-firing
the legacy emission.
Lift the registration into a new exported helper RegisterWebhookHook, and
have the standalone server's wire provider createTelemetryService call it
right after construction so behavior is unchanged for callers that don't
wrap the service.
When a client hits /rtc/v[01]/validate with a base64 WrappedJoinRequest
whose embedded JoinRequest.ClientInfo is unset, validateInternal called
AugmentClientInfo with a nil *ClientInfo and panicked at ci.Address =
GetClientIP(req). The non-wrapped branch already allocates via
ParseClientInfo; do the same here so pi.Client always gets at least the
resolved client Address.
