Route presence like a reverse proxy keyed on (project, agent_name, deployment):
any of a deployment's workers is a candidate and the worker's own router returns
the real status (a 404 for a path it doesn't serve during a rolling deploy is
returned, not re-relayed). This removes the whole per-node route-advertisement
path: the cuckoo route filter and matcher (routematch.go), the miss-tagging /
retry-past-miss machinery, and the route-depth cap that only existed to bound
the filter. Text routing is now the same shape as voice job dispatch.
A slash mismatch (e.g. /hook/ for a route registered as /hook) is now rewritten
to the registered form and served directly, rather than 307-redirected. Webhook
clients often don't follow redirects, and a redirect from the final routing hop
resends the body and pays the whole routing path twice. Exact form is matched
first, so a route registered with a trailing slash is served as-is.
Use endpoint.CurrentProtocol and a named SessionCloseOK instead of bare
literals, factor the repeated 503 and header-copy blocks into helpers, and drop
a few restating test comments and a redundant assertion. No behavior change.
Replace the capsule mux / credit / attach-pool machinery with one WebTransport
(QUIC) session per worker: a control stream carries registration and status,
and each HTTP exchange rides its own QUIC stream (native mux, per-stream flow
control, half-close, and reset). Encode the HTTP method into the cross-node
route filter so candidate selection is method-aware. Keep the legacy WebSocket
control transport for backward compatibility.
Stream now implements io.ReaderFrom, so http.Request.Write's bufio.Writer
delegates the body to it and a large upload flows in MaxFrameSize frames instead
of ~4KiB ones - 16x fewer frames, marshals, scheduler cycles and websocket
messages. Large frame buffers (>= 16KiB) are drawn from a per-conn sync.Pool and
recycled by the scheduler after each write, so a big upload stops allocating a
buffer per frame; small/webhook bodies stay below the threshold and use a
right-sized make, so the average request is unaffected.
Measured on a 1MiB upload: -82% allocations, -66% latency, -17% bytes; a small
GET/POST is unchanged.
- ReadFrame reads each frame into a reused per-conn buffer via NextReader
instead of letting ReadMessage allocate one per frame; ~41% fewer bytes on a
1MiB download. Safe: proto.Unmarshal copies the payload out and readLoop is
the sole reader.
- the data wire's websocket read is bounded by its own max message size (a full
frame + framing overhead), independent of the signalling message limit, so a
peer cannot OOM the server with an oversized frame even when signalling limits
are disabled. Drops the old coupling that clamped MaxFrameSize under the
signalling limit.
- reuse the 32KiB response-copy buffer and the response-head reader from
sync.Pools; ~27% fewer bytes allocated per small request.
- reuse a per-conn marshal buffer in WriteFrame instead of allocating one per
frame; ~19% fewer bytes on a 1MiB upload. Safe because stream.Write copies
the payload and gorilla WriteMessage does not retain it.
- remove cmd/agent-endpoint-client: the Python SDK has landed, so the manual
sidecar for non-SDK workers is no longer needed.
- move pkg/agent/endpoint/client to pkg/agent/endpoint/conformance (package
conformance) so it reads as the test harness it is, not a product client
SDK; only the acceptance suite drives it.
- worker selection uses power-of-two-choices over live in-flight streams
(InflightStreams / SpareStreams) instead of weighted-random over a
reported load; the conn pick stays least-loaded. In-flight is observed
locally so it is correct for adopted wires, and the reported Load hook is
dropped from the endpoint path.
- the front tags its own routing misses (X-Livekit-Endpoint-Miss) so a relay
caller can distinguish them from a worker app's own 404/405 and retry past
them across the remaining candidates.
- a data-wire attach is bound to the connecting grant's api key, so a leaked
attach token cannot bind a conn to another project's worker.
- remove the dead scope-hook machinery (it only fed the removed bus resolve)
and rename the request identity from "scope" to apiKey.
- a path convertor anywhere in a template (non-terminal, or mixed with literal
text in a segment) now canonicalizes to a terminal glob that spans slashes,
instead of being dropped or narrowed to a single-segment str - the old
behavior silently failed to match such routes (a false negative -> misroute).
- enforce MaxRouteDepth at ParseManifest (wires up RouteDepth), bounding the
replicated filter's prefix count and the matcher's walk.
Shape-matches a request path against a set of route templates via canonical
typed-wildcard tokens and a left-to-right pruned walk: linear in path depth
(no 2^segments), type-aware (int/float/uuid/str/path), glob catch-alls, and a
PrefixIndex interface so the same walk runs over an exact map (local) or a
compact replicated backing (multi-node). Worker stays authoritative for the
exact convertor/method match.
A multi-node layer can now replicate a node's route set and match request
paths against it locally, instead of a manifest digest. Drops the digest
helper added for that (superseded).
Order-independent fnv64a of the route set so a multi-node layer can tell
whether the nodes serving a deployment agree on the manifest (safe to route
without inspecting the path) or are mid rolling-deploy (fall back to a
path-aware resolve).
A wire whose worker is registered but under a different epoch now returns
ErrWrongEpoch (not ErrAttachRejected); pool-full/closed stay ErrAttachRejected.
The attach adopter runs on unknown-worker OR wrong-epoch, so a satellite left
behind by a reconnected worker is refreshed by re-validating against the
registration holder instead of shadowing the new epoch until it is swept.
Behind a load balancer a data wire may reach a node other than the one
holding the worker's registration; the adopter lets that layer (cloud)
install a local satellite registration before the attach is retried locally.
Serve worker-declared FastAPI routes at /agents/{deployment}/{path} without
any worker-side listener: workers dial a fixed pool of wires speaking
AgentHttp.Frame, the server opens multiplexed streams carrying one opaque
HTTP/1.1 exchange each (two-level credit flow control, prioritized write
scheduler, attach epoch fencing). The front does starlette-exact manifest
matching with per-endpoint public access, typed 401/404/405, a retry table,
and SSE/WebSocket passthrough; a pluggable fallback hook lets multi-node
deployments resolve misses elsewhere. Includes a conformance client, a
manual sidecar, and the acceptance suite.
The auth middleware now reads access_token from the query string only:
FormValue consumed the bodies of proxied url-encoded POSTs.
- Renovate
config:recommended (config:base is deprecated) and matchPackageNames globs instead of the deprecated matchPackagePrefixes.
Vulnerability alerts get a fast path: 2-day quarantine, no concurrency/hourly/schedule limits.
Go modules are no longer grouped into one "go deps" PR — each gets its own, so a bad bump can be reverted alone. The pion modules stay grouped as a documented exception: they're co-released and interdependent, so individual PRs wouldn't build.
First-party github.com/livekit/** skips the 2-week quarantine.
go.mod's go directive is no longer an update target — the build toolchain is pinned in the Dockerfile instead.
Dockerfile deps get pinDigests; the golang image is ungrouped with separateMinorPatch so a patch and a minor bump are each separately approvable.
Custom manager to bump the builder image's -alpineA.B suffix together with its digest, which the stock docker manager holds fixed.
- Pinning
Both Dockerfiles pin golang and alpine by digest alongside the readable tag.
GOTOOLCHAIN=local so a go.mod bump fails loudly instead of silently downloading a different toolchain.
apk upgrade in the runtime stage — a digest pin plus the 2-week quarantine would otherwise ship base-package CVEs Alpine has already fixed. This relies on a cold layer cache, which holds today because the release workflow configures no buildx cache; there's a comment saying so.
Workflows resolve the Go version from the Dockerfile via .github/scripts/go-version.sh, so tests, releases and images share one toolchain.
- Tools
All four code generators now come from the module graph, and tools/tools.go (the pre-Go-1.24 blank-import idiom) is replaced by go.mod tool directives:
tool how why
goimports go tool lives in x/tools — its own module is the one being selected
gotestfmt go tool zero dependencies, nothing to skew
wire go run pins x/tools v0.24.1; building it in our graph changes its output
counterfeiter go run unchanged, matches its //go:generate directives
The wire distinction is load-bearing. Building wire inside our module raises it from the x/tools v0.24.1 it pins to our v0.48.0, and that module version difference changes what it generates: it falls back to v/v2/v3 instead of deriving real identifiers from the type. wire_gen.go is regenerated here to match the in-module build — a cosmetic rename of 9 lines, with no other change to the generated code.
golangci-lint deliberately keeps its action rather than becoming a tool: it pins its own x/tools (v0.44.0 vs our v0.48.0) for the analyzers it bundles, adding it to go.mod would double our go.mod/go.sum (158→338 / 441→889 lines), and the action supplies caching, only-new-issues and PR annotations that invoking a binary can't. Its version stays manual by request.
* Redact stream keys in UpdateStream API log fields
UpdateStream add/remove urls were logged raw into the API request log,
including rtmp stream keys and mux/twitch shorthand keys. Redact them
with utils.RedactStreamKey before appending log fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump protocol for query-value redaction fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Flush the downtrack sequencer on stream restart (Resync, ReceiverRestart,
codec change) so NACK retransmissions can't use metadata that no longer
matches the resynced bucket. Add a defensive bounds guard on the RTX and
forward payload slicing.
Cap the PacketHistory and FrameIntegrityChecker catch-up loops to the ring
size so a large sequence/frame-number jump can't drive a big per-packet
iteration count.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-NACK-packet goroutine spawn in DownTrack.handleRTCP with a
single long-lived worker that coalesces pending NACKs into one retransmit
pass. Pending sequence numbers are capped so a high NACK arrival rate cannot
grow goroutine count or memory unboundedly.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bound the size of HTTP request bodies on the main API listener so large
messages cannot exhaust memory. Configurable via limit.max_api_request_body_size
(defaults to 10 MiB, 0 disables).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Limit number of pending tracks per participant.
Prevents just a signalling connection adding tracks without actually
publishing them growing a large number.
* add to supervisor only if pending track is accepted
With https://github.com/livekit/livekit/pull/4706, there was a case of
some downstream component taking a long time while lock was held. While
the underlying cause of holding a lock while doing callback was removed
in that PR, to catch such cases, some publish side metric anomaly would
be useful to monitor and alert on.
Adding a publish time record for pending tracks on participant close.
That would inflate the publish time for participants not being able to
publish and can be alerted on as it will spike up the value at node
level and at cluster level if multiple nodes have the issue.
* Close web socket connections in all paths.
There was a leak of WebSocket pingWorker if the initial response write
errored as it did not close the WebSocket connection.
* graceful close
* Add configurable read-message size limit on signalling WebSockets
Set a read limit on both the client-facing (/rtc) and agent worker
WebSocket connections so an oversized frame is rejected by the transport
before being buffered. The limits are operator-tunable via
signal_message_size_limit and agent_signal_message_size_limit, both
defaulting to 2 MiB (0 disables).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add tests for signalling WebSocket read-message size limit
Cover the configurable signal_message_size_limit added in the prior
commit:
- config: assert both limits default to 2 MiB and that a YAML override
(including 0 to disable) is parsed correctly.
- full-path integration: a real client connects to /rtc on a single-node
server and an oversized frame is rejected by the transport with a 1009
close; a 0 limit leaves the connection unbounded and signalling
proceeds.
Adds setupSingleNodeTestWithConfig so a single-node server can be started
with config overrides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound decompressed size of signalling WebSocket messages
conn.SetReadLimit only accounts for the compressed bytes read off the
wire, and the client-facing /rtc upgrader negotiates permessage-deflate,
so a small compressed frame could still expand into a much larger buffer
once inflated. Enforce the same limit on the decompressed message by
reading through NextReader + io.LimitReader in WSSignalConnection instead
of the unbounded ReadMessage.
The transport-level SetReadLimit is kept as the cheap wire-level guard;
the new check is the decompressed-size backstop.
Adds NextReader to the WebsocketClient interface (regenerated fake) and a
unit test plus permessage-deflate integration tests covering the
amplification case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover a couple of more cases on data track runt packet handling.
* Guard data track header parser against extensions-size integer wraparound.
Widen the extensions-size arithmetic to int so a 0xFFFF wire value no
longer wraps in uint16, and reject any packet whose computed hdrSize
exceeds the buffer before slicing the payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix publish track count on migration in.
https://github.com/livekit/livekit/pull/4707 addressed the case of
publish tracks overcounting due to synthesised track publish on migrate
in. But, it introduced an issue where published tracks count could go
negative because unpublish subtracted the counter irrespective of the
track actually migrated in or not.
Fix it by keeping track of local publish.
Also, the older code was skipping publisher track count increase if the
synthesised publish was handled first. Address it by checking if the
track is actually new (i. e. fresh local publish) when the track was
already created in the migrate in path.
* fix pub time for tracks published after migration
* test
* prevent multiple track egresses
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
Théo Monnom