On a gap the RTP munger gives the missing sequence numbers outgoing slots.
If a late packet then arrives and is dropped, nothing is ever sent in its
slot, and a subscriber that finds frame references by sequence number
stalls until a key frame. The forwarder now flags such drops, and the
downtrack sends a one byte padding only packet in the slot.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP8 temporal layers by frame when the dependency descriptor is sent
With the dependency descriptor, libwebrtc sends a one byte VP8 payload
descriptor without a picture ID, so the VP8 munger's picture ID based
temporal drop logic stops filtering after the first gap. A new VP8DD codec
munger decides once per frame from the descriptor's frame number and
temporal ID and applies that decision to every packet of the frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: use the VP8 munger when picture IDs come with the dependency descriptor
VP8DD needs the one byte VP8 payload descriptor that libwebrtc sends with
the dependency descriptor. If a publisher keeps picture IDs next to the
descriptor, they need rewriting, so the forwarder switches to the VP8
munger and keeps switching temporal layers with the descriptor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GetRecentlyDisconnectedParticipants walked the participant update cache
with Get, which moves each entry to the front of the LRU. Walking from
newest to oldest left the cache in reverse order, so after a reconnect
the next updates evicted the most recently updated participants first.
In a room where the subscriber has tracked 128 or more participants, a
participant that had just disconnected was then dropped from the cache
and a later reconnect did not report it as disconnected, and the
out-of-order version check was lost for recently updated participants.
Use Peek, which reads an entry without changing its recency.
libwebrtc strips the VP8 payload descriptor when it sends the dependency
descriptor, so the layer sync bit that #4947 waits for is never set and a
subscriber that lowers its frame rate stays at the base layer until a key
frame. A new temporal layer selector reads only the dependency descriptor:
it switches up at a switch point whose referenced frames were all
forwarded. The forwarder uses it for VP8 when the extension is negotiated
and falls back to the Y-bit selector when the descriptor is not sent.
Fixes#4973.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Firefox 155 enabled AV1 for WebRTC by default, but Firefox 155/156
decode only the base spatial layer of an AV1 SVC stream. SVC is not
visible in SDP, so such a subscriber that negotiates AV1 is forwarded
the publisher's AV1 SVC stream instead of its backup codec, rendering
black whenever a higher layer is selected. Firefox 157 decodes the
higher layers correctly (client-sdk-js#2116), so it keeps AV1.
Removing AV1 from the subscriber's enabled codecs keeps it out of the
SFU's offers and answers, so the down track binds to the backup codec.
Firefox never publishes AV1 either, so it is disabled in both
directions, the same as Safari.
The Firefox for Android H.264 rule is switched to merge so it no longer
drops the AV1 rule for that browser.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* sfu/buffer: parse the dependency descriptor without allocating
Every VP9 and AV1 SVC packet allocated six objects in the dependency
descriptor parse: the descriptor, a clone of the frame template with
its three slices, a resolution copy, and a frame entity that took the
address of a parameter.
The pooled ExtDependencyDescriptor now owns the storage for the
descriptor, the template and the resolution, with fixed arrays for the
template's slices. The reader copies the matching template into that
storage.
Delta frame parse: 6 allocs, 176 B, 174 ns -> 0 allocs, 0 B, 122 ns.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/dd: never copy a parsed frame into a structure template
A caller can hand Unmarshal a descriptor whose FrameDependencies points
at one of the structure's templates. Copying the packet's custom fields
into it would change the template for every later packet, so the reader
allocates a fresh template in that case.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A large frame number jump made addEntity walk every skipped frame,
twice. The ring holds only numEntries frames, so older ones get
overwritten anyway. Both loops now start at most numEntries behind
the new frame, like PacketHistory and FrameIntegrityChecker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu/buffer: stop allocating per ingested packet
Buffer.Write allocated six objects for every RTP packet. They were
the parsed rtp.Packet, a copy of it in patchExtPacket, and the flow
state, whose address two rare log calls took. The rest were an
AbsCaptureTime, present on about one packet per second, and the
extension slice that pion grows with append.
Unmarshal into a pooled scratch packet. Give ExtPacket its own backing
rtp.Packet and an [8]rtp.Extension array, and copy the header into
them. Patch the payload in place. If the abs-capture-time extension is
present, parse it. Log a copy of the flow state.
Per packet, Write plus ReadExtended. Opus: 6 allocs, 448 B -> 0.
VP8: 6 allocs, 416 B -> 1 alloc, 32 B. Opus 240 -> 142 ns, VP8 269 -> 204 ns.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/buffer: point extension payloads at the bucket copy on read
Queued ExtPackets kept extension payload slices into the write input,
which the transport reuses for the next packet. ReadExtended now parses
the header again from the bucket copy, so extensions and CSRCs point
into RawPacket like the payload does.
The test writes every packet through one input buffer and checks the
audio level of each packet read back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/buffer: test that a dropped padding packet keeps the shifted sequence number on read
The bucket stores the shifted number, so re-parsing the header on read
returns it. This test guards that.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The static client config disabled H.264 publish for Firefox on Linux or
Android. The Android workaround came from #2183, but the linux clause
only ever hit desktop Linux Firefox, which publishes H.264 fine through
OpenH264. Narrow the rule to Android only.
Fixes#4929
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* bump protocol to get unlock on no key fix
* update warp pointer
* update protocol and psrpc deps
* update protocol dep
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A subscribe that found the reconcile queue full was dropped and waited
for the 3 s ticker. Flag the overflow, and have the worker reconcile
all subscriptions after its next queued item.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* agent: HTTP endpoints data plane
Serve worker-declared FastAPI routes at /agents/{deployment}/{path} without
any worker-side listener: workers dial a fixed pool of wires speaking
AgentHttp.Frame, the server opens multiplexed streams carrying one opaque
HTTP/1.1 exchange each (two-level credit flow control, prioritized write
scheduler, attach epoch fencing). The front does starlette-exact manifest
matching with per-endpoint public access, typed 401/404/405, a retry table,
and SSE/WebSocket passthrough; a pluggable fallback hook lets multi-node
deployments resolve misses elsewhere. Includes a conformance client, a
manual sidecar, and the acceptance suite.
The auth middleware now reads access_token from the query string only:
FormValue consumed the bodies of proxied url-encoded POSTs.
* agent: attach adopter hook for wires landing on foreign nodes
Behind a load balancer a data wire may reach a node other than the one
holding the worker's registration; the adopter lets that layer (cloud)
install a local satellite registration before the attach is retried locally.
* agent: distinguish wrong-epoch attach so a multi-node adopter can refresh
A wire whose worker is registered but under a different epoch now returns
ErrWrongEpoch (not ErrAttachRejected); pool-full/closed stay ErrAttachRejected.
The attach adopter runs on unknown-worker OR wrong-epoch, so a satellite left
behind by a reconnected worker is refreshed by re-validating against the
registration holder instead of shadowing the new epoch until it is swept.
* agent/endpoint: Manifest.Version() digest for skew detection
Order-independent fnv64a of the route set so a multi-node layer can tell
whether the nodes serving a deployment agree on the manifest (safe to route
without inspecting the path) or are mid rolling-deploy (fall back to a
path-aware resolve).
* agent/endpoint: carry raw endpoints on the registration
A multi-node layer can now replicate a node's route set and match request
paths against it locally, instead of a manifest digest. Drops the digest
helper added for that (superseded).
* agent/endpoint: canonical route matcher (typed tokens + pruned walk)
Shape-matches a request path against a set of route templates via canonical
typed-wildcard tokens and a left-to-right pruned walk: linear in path depth
(no 2^segments), type-aware (int/float/uuid/str/path), glob catch-alls, and a
PrefixIndex interface so the same walk runs over an exact map (local) or a
compact replicated backing (multi-node). Worker stays authoritative for the
exact convertor/method match.
* agent/endpoint: fix glob matching and enforce route depth
- a path convertor anywhere in a template (non-terminal, or mixed with literal
text in a segment) now canonicalizes to a terminal glob that spans slashes,
instead of being dropped or narrowed to a single-segment str - the old
behavior silently failed to match such routes (a false negative -> misroute).
- enforce MaxRouteDepth at ParseManifest (wires up RouteDepth), bounding the
replicated filter's prefix count and the matcher's walk.
* agent endpoints: P2C worker balancing, routing-miss sentinel, api-key-scoped attach
- worker selection uses power-of-two-choices over live in-flight streams
(InflightStreams / SpareStreams) instead of weighted-random over a
reported load; the conn pick stays least-loaded. In-flight is observed
locally so it is correct for adopted wires, and the reported Load hook is
dropped from the endpoint path.
- the front tags its own routing misses (X-Livekit-Endpoint-Miss) so a relay
caller can distinguish them from a worker app's own 404/405 and retry past
them across the remaining candidates.
- a data-wire attach is bound to the connecting grant's api key, so a leaked
attach token cannot bind a conn to another project's worker.
- remove the dead scope-hook machinery (it only fed the removed bus resolve)
and rename the request identity from "scope" to apiKey.
* agent endpoints: drop the sidecar cmd, make the conformance client test infra
- remove cmd/agent-endpoint-client: the Python SDK has landed, so the manual
sidecar for non-SDK workers is no longer needed.
- move pkg/agent/endpoint/client to pkg/agent/endpoint/conformance (package
conformance) so it reads as the test harness it is, not a product client
SDK; only the acceptance suite drives it.
* agent endpoints: pool the per-request bridge buffers
- reuse the 32KiB response-copy buffer and the response-head reader from
sync.Pools; ~27% fewer bytes allocated per small request.
- reuse a per-conn marshal buffer in WriteFrame instead of allocating one per
frame; ~19% fewer bytes on a 1MiB upload. Safe because stream.Write copies
the payload and gorilla WriteMessage does not retain it.
* agent endpoints: reuse the read buffer and bound the data wire
- ReadFrame reads each frame into a reused per-conn buffer via NextReader
instead of letting ReadMessage allocate one per frame; ~41% fewer bytes on a
1MiB download. Safe: proto.Unmarshal copies the payload out and readLoop is
the sole reader.
- the data wire's websocket read is bounded by its own max message size (a full
frame + framing overhead), independent of the signalling message limit, so a
peer cannot OOM the server with an oversized frame even when signalling limits
are disabled. Drops the old coupling that clamped MaxFrameSize under the
signalling limit.
* agent endpoints: frame large uploads at MaxFrameSize and pool the buffers
Stream now implements io.ReaderFrom, so http.Request.Write's bufio.Writer
delegates the body to it and a large upload flows in MaxFrameSize frames instead
of ~4KiB ones - 16x fewer frames, marshals, scheduler cycles and websocket
messages. Large frame buffers (>= 16KiB) are drawn from a per-conn sync.Pool and
recycled by the scheduler after each write, so a big upload stops allocating a
buffer per frame; small/webhook bodies stay below the threshold and use a
right-sized make, so the average request is unaffected.
Measured on a 1MiB upload: -82% allocations, -66% latency, -17% bytes; a small
GET/POST is unchanged.
* agent endpoints: drop client WebSocket route support, repin protocol
* agent endpoints: repin protocol for the enum renumber
* agent endpoints: WebTransport data plane
Replace the capsule mux / credit / attach-pool machinery with one WebTransport
(QUIC) session per worker: a control stream carries registration and status,
and each HTTP exchange rides its own QUIC stream (native mux, per-stream flow
control, half-close, and reset). Encode the HTTP method into the cross-node
route filter so candidate selection is method-aware. Keep the legacy WebSocket
control transport for backward compatibility.
* agent endpoints: tidy front helpers and constants
Use endpoint.CurrentProtocol and a named SessionCloseOK instead of bare
literals, factor the repeated 503 and header-copy blocks into helpers, and drop
a few restating test comments and a redundant assertion. No behavior change.
* agent endpoints: assert deployment scoping in the registry test
* agent endpoints: drop file-narration header comments
* agent endpoints: normalize trailing slashes during routing instead of 307
A slash mismatch (e.g. /hook/ for a route registered as /hook) is now rewritten
to the registered form and served directly, rather than 307-redirected. Webhook
clients often don't follow redirects, and a redirect from the final routing hop
resends the body and pays the whole routing path twice. Exact form is matched
first, so a route registered with a trailing slash is served as-is.
* agent endpoints: route at deployment granularity, drop the route filter
Route presence like a reverse proxy keyed on (project, agent_name, deployment):
any of a deployment's workers is a candidate and the worker's own router returns
the real status (a 404 for a path it doesn't serve during a rolling deploy is
returned, not re-relayed). This removes the whole per-node route-advertisement
path: the cuckoo route filter and matcher (routematch.go), the miss-tagging /
retry-past-miss machinery, and the route-depth cap that only existed to bound
the filter. Text routing is now the same shape as voice job dispatch.
* agent endpoints: audit cleanups (drop dead Registration.Endpoints, test the fallback flow)
Remove the write-only Registration.Endpoints field (matching goes through
Manifest; the multi-node layer no longer replicates the raw route set). Add
Front-level fallback tests (fires with the resolved identity, declined -> 404
with a local worker or 503 without), and fix two stale comments.
* agent endpoints: byte-transparent HTTP/1.1 exchange
Replace the protobuf frame layer on exchange streams with a length-prefixed
StreamPreamble followed by opaque HTTP/1.1 bytes. QUIC already provides the
multiplexing and per-stream flow control, so the frame layer only re-encoded a
message both peers can already parse, and every SDK had to re-materialize HTTP
from it.
The front serializes a canonical request head from its own parsed
*http.Request and never forwards the client's bytes, which is what keeps
request smuggling out of the worker. Responses are read with
http.ReadResponse, so informational heads need no httptrace hook and the
conformance worker loses its HTTP reconstruction entirely.
Completion splits by outcome: a body ends by its own framing, a failure after
bytes have flowed travels in x-lk-completion / x-lk-error trailers, and a
failure before any byte resets the stream with an HttpStreamResetCode. REFUSED
stays distinct as the retry-safety signal, and the retry/idempotence rules are
unchanged.
The x-lk- prefix is reserved for this signalling: it is stripped from
client-supplied request headers so a caller cannot forge an outcome, and from
responses so it never reaches the end client. Exchange-stream targets are
split from the escaped path so a percent-encoded '?' or '/' cannot change
which resource the worker routes to.
Per-request state moves to attempt.go so the Front and attempt lifecycles stop
interleaving; the remaining files are grouped type-first with free helpers
last. Those moves are ordering only.
* go.mod: pin protocol with the agent endpoints preamble
Picks up livekit/protocol 134fb913, which replaces the HTTP framing messages
with StreamPreamble and renumbers HttpStreamResetCode. The previous pin
predates AgentHttp entirely, so the endpoint packages only built through the
local go.work.
* agent endpoints: require a scoped grant for non-public routes
The front gated non-public routes on a bool that was true for any valid token
in the project, so a roomJoin token minted for an end user reached every
non-public endpoint the project's agents expose.
Replace the resolver's (apiKey, authenticated) pair with an Access value and
give it the agent and deployment from the URL, so the scope rules stay with
the grant in protocol/auth and this package consumes only the verdict.
Credentialed now separates a caller with no credential from one whose
credential lacks the grant: the first is challenged with 401, the second gets
403 and no WWW-Authenticate, since a retry cannot succeed.
A denied request still reaches the fallback. Another node's worker may declare
the same path public, and that node is the authority.
* go.mod: pin protocol with the agent endpoint grant
* agent endpoints: accept any agent name, keep request escaping intact
The name is percent-encoded into its URL path segment, so it needs no charset.
Only "_", "." and ".." are refused, and only for workers declaring endpoints:
all three are unreserved, so no encoded form addresses them distinctly. "_"
now addresses a worker registered with an empty agent name, which the
implicit-dispatch default leaves unset.
Two escaping fixes the escaped-path split depends on:
- the trailing-slash alternate treated a decoded slash as literal, so
/known%2F normalized onto route /known and the worker was handed a path it
does not serve.
- RemoveDoubleSlashes trimmed URL.Path and left RawPath, which makes
EscapedPath() re-encode from Path and lose every escape in the request.
Request logs now carry the encoded path, since a decoded %0A forges a log line.
* agent endpoints: return the abort from bridge rather than panicking inside it
A committed response that cannot be completed is torn down with
panic(http.ErrAbortHandler): the head is on the wire, so no status can report
the failure, and this is the only teardown net/http offers. HTTP/1.1 closes
without the final chunk and HTTP/2 sends RST_STREAM, so a short body cannot
read as whole.
bridge raised it from inside the response copy loop, through a helper whose
doc had to note that it does not return. It now reports a bridgeOutcome and
ServeHTTP performs the teardown at the handler boundary, where the constraint
that nothing may recover the sentinel is visible to a reader. The outcome
replaces the (done, retryable) pair, whose four combinations only ever carried
three states.
No behavior change.
* agent endpoints: serve the front on its own middleware chain
The front was mounted on the API mux and served through the API middleware
chain, which is shaped for handlers that buffer a request and write one
response. Four consequences.
negroni.NewRecovery() recovers every panic with no http.ErrAbortHandler
exemption, and NewRecovery sets PrintStack. On an abort it wrote "PANIC: ..."
and a goroutine stack into the already-committed body, then returned normally,
so net/http terminated the chunked stream cleanly. A truncated response
reached the client as complete, with a stack trace appended. Detection was
already correct; delivery was not, and the x-lk- trailers that would otherwise
signal it are stripped before the response leaves. A Content-Length response
still failed safe, since net/http enforces the declared length itself, so the
gap was the chunked and trailer paths.
Endpoints.Disabled documents that it turns the front off, but only
registrations were refused; the mount was unconditional.
The CORS method list omits PUT, which a manifest may declare.
The API body limiter capped request bodies at MaxAPIRequestBodySize, though
the front streams a body through a pooled buffer and never holds one.
NewHTTPHandler now routes the prefix to a chain carrying AgentRecovery, a CORS
list matching the methods a manifest may declare, and the api-key auth
middleware the front resolves a caller's access from. The mount is built only
when endpoints are enabled, so the prefix otherwise falls through and 404s.
RemoveDoubleSlashes moves above the split, so routing and both chains see one
path form.
Taking the front off the mux also stops ServeMux rewriting the paths it is
handed: "//x", "/../" and interior "//" were answered with a redirect rather
than proxied, which a byte-transparent exchange cannot do.
The endpoint stack tests now build the production handler, so they run on the
chain the node serves on.
* agent endpoints: build the front and registration from params
Front is an http.Handler, so fallback, identity and singleKeyFallback were read
on every request with no synchronization while WithFallback, WithIdentity and
WithSingleKeyFallback wrote them on the live object. Each returned the same
pointer, so the chain read as though it built a value. Every call site happens
to run before the handler is mounted, so nothing races today, but nothing in
the type prevents it, and WithSingleKeyFallback took no argument and could only
be turned on. NewFront now takes FrontParams and configuration is fixed at
construction.
NewWorkerRegisterer takes its EndpointSettingsFunc directly, and
HandshakeAgentWorker takes one in place of a variadic of raw closures that
existed only to reach the setter it replaced.
Registration gains NewRegistration and RegistrationParams, moving the Draining
callback off an exported mutable field and folding SetSession into
construction. IsDraining absorbs the nil check at both call sites.
Access carried a three-state ladder as two bools, with "granted implies
credentialed" documented but unenforced. It is now an ordered AccessLevel, so
callers compare a rank rather than combining flags and the invariant holds by
construction.
Registry.Register returned an error that was always nil, with a dead branch at
each call site. Both registry maps and the per-registration session are
read-heavy, so they take RWMutex.
CopyBody returned two errors to separate a source failure from a destination
one. It returns one, wrapping a source failure in *SourceError, which is what
the caller discriminates on.
NewWebTransportServer took a callback to break the handler/server init cycle;
the caller assigns wt.H3.Handler after construction instead. StartWebTransport
reads Development off the service rather than taking a bool, and returns a nil
stop function where it starts no listener.
Smaller: slices.Sort for sort.Strings, for range for an unused counter, a nil
slice for regs[:0:0], p2c generic over its slice so the call site passes a
method expression rather than allocating a closure per request, and streamCode
deduplicated into wire.StreamCode so both peers map reset codes in one place.
Comments on the touched declarations drop remote behavior, migration narration
and contrastive framing, keeping the constraints and invariants.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: serve webtransport from the node's server set
StartWebTransport built the node's only HTTP/3 listener from inside AgentService:
it minted the TLS config, bound the UDP socket, assembled an h3 mux and returned
a stop closure LivekitServer held in agentWTStop. Every other listener is
constructed in NewLivekitServer and delegates to a service, and agents are the
first consumer of HTTP/3 rather than the last.
webtransport.go now carries the transport with no agent references:
WebTransportTLS, NewWebTransportServer, NewWebTransportHandler,
ListenWebTransport, and WithWebTransportServer/GetWebTransportServer, which put
the serving server in the request context so a route that upgrades is a
conventional http.Handler instead of a closure over the server. The listener is
built in NewLivekitServer beside the prometheus and debug servers, bound in Start
with the other listeners, and stopped once doneChan unblocks, so no stop closure
is held on LivekitServer. /agent is a route on its mux, as it is on the API mux.
UpgradeWebTransport unwraps the ResponseWriter before upgrading. webtransport-go
type-asserts it to http3.Settingser and http3.HTTPStreamer without checking and
negroni wraps the writer on every chain, so the h3 listener could not carry a
middleware chain at all. It now runs the same recovery, api-key auth and path
normalization as the TCP chain.
AgentService splits into AgentWSService and AgentWTService over the shared
AgentHandler, and the endpoint front becomes AgentEndpointService. wire builds
all four and threads one endpoint.Registry into the handler and the front.
NewAgentHandler takes the config and builds its own ServerInfo, so endpointsConfig
and singleAPIKey are set at construction and the psrpc server is registered
against a fully formed handler; it previously bound one whose embedded
*AgentHandler was still nil.
The listener's port and certificate move to a top-level webtransport config
block. The listener no longer reads agents.endpoints.disabled, which continues to
refuse endpoint registrations. ListenWebTransport binds one socket per bind
address, where the h3 listener ignored bind_addresses that every TCP listener
honoured.
NewWorkerRegisterer takes a variadic WorkerRegisterHandler, each handler reading
the request and filling its part of the response and the registration, replacing
the single EndpointSettingsFunc threaded through HandleRegister.
EndpointRegisterHandler is the endpoint declaration's turn, and
endpoint.NegotiateSettings holds the protocol negotiation the OSS and cloud
handshakes each carried a copy of.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: accept webtransport connections rather than racing Serve with Close
ListenWebTransport ran wt.Serve on a goroutine per socket and returned a stop
func calling wt.Close. Serve takes a reference on the server's refCount
WaitGroup and Close waits on it, so the pair breaks the WaitGroup's own rule
that an Add starting from zero must happen before a Wait. The race detector
models that rule as a read of wg.sema in Add against a write in Wait, and
reports it whenever Wait observes a non-zero counter: a listener stopped
before its serve goroutine has run at all, which is every test that builds a
stack and tears it down without a worker connecting. sync is compiled without
instrumentation, so the report names the two ListenWebTransport call sites
with no frame in between. webtransport-go v0.13.0 carries the same code.
The accept loop moves here. quic.ListenEarly builds the listener Serve would
have built, with the datagram and partial-delivery options the session layer
requires, and each connection goes to Server.ServeQUICConn, which touches no
part of that WaitGroup. The stop func keeps the order Close established:
Close first, so every CONNECTION_CLOSE frame is transmitted while the sockets
are still open, then the accept loops are cancelled and drained, then the
listeners and sockets close.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: sequence the endpoint tests on the registry
handleConnection writes the register response from inside HandshakeAgentWorker
and installs the worker's routes after it returns, so a worker learns it is
registered a moment before the node can route to it. The conformance worker's
WaitRegistered unblocks on that response and startWorker returned straight
into the first request, which on a loaded runner arrived while the registry
was still empty and came back 503. TestAgentEndpointsStatusMapping saw that as
503 for the first few mapped statuses, TestAgentEndpointsHOL as an unexpected
EOF, reading 1024 bytes out of a short error body.
The stack keeps the registry it builds and startNamedWorker waits for the
worker's registration to appear in it, which is the condition the requests
actually depend on. TestAgentEndpointsRetrySafety builds its unreachable
worker by hand and waits the same way. The HOL test asserts its status before
reading the body, so a 503 there reports as a status mismatch rather than a
truncated read.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: log the deployment with worker registration
Workers are keyed by deployment alongside agent name, namespace and job type.
The registered and last-deregistered lines named every part of that key except
the deployment.
* agent endpoints: add a path router that matches in declaration order
Manifest matching runs a compiled regex per route over a linear scan, once per
candidate registration, on the serving path. At the 256-route cap that is up to
256 RE2 executions per replica per request.
The router compiles a manifest into a compressed trie instead. Starlette
selects the first route declared that matches, so the trie cannot resolve
static before wildcard the way net/http's does: every node carries the lowest
route index in its subtree, the walk tracks the lowest full match found and
prunes any subtree that cannot improve on it, and inserting in declaration
order leaves edges and leaves sorted by that index with no sort pass.
Params are not segment-aligned - starlette allows /f/{name}.{ext}, a {p:path}
anywhere, and a float whose fraction backtracks - so the walk must search. An
edge is single when nothing below it can start with a byte its own convertor
could have consumed, which is a property of the target node and so is settled
at build time; a single edge takes its greedy run and descends once. Templates
whose params are segment-aligned are entirely single and never search. What
remains is bounded by a step budget, and exhausting it returns ResultOverBudget
rather than a route the cut-short search cannot vouch for.
Templates parse without regexp. Scanning the grammar by hand keeps a brace that
opens nothing well-formed as an ordinary literal, which is what starlette's
finditer does. Anchoring is `\n?$` rather than `$`: python's '$' matches before
one trailing newline where Go's does not, and [^/] accepts a newline where .
refuses one, so the two convertors diverge in opposite directions on a decoded
%0A.
The regex implementation moves into the tests as an oracle, carrying its own
parser so it references the hand-rolled scanner as well as the trie.
FuzzMatchAgainstOracle generates a table and a request and asserts the two
agree on both the winning template and the result. Matching allocates nothing,
asserted by AllocsPerRun on hit, 405, miss and on the searching path.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: route manifests through the path router
ParseManifest builds a router.Router rather than a slice of compiled templates,
Manifest aliases it, and Route keeps the template and the public flag the front
reads. Declared methods become a mask at build time, so a method no route could
declare masks to 0 and matches nothing, as an undeclared one already did.
Three behavior changes ride along.
Templates anchor as `\n?$`. A manifest declaring private /files/{p:path} before
public /files/{p} routed GET /files/x%0A to the public route here and to the
private one on the worker, because the path convertor refuses a newline where
str accepts it. The access gate reads the public flag of the route the front
picked, so it opened on a route starlette would not have selected.
slashAlternatePaths derives the escaped form from the transform it applied
rather than from the result's suffix. A request path of // trimmed to /, the
caller saw a trailing slash on that result, took the append branch, and handed
the worker a request line of /// against a decoded path of /.
The decoded path is capped at MaxPathLength, answering 414 above it. Matching
runs before anything sizes the request head, so net/http's own limit was the
only bound on what the matcher was handed.
A table too ambiguous to decide now dispatches rather than 404s: the worker's
own router still knows, and the preamble carries no route for it to be told
about. Nothing here knows whether that route is public, so the request needs a
grant. Dispatch keys on the match set, since a candidate can now join it
without resolving a route.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: log the manifest as a structured object
Router implements zapcore.ObjectMarshaler, so registration logs the table's
size and the templates whose shape forces the matcher to search under one key
rather than assembling fields at the call site. Which templates those are is
settled when the manifest is parsed, so it is worth reporting before any
request is served; what a single match cost belongs to that request instead.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: key templates by shape and export their ambiguity
Canonical renders a template as its literals and convertor kinds, so two
templates the trie cannot tell apart share a key. Param names reach the trie
only as a kind, so /x/{a} and /x/{b} accept the same paths; anything keying
routes by the declared spelling splits them apart. Literals are length-prefixed
because a brace that opens no well-formed param is literal text: /x/{:str}
matches itself and must not collide with /x/{a}.
ambiguousTemplate becomes Template.Ambiguous. It reads only the template's own
elements, so it answers for a manifest that was validated but never compiled
into a table of its own.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: merge worker manifests into one table per deployment
A deployment owns its route table. A route is one template shape and one method
together with the workers that declared it; registration takes the union of the
table and the arriving worker's manifest, deregistration retracts the worker and
drops the routes nothing is left declaring. Manifest stops being a compiled
router and becomes the ordered list a worker declared, so a fleet holds one
table rather than one per replica, and a request resolves in a single walk.
A mixed fleet is the point. Matching each worker's own table in turn already
served heterogeneous methods, but the route the front reported was the first
candidate's while the worker it dispatched to was any of them, and partial and
denied were accumulated across workers whose manifests disagreed. Both become
properties of one table, so the status a request gets and the template the
worker is told are decided together.
Merge order is the lowest position a route holds in any declaring manifest, tied
by shape then method. A fleet declaring one manifest reproduces that manifest's
order, and the result reads off the current set of manifests, so nodes holding
the same fleet agree. This does not preserve any one worker's internal order: a
worker can be handed a request the merged table resolved to a public route while
its own table routes that path to a private one. Merging the declaration chains
in order would hold the property and is not done here;
TestRouteTableMergeOrderInversion carries the case.
Routes are keyed by shape, so /x/{a} and /x/{b} are one route and a renamed path
param does not halve the fleet serving it. Each worker keeps the spelling it
declared, and the preamble carries that, since it is re-serialized per attempt.
The tree holds routes by pointer and is rebuilt only when the route set or its
order moves, so a worker joining or leaving an existing route swaps nothing. A
supersede runs the removal and the addition as one transaction: an unchanged
manifest would otherwise empty every route, drop it, and build a new one under
the same key, leaving the published tree naming routes nothing can reach and no
change to the key set to show for it. A reconnect can also land on a different
agent name or deployment, so the retiring epoch is retracted from its own table.
Ambiguity is now a property of the merged tree, and one worker's template shape
can carry the whole deployment's matcher over its step budget, which leaves no
route decided and so no public flag to clear an ungranted request.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: scope the registry, let the embedder own tenancy
The registry was doing three jobs at once: fencing worker epochs by id,
listing a deployment's candidates, and holding its merged route table. The
last two are per-deployment state, so it keyed them on
(api key, agent name, deployment) and grew a tenancy concept that only an
embedder can actually define. Cloud has to lie to it, passing a project id
in a field named APIKey.
Split them. Scope is one deployment's serving state and stores no identity
at all; whoever embeds the package keys a map of scopes however its own
tenancy works, and hands the front a resolved one. Registry keeps only the
worker-id fence, which is genuinely node-wide: worker ids are server-issued,
so an epoch is superseded wherever it was scoped.
The front loses its registry, its SingleKeyFallback and FallbackRequest: the
resolver now returns the scope and a fallback already curried on the
deployment, plus an ok that carries the 401-vs-503 split the empty api key
used to encode. routeTable drops its key and takes the scope's logger, so
identity is curried in rather than stored.
pkg/service takes ownership of the "api key is the tenant" rule, which is
true there and nowhere else, and of releasing a scope once nothing holds it.
Behavior is unchanged, including serving public routes to an unauthenticated
caller when one configured key or one attached tenant makes the key
unambiguous.
* agent endpoints: give the http3 listener a server type
ListenWebTransport returned a stop closure, so every caller had to park it
next to the server it belonged to. In cloud that meant a second field on the
node base, written at start and read at stop, for one listener.
WebTransportServer embeds webtransport.Server and owns the sockets and
accept loops, so Listen and Shutdown are methods and the shape matches
http.Server. Shutdown takes a context: it stops accepting, sends GOAWAY, and
closes whatever has not drained by the deadline, where the closure closed
everything at once with no deadline of its own. It is safe on a server that
never listened, which is what lets a holder key teardown off the field alone.
The embedded Close still releases sessions without releasing the sockets;
that is what the type comment warns about.
* agent endpoints: fold the ident-start test into a helper
staticcheck QF1001 flagged the negated disjunction in scanIdent, which was
failing the lint gate and so keeping the test job from running at all. The
positive helper reads better than the De Morgan form and lets the continue
condition drop its own '_' case.
* go.mod: bump protocol to the merged branch head
* rtc tests: give the logger its own logging config
init() handed the logger &config.DefaultConfig.Logging, so the leveler's
mutex lived inside the global that NewConfig yaml-marshals. Marshalling
reflectively copies the whole struct, mutex included, which raced the
component-level lookups pion makes from its own goroutines and failed
TestPreferMediaCodecForPublisher under -race.
The values are the same; only the storage is now test-local. The integration
helper aliased the same global and gets the same treatment.
* rtc tests: drop the logging config comment
* auth: name the call the query-string read avoids
The token comes from RawQuery so the body reaches a proxied agent endpoint
intact. That only holds as long as nobody reaches for FormValue, which is
the obvious way to read a param and the one that drains the body, so the
comment names it.
* agent endpoints: tunnel WebSocket upgrades to workers
An exchange stream already carries opaque HTTP/1.1, so a WebSocket is an
ordinary GET route: the front forwards an HTTP/1.1 Upgrade: websocket
handshake, the worker's HTTP stack decides, and on a 101 the front hijacks
the client connection and splices it onto the stream. No protocol change.
- Only HTTP/1.x GET with Connection: upgrade and Upgrade: websocket (and no
body) is an upgrade; every other Upgrade token is still stripped. A 101 is
accepted only for such a request and only with Upgrade: websocket;
anything else stays a protocol error.
- A non-101 answer to an upgrade is an ordinary response; the stream then
ends with a FIN rather than a reset.
- Retries are unchanged until the 101; after it the attempt is committed.
- Each worker takes at most MaxTunnelsPerWorker tunnels (default half its
session's MaxStreams) so WebSockets cannot starve its HTTP routes; an
upgrade no worker can take falls to the fallback, then 503.
- Drain: a tunnel outlives its worker starting to drain by
TunnelDrainTimeout (default 30s), and Front.Drain closes tunnels once its
context is done. Both close with a 1001 frame when the client-bound side
is between frames, abruptly otherwise. The server drains tunnels while
participants leave, since http.Server.Shutdown cannot see them.
- access_token is dropped from the target and a LiveKit bearer token from
Authorization for every request: the worker gets identity from the
preamble, and the token must not reach application code.
- Full duplex: enable it before the first flush, so an HTTP/1 client can
keep streaming its body while the response streams back.
Config: agents.endpoints.max_tunnels, agents.endpoints.tunnel_drain_timeout.
* agent endpoints: keep one value per CORS header
The agent chain's CORS middleware writes Access-Control-* and Vary before
the front runs. A worker's head was added on top, duplicating
Access-Control-Allow-Origin, and the clear after a 1xx head dropped the
middleware's headers from the final response.
- A worker's Access-Control-* and Vary values replace the middleware's;
other headers are still added (MergeWorkerHeaders).
- Headers set before the front ran are restored after each 1xx head.
- Preflights stay with the middleware; a plain OPTIONS still reaches a
declared route. NewAgentEndpointCORS holds the agent chain's CORS
middleware so other chains can share it.
* agent endpoints: withhold endpoint settings from sessionless workers
A worker registering over WebSocket has no data plane, yet the handshake
negotiated endpoint settings, telling it its routes were live while every
request 503'd. Negotiate only when a WebTransport session exists, per the
endpoint_settings contract, and keep the worker registered for jobs.
* deps: pin protocol to the agent endpoints branch head
* deps: pin merged protocol
---------
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep one subscription when first subscribes race
SubscribeToTrack and SubscribeToDataTrack looked up the subscription,
then created one and stored it under a separate lock. Two concurrent
first subscribes, or a subscribe and a settings update, could each
store their own entry. The replaced entry could still add the down
track, which nothing owned after that, and the settings could land on
the replaced entry.
Find or create the subscription and set it as desired under one lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: keep early settings and a racing data track resubscribe
A settings or options update before the first subscribe creates an
entry that is not desired. Cleanup removed it on the next pass, so the
subscribe created a new entry without the settings. Keep such an entry
for notFoundTimeout.
After a data track unsubscribe, the entry was deleted even if a
subscribe set it as desired during the removal. Delete it only if it
is still not desired, else reconcile it again.
Cleanup deletes only the entry it checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
RedReceiver kept pointers to the last forwarded packets for its
redundant blocks, but their payload is in the read buffer that
forwardRTP reuses for every packet, so the blocks carried the current
packet's bytes. Copy the history packets into slots RedReceiver owns.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: use the most common recent packet rate in connection quality
The loop that finds the mode packet rate compared a bin's count with the
index of the bin picked so far, so it did not find the most common rate.
Take the bin with the highest count, skipping rates below 10 pps so that
idle, DTX and static content rates do not become the reference. Halve the
counts at each mode update so that the mode follows a lasting change of
rate, and keep the previous mode through a long stretch of low rates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: start the mode timer from the start time
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GenerateCLIFlags created an Int64Flag for every time.Duration config
field. Setting one through its LIVEKIT_* env var or --flag with a
duration string such as "30s" (the format the YAML config and the
generated help text use) made the server exit at startup with a parse
error. A value without a unit was accepted and read as nanoseconds.
Use a DurationFlag for these fields, so they take the same values as in
the YAML config.
* fix: use the fast debounce for the first negotiation
A new transport set lastNegotiate to its creation time. A non-forced
Negotiate soon after creation then waited for the 150 ms debounce
instead of 10 ms, although nothing was negotiated yet. Start from the
zero time so that the first offer uses the fast debounce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the first negotiation condition without a deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check that the first offer goes out before a debounced one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: do not read invalid delta
In TWCC feedback, only the small delta and large delta symbols carry a
receive delta. The send side BWE loop used a delta for every received
symbol, so the reserved symbol read a delta that does not exist. Skip
that symbol and do not read past the last delta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: assert loss count and delta use for the reserved symbol
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test: give mock participants a connection quality
A real participant never returns a nil connection quality. The mock
did, which can crash code that does not expect nil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: close test rooms when each test ends
Most room tests left their rooms open, so each room's workers kept
running for the rest of the package.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
selectAlternativeVideoCodec walked the enabled codecs and returned the
first one with a video/ prefix. video/rtx and video/flexfec-03 both have
that prefix, so when the requested codec was not enabled the track could
fall back to RTX. RTX is a retransmission stream, not a real codec, so
the track got registered as video/rtx and subscribers saw a blank tile.
Skip RTX and FEC entries so the fallback is always a valid video codec.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* service: start room composite egress configured in a room preset
StandardRoomAllocator.CreateRoom merged a room preset into a copy of the
request and discarded it. RoomInternal has no field for room composite
egress, and getOrCreateRoom started it from the caller's original request,
so a room composite egress from a preset was never started.
RoomAllocator.CreateRoom now also returns the request with the preset
applied, and getOrCreateRoom starts the egress from it.
* service: return ErrEgressNotConnected instead of calling a nil egress launcher
getOrCreateRoom started a room composite egress without checking that an
egress launcher exists. RoomService.CreateRoom rejects egress on the request
when there is none, but not egress from a room preset or a token's roomConfig.
send_side_bwe_pacer accepted any string and silently ran no-queue. leaky-bucket
is a defined PacerBehavior with a full LeakyBucket implementation, but the
selection switch in NewPCTransport only handled pass-through and no-queue, so
both leaky-bucket and outright typos fell through to the default with no
diagnostic.
leaky-bucket is deliberately left unwired here. LeakyBucket paces to a bitrate
supplied via Pacer.SetBitrate, and no production code calls SetBitrate or
SetInterval -- Base no-ops both and the only caller in the tree is the
test-only capturingPacer. Wiring it as-is would pace at a static bitrate and
ignore the bandwidth estimate, which is worse than the fallback.
Instead, make the existing behaviour visible: CongestionControlConfig.Validate
rejects unrecognised values at startup, the fallback logs a warning naming the
requested pacer, and config-sample.yaml documents send_side_bwe_pacer and
use_send_side_bwe for the first time.
Refs #4724
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* sfu: report forwarding latency as p90 instead of the mean
The forwarding-latency metric was the mean transit over all forwarded
packets. A mean is dominated by a few slow outliers, so a handful of
packets stalled on the forward path (e.g. goroutine scheduling latency)
inflated the whole node's reported latency even when nearly every packet
was forwarded promptly.
Report p90 instead: p90 rising means roughly a tenth of forwarded packets
are slow, a broad signal of systemic forwarding load rather than a sparse
tail. To read a percentile over the report window, the mergeable
per-interval summary now keeps a small power-of-two-bucket histogram of
transit instead of running moments (sum, sum-of-squares).
Drop the jitter (transit std dev) gauge: nothing consumed it, and any
spread is derivable from the forward-latency histogram. The protobuf
ForwardJitter field is left in place, now unset, to deprecate separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: clamp forwarding percentile to the observed [min, max]
Bucket interpolation assumes a uniform fill, so a single 20ms packet (or
uniform traffic) could report a p90 above every observed sample. Clamp the
interpolated value to the summary's already-tracked min/max, so a quantile
never falls outside the data. Exact for single samples and repeated
identical latencies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: fix stale p50 comment in the percentile test
The reported metric is p90; the test comment still said p50 replaced the
mean. Reword it to reflect that a percentile, not the mean, is reported.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: quarter-octave forwarding-latency buckets for threshold resolution
Octave buckets are too coarse near the overload thresholds: 300us falls in
[256,512), so a p90 clustered at ~265us and one at ~500us interpolate to the
same value and would trip (or not) identically. Split each octave into four
linear sub-buckets so the two land in different buckets, on the correct side
of the threshold. Min/max clamping alone does not fix this once a node has a
high tail, since its max no longer bounds the interpolation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: interpolate percentiles within each bucket's observed range
Replace the quarter-octave split with plain octave buckets that also carry
the observed [min, max] of their samples, and interpolate a percentile
within that range instead of the bucket's nominal edges. This is exact when
a bucket's samples cluster, so a p90 near an overload threshold that falls
mid-bucket lands on the correct side of it regardless of bucket width -- no
threshold-aware boundaries needed. It subsumes the min/max clamp, since an
estimate can no longer leave the observed samples.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: cover the 500us cluster in the threshold test
Assert milos's full review example exactly: 265us and 500us clusters that
octave-nominal interpolation both read as ~341us now read 265us and 500us.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: make forwardSummary.addSample a pointer receiver
The summary grew from a small moments struct into a per-bucket histogram
(~700 bytes), so the value-receiver addSample copied the whole summary on
every drained sample in the flush loop. Mutate in place instead: ~28ns ->
~2ns per sample in the background fold, no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: eighth-octave buckets for accuracy near thresholds
Split each octave into 8 linear sub-buckets (was plain octaves). With the
per-bucket min/max interpolation this reports a tight p90 cluster exactly
even when it sits mid-octave: 850x265us + 150x410us now reads 410us (above a
400us threshold) instead of 395.5us, and a lognormal p90 lands within ~0.2us
of exact. Per-sample add cost is unchanged (~2ns); cost is ~5KB per summary
and a larger but per-report merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Reverts #4816. Since livekit/cloud#4747 a room re-key closes every
participant's stats worker under the old room id and opens a new one
under the new id (LeaveSession/JoinSession, #4917), so RoomIDChanged
and reKeyRoom have nothing left to move, and with them go the sealed
stats batches, SetRoom, ForceClose and the reference hand-over that
only the re-key collision case needed. The worker map goes back to a
flat (room, participant) key.
The guard fixes that landed after #4816 (#4817, #4856, #4860) are kept.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Deregister agent job when the server terminates it
JobTerminate ended the job on the worker but left its JobTerminate
handler and jobToWorker entry registered. Those were only cleaned up
when the worker later sent an ended job status, or on worker disconnect
for jobs still running. Workers that do not report job status after a
termination (agents-js never sends UpdateJobStatus) leaked a psrpc
handler for every job terminated by the server, e.g. each time an agent
participant left its room.
* Wait for worker registration in JobTerminate test
The worker's registration response is sent before its job request topic
is registered, so wait for the WorkerRegistered event before requesting
jobs. Also cover a worker reporting an ended status after the server
terminated the job, and clarify the JobTerminate comment.
* rtc: record a room's departure when the participant leaves it
RemoveParticipant removed the participant from the room, closed it, and
only then recorded leftAt. CloseIfEmpty runs every second, so a tick
landing during the close saw an empty room with a stale or zero
departure time and closed it at once: with no departure recorded it
compares against the room's creation time and empty_timeout, and with an
earlier departure it measures the grace period from that older
timestamp. Either way departure_timeout is skipped.
Record the departure where the participant is removed, under the lock
that already covers it.
* rtc: drop the removal wait from the departure timeout test
The close signal already pins the interleaving, and the room's close
state is decided by the CloseIfEmpty call itself.
* rtc: split EndSession out of MoveToRoom
A participant's session in a room can end without the participant
closing: today on a move, and next on a room re-key, where a room ended
by the sweeper while participants were joining restarts under a new room
id with the same participants in it.
EndSession is the room-independent half of MoveToRoom: report the
published tracks unpublished, report the session end, run the caller's
leave under the ending session's telemetry guard, then take a fresh
guard and reset the deferred resolvers for the next session. The guard
matters: ParticipantLeft releases it and the telemetry service refuses
to open a stats worker for a released guard, so without a fresh one the
next session's join would silently get no worker.
MoveToRoom keeps the subscriber teardown and the onClose fan-out, passed
in as the leave.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rtc: rename EndSession to LeaveSession, add JoinSession
EndSession read as terminal, but the participant survives it and joins
another session, so name the pair for what happens to the participant:
LeaveSession and JoinSession.
JoinSession is the other half that review pointed out was missing from
the API: the caller reports the join, then the published tracks are
reported published again under the new session, pairing the unpublish
LeaveSession reported. Without it the re-publish lived only in the
caller, with nothing in OSS proving the two halves balance.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
participants.
While migrating, the old node will have the recently disconnected
participants in the migrating out participant's cache and needs to sent
out to the participant from the migrating in node via the new WebSocket.
This method can be used to exchange that data between the two nodes.
Keeping it as `recently` disconnected and leaving the semantics upto how
recent for implementation.
There are reports of disconnected participant update missing -
https://github.com/livekit/client-sdk-js/pull/2096. That results in
clients retaining disconnected participant(s) in its roster.
Two changes
1. Look for participant disconnects a little farther back than last
signal time as client app might have lost visibility earlier.
2. Look at all participants within the relaxed window above. The update
cache is LRU. So, in theory can stop when an entry is before the time
threshold above. But, theoretically it is possible that `updatedAt`
time and LRU queuing time can flip ordering although very unlikely.
I think we should do one of the following to make it more robust, but
both require client side changes
1. Send full participant list in `ReconnectResponse`. Clients can use
that to drop participants not in the list.
2. Add a sequence number to all signal responses (maybe signal requests
as well) and have client send up the last response it saw in
`SyncState` message. Server can then replay all the messages after
that. This also requires a message cache on server side.
* dynacast: re-send subscribed qualities after every publisher answer
The SFU answers publisher offers without the simulcast pause markers
(SetIgnoreRidPauseForRecv), so applying the answer re-enables in the
browser the layers that dynacast had paused. Dynacast only notifies
changes, so those layers stayed on until the subscribed quality of the
track changed again.
Send the publisher its committed subscribed qualities again 1 s after
each answer, once the client has applied it: livekit-client handles the
next signal message without waiting for the answer to be applied.
Fixes https://github.com/livekit/livekit/issues/4906
* dynacast: resend once per burst of publisher answers, only for paused layers
In SPC, every subscription change of a participant goes through a new publisher offer and answer, so answers come in bursts when a room is busy. Each answer scheduled its own resend of the subscribed qualities, sending the same update several times in a row.
Thist change keeps one pending resend per participant instead. It fires 2 s after the last answer, and never later than 8 s after the first answer of the burst, so a steady stream of answers cannot postpone it forever. It also skips the resend for tracks whose committed qualities are all HIGH: with every layer enabled, an answer has nothing paused to re-enable. We can save those messages.
* agent: release JobTerminate handler when job ends before registration
JobRequest registered the job's JobTerminate handler after AssignJob
returned, so an ended UpdateJobStatus or worker disconnect handled in
between deregistered nothing and the handler leaked. AssignJob also sent
the assignment before recording the job as running, so an immediate
ended update was dropped.
Record the job before sending the assignment, and after registering the
handler recheck under h.mu that the job is still running and its worker
still registered, deregistering otherwise.
Fixes#4901
* replace utils.CloneProto with proto.CloneOf
* agent: wait on worker readiness and poll for released handlers in test
* Increase receiver loadbalance threshold when batch io enabled
With batch IO the send syscall moves to the BatchConn flush goroutine, so WriteRTP only
translates the packet and enqueues it. Increash the threshold to reduce
* lint
* sfu: don't hold bindLock across the blank-frame flush in CloseWithFlush
CloseWithFlush held bindLock across the up-to-flushTimeout (1s) wait for
the closing blank-frame flush. bindLock is a control-plane lock, also
taken by Bind/SetConnected/ReceiverRestart, so every close serialized
those behind its flush wait. This was the single largest accumulator in
mutex contention profiling under track churn, aggregated across all
closing downtracks.
The flush runs in its own goroutine (writeBlankFrameRTP) and is cancelled
via blankFramesGeneration, so the lock guards nothing during the wait;
release it and re-acquire for the unbind/delete teardown.
Guard the control-plane mutators on isClosed so none can act on a
downtrack through the newly opened window. Bind in particular was
reachable: its bindState != unbound check does not catch a closed track,
since close leaves the state unbound.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: drop redundant isClosed guard in Bind
Bind already handles a closed downtrack after codec matching (returns the
codec with no error, preserving pion's non-error contract and never
rebuilding state). The early guard returned errDownTrackClosed, which
handleRemoteAnswerReceived treats as a negotiation failure, so a
close/negotiation race could abort the remote answer. Remove it and the
error type.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Do not request media sections for transceivers that cannot send
getNumUnmatchedTransceivers counted every local transceiver without a mid.
A transceiver whose track was removed before its section was negotiated
(direction recvonly, no sender) or that was stopped (inactive) can never be
matched to the recvonly section the remote adds in response to a
MediaSectionsRequirement, so it stayed "outstanding" forever: every answer
asked the remote for a section for it again, the remote appended another
batch of transceivers each round, and its offer grew by a fixed amount per
negotiation until it exceeded limit.signal_message_size_limit and the
signalling connection was closed (full reconnect, then the cycle restarts).
Count only transceivers that have something to send. A released transceiver
that AddTrack reuses later becomes sendrecv and is counted again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Address review: blank lines between test functions, assert transceiver reuse
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* ccutils: use millisecond resolution for probe interval backoff
ProbeSignal grew probeInterval via Seconds(), which truncates the float
to the nearest nanosecond before scaling by time.Second:
time.Duration(probeInterval.Seconds() * BackoffFactor) * time.Second
With BackoffFactor=1.5 and the default BaseInterval of 3 s, the first
congestion signal produced 4 s instead of 4.5 s, and the error compounded
on every subsequent signal (4 s → 6 s → 9 s → 13 s ... instead of
4.5 s → 6.75 s → 10.125 s ...). The interval converges to MaxInterval
later than intended, so the server probes more frequently during sustained
congestion than the configuration requests.
probeDuration in the same function already uses the correct idiom:
time.Duration(float64(probeDuration.Milliseconds()) * Factor) * time.Millisecond
Apply the same pattern to probeInterval so both fields are computed
consistently with millisecond precision.
Found by a defect-hunting pipeline I build and run ([Dev-next-gen](https://github.com/Dev-next-gen)), using Claude Code with Anthropic's Claude Opus 5.
* ccutils: remove probe interval backoff test
Requested by reviewer.
* sfu: add opt-in verbatim abs-capture-time forwarding
Downtrack ACT forwarding gates on an RTCP sender report and
unconditionally rewrites the capture timestamp into SFU clock domain,
which suits cross-stream A/V sync but breaks consumers that need the
original publisher capture time (e.g. end-to-end latency measurement).
Add ForwardAbsCaptureTimeVerbatim (RTC config: forward_abs_capture_time_verbatim),
off by default, which skips both the gate and the rewrite.
* fix: use DisableSenderReportPassThrough setting to also gate abs-capture-time forwarding
* deps: update psrpc to v0.8.0
psrpc v0.8.0 splits the brokers into leaf packages so importing psrpc no
longer pulls in nats.go and go-redis, and drops the root constructors:
- return psrpc.NewRedisMessageBus(rc, opts...)
+ return redisbus.New(rc, opts...)
getMessageBus keeps its NewLocalMessageBus branch -- that shim stays at the
root because localbus costs no dependency. wire_gen.go carries a verbatim
copy of the provider, so it gets the identical edit rather than a wire
rerun, which would churn unrelated output.
redisbus.New takes ...bus.BusOption and psrpc.BusOption is an alias for it,
so PSRPCConfig.BusOptions() spreads in unchanged.
The nats-io requirements drop out of go.mod; this module only ever used the
Redis bus, and nats came in indirectly through psrpc's root package.
* Reorder import statements in ioservice_sip_test.go
