RED depacketizing now lives in mediatransportutil so the SFU and
replay export share one implementation. Drop the local copies; the
encoder round-trip tests now exercise the shared package.
Callers passed mono.Now() as a time.Time, which made it easy to pass
time.Now() by mistake and mix clocks. Take mono nanoseconds instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
LastSenderReportTime() returned time.Unix(0, srNewest.At). But At is a
mono timestamp, so the result is not a real wall time. Connection quality
compares it to time.Now() to decide if RTCP is stale. If the wall clock
drifts from mono, for example after an NTP step, streams can look stale
when they are not, or the reverse.
- Convert At to wall time: now minus the mono time since the report.
- Return zero time when At is 0, so the scorer skips the stale check
(before, it got the 1970 epoch and saw the stream as stale).
Follow-up to #5005, which fixed the same mix-up in DLSR.
Tests: rtpstats and connectionquality pass. No new test, because a
test process cannot make the mono and wall clocks drift.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: apply simulcast rids from SDP for WHIP (one-shot) publishers
In one-shot signalling mode HandleRemoteDescription set the remote offer
directly and never called OnSetRemoteDescriptionOffer, so
updateRidsFromSDP did not run for WHIP participants. Every simulcast layer
then mapped to spatial layer 0 and all but the first were rejected with
'duplicate layer'.
* rtc: test that a one-shot simulcast offer applies rids to the pending track
* rtc: use h/q rids in one-shot simulcast test, cover single-layer offer
Name the simulcast test layers medium/h and low/q per convention, and add
TestOneShotSingleLayerOfferUnchanged to check a one-shot offer without rids
still yields a single layer with rid "" at spatial layer 0.
* agent: fix flaky TestAgent job dispatch
Retry JobRequest until the worker subscribes the job topic, because
registration is async and the topic is subscribed after the register response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* agent: report last JobRequest error on dispatch timeout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: reclaim abandoned pending tracks
A pending track is freed only when its first media arrives or the
participant leaves. A client that calls AddTrack and then gives up
before any media flows leaves the slot held for the whole session.
After 20 such slots, every later AddTrack is rejected.
Reclaim these slots when the cap is hit, if no media arrived and the
slot is older than the publish timeout. Keep the supervisor monitor in
step for queued tracks that share a track id. Add a publish_timeout
config value, default 30s.
Fixes: #4970
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: fix publish_timeout comment to not name an internal constant
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: release supervisor-timed-out pending tracks at the limit
Replace the createdAt sweep with the supervisor's publish timeout. The
supervisor counts time only while the publisher is connected and the
track is unmuted, and never reports a live track, so a successor queued
behind a live track is not released.
Restart the publish timer when a queued publish reaches the front of
the queue, so it does not inherit the clock of the track ahead of it.
Write the publication error under the write lock, since it is now read
from another goroutine.
Drop the publish_timeout config value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: skip stale supervisor timeouts and keep live monitors
TimedOutPublications re-checks the monitor, because the cached error is
cleared only on the next tick. A track that published since is not
reported, so its queued successor is not released.
RemovePublication keeps a monitor that still holds a published track.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: drop late media for released pending tracks
Media or an offer with no matching pending track falls back to the first
pending track of the same type. After a release, late media for the
released track could take another pending track's identity.
Record the released track's signal and SDP client ids, the 32 most
recent, and drop media or offers for them. An AddTrack that reuses the
client id clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: clear all ids of a released track when it is re-added
A re-add cleared only the signal client id, so a Firefox republish that
keeps the old SDP id had its offer and media dropped. Keep the ids of
each released track together, and clear the whole track when a new
AddTrack reuses its signal or simulcast codec client id. The limit now
counts released tracks, and a track ages out with all of its ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
An unnamed deployment was addressed as "default" and that spelling leaked
into scope keys, so embedders keyed endpoints differently from job dispatch.
Map "_" back to "" at the URL boundary, as the agent name already does,
and drop NormalizeDeployment.
LoadIngress reads the ingress info and its state from two separate Redis
keys non-atomically, while DeleteIngress removes both keys in a single
transaction. If the delete lands between the two reads, LoadIngress
returns an IngressInfo with a nil State and no error, and
UpdateIngressState panics dereferencing info.State.Status — taking down
the whole server with every room it hosts.
The race fires in practice when an ingress session is being torn down by
DeleteIngress and the ingress service reports the session state change
concurrently: with short-lived per-room WHIP ingresses this crashed
production servers within minutes (observed on v1.13.5 and v1.13.6,
SIGSEGV at ioservice_ingress.go:82, addr=0x8).
Guard the nil State by treating it as an empty state, matching how the
store's UpdateIngressState already tolerates a missing state key.
* turn: refuse TCP relays and limit relays to this host to media ports
Embedded TURN let any participant relay to arbitrary peers (#4924):
- RFC 6062 TCP relays, accepted since pion/turn v5, let a client open TCP
connections to any permitted address, e.g. SSH on the node itself.
WebRTC clients never request them, so they are now refused for both
embedded and standalone TURN.
- Relays to the host's own addresses are delivered locally and bypass any
firewall in front of the host. Embedded TURN now only relays to this
host's RTC media UDP ports.
- Carrier-grade NAT, NAT64 and a few other special-purpose ranges are
added to the restricted peer ranges.
The issue also suggested allowing relays only to the media node's IPs.
That is not done here because, with the changes above, the remaining
relays do not form a security concern: they are UDP only, to public
addresses outside the restricted ranges, and reach a peer exactly as the
client could from its own address (no amplification, no spoofed source).
This is standard TURN behavior. Limiting destinations to media nodes would
also require TURN server to know media node all endpoints, for little benefit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* solve comment
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP9 and AV1 simulcast temporal layers with the dependency descriptor
Every simulcast codec gets an explicit temporal layer selector. With the
dependency descriptor, VP8, VP9 and AV1 filter temporal layers by frame and
forward the descriptor. Without it, a pass-through selector forwards every
temporal layer and reports the target as reached, so a pending target cannot
block allocation. VP8DD becomes the codec-neutral DDTemporal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: VP8-only padding header in DDTemporal, structure gate on presence
DDTemporal returns the one byte VP8 payload descriptor for padding only for
VP8. The descriptor forwarder takes a structure from any packet that carries
one, as the parser accepts it only on the first packet of a frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers across migration
The forwarder state carries the last dependency descriptor frame number
sent, so a migrated subscriber's first key frame continues the numbers
instead of being dropped as older than its last structure. Bumps protocol
for RTPForwarderState.dependency_descriptor_forwarder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: pass on a pending descriptor seed when migrating again
A forwarder seeded before its codec was known returns the pending seed in
its state, so a second migration keeps the frame numbers continuous.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: forward the dependency descriptor for VP8 simulcast
libwebrtc sends VP8 with a one byte payload descriptor when it sends the
dependency descriptor, so a VP8 subscriber gets neither picture IDs nor the
descriptor and finds frame references by sequence number alone. Under loss
with temporal filtering it stalls until a key frame.
DDForwarder rewrites the publisher's descriptor for each forwarded packet:
frame numbers continue across simulcast layer switches and stream restarts,
the structure goes out with the current layer's key frame, and the active
decode targets follow the forwarded temporal layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers from going back
Rebase frame numbers only when a new stream would go back past the last
number sent: on a layer switch, a stream restart (also when it starts with a
delta frame) and a new forwarder on the same downtrack, which now carries
over the last number sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: set VP8 descriptor decode targets from the packet's layer
The active decode targets use the temporal layer the packet is forwarded at,
not the selector's next layer on the last packet before a down switch. They
also go out on key frames, combined with the publisher's own mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: carry the current active decode targets on every DD packet
The parser puts the stream's current mask on every packet, so a mask change
on a dropped packet still reaches the VP8 descriptor forwarder, which no
longer tracks the publisher's mask itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: take the first DD active decode targets mask at any sequence number
A publisher whose first packet has RTP sequence number 0 never had its key
frame's mask applied, so every packet reported no active decode targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A filtered packet that follows a sequence number gap now keeps its slot
and gets padding in it. Rolling it back moved the last outgoing number
below the missing packets' slots, so their retransmissions were rejected
and the subscriber stalled on the hole until a key frame.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: parse VP8 and VP9 payload descriptors without an allocation
Replace ExtPacket.Payload with typed VP8 and VP9 pointer fields that point
at backing structs inside the ExtPacket, so parsing does not allocate and a
wrong codec type is a compile error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: copy the VP8 descriptor in the test packet helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The key frame requester re-armed its timer on every loop, so every video down
track woke up every 200 ms to 1 s for its whole life, even when the layer was
locked and there was nothing to do.
Re-arm the timer only while the requested layer is not locked, or while the
lock can change without an event: an acquisition grace is armed, or layers are
selected with the dependency descriptor, which can lose the lock on the packet
path. Post the event on resync, mute, max published layer change and packet
path layer switch, since those can change the lock without an allocation.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On a gap the RTP munger gives the missing sequence numbers outgoing slots.
If a late packet then arrives and is dropped, nothing is ever sent in its
slot, and a subscriber that finds frame references by sequence number
stalls until a key frame. The forwarder now flags such drops, and the
downtrack sends a one byte padding only packet in the slot.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP8 temporal layers by frame when the dependency descriptor is sent
With the dependency descriptor, libwebrtc sends a one byte VP8 payload
descriptor without a picture ID, so the VP8 munger's picture ID based
temporal drop logic stops filtering after the first gap. A new VP8DD codec
munger decides once per frame from the descriptor's frame number and
temporal ID and applies that decision to every packet of the frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: use the VP8 munger when picture IDs come with the dependency descriptor
VP8DD needs the one byte VP8 payload descriptor that libwebrtc sends with
the dependency descriptor. If a publisher keeps picture IDs next to the
descriptor, they need rewriting, so the forwarder switches to the VP8
munger and keeps switching temporal layers with the descriptor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GetRecentlyDisconnectedParticipants walked the participant update cache
with Get, which moves each entry to the front of the LRU. Walking from
newest to oldest left the cache in reverse order, so after a reconnect
the next updates evicted the most recently updated participants first.
In a room where the subscriber has tracked 128 or more participants, a
participant that had just disconnected was then dropped from the cache
and a later reconnect did not report it as disconnected, and the
out-of-order version check was lost for recently updated participants.
Use Peek, which reads an entry without changing its recency.
libwebrtc strips the VP8 payload descriptor when it sends the dependency
descriptor, so the layer sync bit that #4947 waits for is never set and a
subscriber that lowers its frame rate stays at the base layer until a key
frame. A new temporal layer selector reads only the dependency descriptor:
it switches up at a switch point whose referenced frames were all
forwarded. The forwarder uses it for VP8 when the extension is negotiated
and falls back to the Y-bit selector when the descriptor is not sent.
Fixes#4973.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Firefox 155 enabled AV1 for WebRTC by default, but Firefox 155/156
decode only the base spatial layer of an AV1 SVC stream. SVC is not
visible in SDP, so such a subscriber that negotiates AV1 is forwarded
the publisher's AV1 SVC stream instead of its backup codec, rendering
black whenever a higher layer is selected. Firefox 157 decodes the
higher layers correctly (client-sdk-js#2116), so it keeps AV1.
Removing AV1 from the subscriber's enabled codecs keeps it out of the
SFU's offers and answers, so the down track binds to the backup codec.
Firefox never publishes AV1 either, so it is disabled in both
directions, the same as Safari.
The Firefox for Android H.264 rule is switched to merge so it no longer
drops the AV1 rule for that browser.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* sfu/buffer: parse the dependency descriptor without allocating
Every VP9 and AV1 SVC packet allocated six objects in the dependency
descriptor parse: the descriptor, a clone of the frame template with
its three slices, a resolution copy, and a frame entity that took the
address of a parameter.
The pooled ExtDependencyDescriptor now owns the storage for the
descriptor, the template and the resolution, with fixed arrays for the
template's slices. The reader copies the matching template into that
storage.
Delta frame parse: 6 allocs, 176 B, 174 ns -> 0 allocs, 0 B, 122 ns.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/dd: never copy a parsed frame into a structure template
A caller can hand Unmarshal a descriptor whose FrameDependencies points
at one of the structure's templates. Copying the packet's custom fields
into it would change the template for every later packet, so the reader
allocates a fresh template in that case.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A large frame number jump made addEntity walk every skipped frame,
twice. The ring holds only numEntries frames, so older ones get
overwritten anyway. Both loops now start at most numEntries behind
the new frame, like PacketHistory and FrameIntegrityChecker.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu/buffer: stop allocating per ingested packet
Buffer.Write allocated six objects for every RTP packet. They were
the parsed rtp.Packet, a copy of it in patchExtPacket, and the flow
state, whose address two rare log calls took. The rest were an
AbsCaptureTime, present on about one packet per second, and the
extension slice that pion grows with append.
Unmarshal into a pooled scratch packet. Give ExtPacket its own backing
rtp.Packet and an [8]rtp.Extension array, and copy the header into
them. Patch the payload in place. If the abs-capture-time extension is
present, parse it. Log a copy of the flow state.
Per packet, Write plus ReadExtended. Opus: 6 allocs, 448 B -> 0.
VP8: 6 allocs, 416 B -> 1 alloc, 32 B. Opus 240 -> 142 ns, VP8 269 -> 204 ns.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/buffer: point extension payloads at the bucket copy on read
Queued ExtPackets kept extension payload slices into the write input,
which the transport reuses for the next packet. ReadExtended now parses
the header again from the bucket copy, so extensions and CSRCs point
into RawPacket like the payload does.
The test writes every packet through one input buffer and checks the
audio level of each packet read back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu/buffer: test that a dropped padding packet keeps the shifted sequence number on read
The bucket stores the shifted number, so re-parsing the header on read
returns it. This test guards that.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The static client config disabled H.264 publish for Firefox on Linux or
Android. The Android workaround came from #2183, but the linux clause
only ever hit desktop Linux Firefox, which publishes H.264 fine through
OpenH264. Narrow the rule to Android only.
Fixes#4929
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* bump protocol to get unlock on no key fix
* update warp pointer
* update protocol and psrpc deps
* update protocol dep
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A subscribe that found the reconcile queue full was dropped and waited
for the 3 s ticker. Flag the overflow, and have the worker reconcile
all subscriptions after its next queued item.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* agent: HTTP endpoints data plane
Serve worker-declared FastAPI routes at /agents/{deployment}/{path} without
any worker-side listener: workers dial a fixed pool of wires speaking
AgentHttp.Frame, the server opens multiplexed streams carrying one opaque
HTTP/1.1 exchange each (two-level credit flow control, prioritized write
scheduler, attach epoch fencing). The front does starlette-exact manifest
matching with per-endpoint public access, typed 401/404/405, a retry table,
and SSE/WebSocket passthrough; a pluggable fallback hook lets multi-node
deployments resolve misses elsewhere. Includes a conformance client, a
manual sidecar, and the acceptance suite.
The auth middleware now reads access_token from the query string only:
FormValue consumed the bodies of proxied url-encoded POSTs.
* agent: attach adopter hook for wires landing on foreign nodes
Behind a load balancer a data wire may reach a node other than the one
holding the worker's registration; the adopter lets that layer (cloud)
install a local satellite registration before the attach is retried locally.
* agent: distinguish wrong-epoch attach so a multi-node adopter can refresh
A wire whose worker is registered but under a different epoch now returns
ErrWrongEpoch (not ErrAttachRejected); pool-full/closed stay ErrAttachRejected.
The attach adopter runs on unknown-worker OR wrong-epoch, so a satellite left
behind by a reconnected worker is refreshed by re-validating against the
registration holder instead of shadowing the new epoch until it is swept.
* agent/endpoint: Manifest.Version() digest for skew detection
Order-independent fnv64a of the route set so a multi-node layer can tell
whether the nodes serving a deployment agree on the manifest (safe to route
without inspecting the path) or are mid rolling-deploy (fall back to a
path-aware resolve).
* agent/endpoint: carry raw endpoints on the registration
A multi-node layer can now replicate a node's route set and match request
paths against it locally, instead of a manifest digest. Drops the digest
helper added for that (superseded).
* agent/endpoint: canonical route matcher (typed tokens + pruned walk)
Shape-matches a request path against a set of route templates via canonical
typed-wildcard tokens and a left-to-right pruned walk: linear in path depth
(no 2^segments), type-aware (int/float/uuid/str/path), glob catch-alls, and a
PrefixIndex interface so the same walk runs over an exact map (local) or a
compact replicated backing (multi-node). Worker stays authoritative for the
exact convertor/method match.
* agent/endpoint: fix glob matching and enforce route depth
- a path convertor anywhere in a template (non-terminal, or mixed with literal
text in a segment) now canonicalizes to a terminal glob that spans slashes,
instead of being dropped or narrowed to a single-segment str - the old
behavior silently failed to match such routes (a false negative -> misroute).
- enforce MaxRouteDepth at ParseManifest (wires up RouteDepth), bounding the
replicated filter's prefix count and the matcher's walk.
* agent endpoints: P2C worker balancing, routing-miss sentinel, api-key-scoped attach
- worker selection uses power-of-two-choices over live in-flight streams
(InflightStreams / SpareStreams) instead of weighted-random over a
reported load; the conn pick stays least-loaded. In-flight is observed
locally so it is correct for adopted wires, and the reported Load hook is
dropped from the endpoint path.
- the front tags its own routing misses (X-Livekit-Endpoint-Miss) so a relay
caller can distinguish them from a worker app's own 404/405 and retry past
them across the remaining candidates.
- a data-wire attach is bound to the connecting grant's api key, so a leaked
attach token cannot bind a conn to another project's worker.
- remove the dead scope-hook machinery (it only fed the removed bus resolve)
and rename the request identity from "scope" to apiKey.
* agent endpoints: drop the sidecar cmd, make the conformance client test infra
- remove cmd/agent-endpoint-client: the Python SDK has landed, so the manual
sidecar for non-SDK workers is no longer needed.
- move pkg/agent/endpoint/client to pkg/agent/endpoint/conformance (package
conformance) so it reads as the test harness it is, not a product client
SDK; only the acceptance suite drives it.
* agent endpoints: pool the per-request bridge buffers
- reuse the 32KiB response-copy buffer and the response-head reader from
sync.Pools; ~27% fewer bytes allocated per small request.
- reuse a per-conn marshal buffer in WriteFrame instead of allocating one per
frame; ~19% fewer bytes on a 1MiB upload. Safe because stream.Write copies
the payload and gorilla WriteMessage does not retain it.
* agent endpoints: reuse the read buffer and bound the data wire
- ReadFrame reads each frame into a reused per-conn buffer via NextReader
instead of letting ReadMessage allocate one per frame; ~41% fewer bytes on a
1MiB download. Safe: proto.Unmarshal copies the payload out and readLoop is
the sole reader.
- the data wire's websocket read is bounded by its own max message size (a full
frame + framing overhead), independent of the signalling message limit, so a
peer cannot OOM the server with an oversized frame even when signalling limits
are disabled. Drops the old coupling that clamped MaxFrameSize under the
signalling limit.
* agent endpoints: frame large uploads at MaxFrameSize and pool the buffers
Stream now implements io.ReaderFrom, so http.Request.Write's bufio.Writer
delegates the body to it and a large upload flows in MaxFrameSize frames instead
of ~4KiB ones - 16x fewer frames, marshals, scheduler cycles and websocket
messages. Large frame buffers (>= 16KiB) are drawn from a per-conn sync.Pool and
recycled by the scheduler after each write, so a big upload stops allocating a
buffer per frame; small/webhook bodies stay below the threshold and use a
right-sized make, so the average request is unaffected.
Measured on a 1MiB upload: -82% allocations, -66% latency, -17% bytes; a small
GET/POST is unchanged.
* agent endpoints: drop client WebSocket route support, repin protocol
* agent endpoints: repin protocol for the enum renumber
* agent endpoints: WebTransport data plane
Replace the capsule mux / credit / attach-pool machinery with one WebTransport
(QUIC) session per worker: a control stream carries registration and status,
and each HTTP exchange rides its own QUIC stream (native mux, per-stream flow
control, half-close, and reset). Encode the HTTP method into the cross-node
route filter so candidate selection is method-aware. Keep the legacy WebSocket
control transport for backward compatibility.
* agent endpoints: tidy front helpers and constants
Use endpoint.CurrentProtocol and a named SessionCloseOK instead of bare
literals, factor the repeated 503 and header-copy blocks into helpers, and drop
a few restating test comments and a redundant assertion. No behavior change.
* agent endpoints: assert deployment scoping in the registry test
* agent endpoints: drop file-narration header comments
* agent endpoints: normalize trailing slashes during routing instead of 307
A slash mismatch (e.g. /hook/ for a route registered as /hook) is now rewritten
to the registered form and served directly, rather than 307-redirected. Webhook
clients often don't follow redirects, and a redirect from the final routing hop
resends the body and pays the whole routing path twice. Exact form is matched
first, so a route registered with a trailing slash is served as-is.
* agent endpoints: route at deployment granularity, drop the route filter
Route presence like a reverse proxy keyed on (project, agent_name, deployment):
any of a deployment's workers is a candidate and the worker's own router returns
the real status (a 404 for a path it doesn't serve during a rolling deploy is
returned, not re-relayed). This removes the whole per-node route-advertisement
path: the cuckoo route filter and matcher (routematch.go), the miss-tagging /
retry-past-miss machinery, and the route-depth cap that only existed to bound
the filter. Text routing is now the same shape as voice job dispatch.
* agent endpoints: audit cleanups (drop dead Registration.Endpoints, test the fallback flow)
Remove the write-only Registration.Endpoints field (matching goes through
Manifest; the multi-node layer no longer replicates the raw route set). Add
Front-level fallback tests (fires with the resolved identity, declined -> 404
with a local worker or 503 without), and fix two stale comments.
* agent endpoints: byte-transparent HTTP/1.1 exchange
Replace the protobuf frame layer on exchange streams with a length-prefixed
StreamPreamble followed by opaque HTTP/1.1 bytes. QUIC already provides the
multiplexing and per-stream flow control, so the frame layer only re-encoded a
message both peers can already parse, and every SDK had to re-materialize HTTP
from it.
The front serializes a canonical request head from its own parsed
*http.Request and never forwards the client's bytes, which is what keeps
request smuggling out of the worker. Responses are read with
http.ReadResponse, so informational heads need no httptrace hook and the
conformance worker loses its HTTP reconstruction entirely.
Completion splits by outcome: a body ends by its own framing, a failure after
bytes have flowed travels in x-lk-completion / x-lk-error trailers, and a
failure before any byte resets the stream with an HttpStreamResetCode. REFUSED
stays distinct as the retry-safety signal, and the retry/idempotence rules are
unchanged.
The x-lk- prefix is reserved for this signalling: it is stripped from
client-supplied request headers so a caller cannot forge an outcome, and from
responses so it never reaches the end client. Exchange-stream targets are
split from the escaped path so a percent-encoded '?' or '/' cannot change
which resource the worker routes to.
Per-request state moves to attempt.go so the Front and attempt lifecycles stop
interleaving; the remaining files are grouped type-first with free helpers
last. Those moves are ordering only.
* go.mod: pin protocol with the agent endpoints preamble
Picks up livekit/protocol 134fb913, which replaces the HTTP framing messages
with StreamPreamble and renumbers HttpStreamResetCode. The previous pin
predates AgentHttp entirely, so the endpoint packages only built through the
local go.work.
* agent endpoints: require a scoped grant for non-public routes
The front gated non-public routes on a bool that was true for any valid token
in the project, so a roomJoin token minted for an end user reached every
non-public endpoint the project's agents expose.
Replace the resolver's (apiKey, authenticated) pair with an Access value and
give it the agent and deployment from the URL, so the scope rules stay with
the grant in protocol/auth and this package consumes only the verdict.
Credentialed now separates a caller with no credential from one whose
credential lacks the grant: the first is challenged with 401, the second gets
403 and no WWW-Authenticate, since a retry cannot succeed.
A denied request still reaches the fallback. Another node's worker may declare
the same path public, and that node is the authority.
* go.mod: pin protocol with the agent endpoint grant
* agent endpoints: accept any agent name, keep request escaping intact
The name is percent-encoded into its URL path segment, so it needs no charset.
Only "_", "." and ".." are refused, and only for workers declaring endpoints:
all three are unreserved, so no encoded form addresses them distinctly. "_"
now addresses a worker registered with an empty agent name, which the
implicit-dispatch default leaves unset.
Two escaping fixes the escaped-path split depends on:
- the trailing-slash alternate treated a decoded slash as literal, so
/known%2F normalized onto route /known and the worker was handed a path it
does not serve.
- RemoveDoubleSlashes trimmed URL.Path and left RawPath, which makes
EscapedPath() re-encode from Path and lose every escape in the request.
Request logs now carry the encoded path, since a decoded %0A forges a log line.
* agent endpoints: return the abort from bridge rather than panicking inside it
A committed response that cannot be completed is torn down with
panic(http.ErrAbortHandler): the head is on the wire, so no status can report
the failure, and this is the only teardown net/http offers. HTTP/1.1 closes
without the final chunk and HTTP/2 sends RST_STREAM, so a short body cannot
read as whole.
bridge raised it from inside the response copy loop, through a helper whose
doc had to note that it does not return. It now reports a bridgeOutcome and
ServeHTTP performs the teardown at the handler boundary, where the constraint
that nothing may recover the sentinel is visible to a reader. The outcome
replaces the (done, retryable) pair, whose four combinations only ever carried
three states.
No behavior change.
* agent endpoints: serve the front on its own middleware chain
The front was mounted on the API mux and served through the API middleware
chain, which is shaped for handlers that buffer a request and write one
response. Four consequences.
negroni.NewRecovery() recovers every panic with no http.ErrAbortHandler
exemption, and NewRecovery sets PrintStack. On an abort it wrote "PANIC: ..."
and a goroutine stack into the already-committed body, then returned normally,
so net/http terminated the chunked stream cleanly. A truncated response
reached the client as complete, with a stack trace appended. Detection was
already correct; delivery was not, and the x-lk- trailers that would otherwise
signal it are stripped before the response leaves. A Content-Length response
still failed safe, since net/http enforces the declared length itself, so the
gap was the chunked and trailer paths.
Endpoints.Disabled documents that it turns the front off, but only
registrations were refused; the mount was unconditional.
The CORS method list omits PUT, which a manifest may declare.
The API body limiter capped request bodies at MaxAPIRequestBodySize, though
the front streams a body through a pooled buffer and never holds one.
NewHTTPHandler now routes the prefix to a chain carrying AgentRecovery, a CORS
list matching the methods a manifest may declare, and the api-key auth
middleware the front resolves a caller's access from. The mount is built only
when endpoints are enabled, so the prefix otherwise falls through and 404s.
RemoveDoubleSlashes moves above the split, so routing and both chains see one
path form.
Taking the front off the mux also stops ServeMux rewriting the paths it is
handed: "//x", "/../" and interior "//" were answered with a redirect rather
than proxied, which a byte-transparent exchange cannot do.
The endpoint stack tests now build the production handler, so they run on the
chain the node serves on.
* agent endpoints: build the front and registration from params
Front is an http.Handler, so fallback, identity and singleKeyFallback were read
on every request with no synchronization while WithFallback, WithIdentity and
WithSingleKeyFallback wrote them on the live object. Each returned the same
pointer, so the chain read as though it built a value. Every call site happens
to run before the handler is mounted, so nothing races today, but nothing in
the type prevents it, and WithSingleKeyFallback took no argument and could only
be turned on. NewFront now takes FrontParams and configuration is fixed at
construction.
NewWorkerRegisterer takes its EndpointSettingsFunc directly, and
HandshakeAgentWorker takes one in place of a variadic of raw closures that
existed only to reach the setter it replaced.
Registration gains NewRegistration and RegistrationParams, moving the Draining
callback off an exported mutable field and folding SetSession into
construction. IsDraining absorbs the nil check at both call sites.
Access carried a three-state ladder as two bools, with "granted implies
credentialed" documented but unenforced. It is now an ordered AccessLevel, so
callers compare a rank rather than combining flags and the invariant holds by
construction.
Registry.Register returned an error that was always nil, with a dead branch at
each call site. Both registry maps and the per-registration session are
read-heavy, so they take RWMutex.
CopyBody returned two errors to separate a source failure from a destination
one. It returns one, wrapping a source failure in *SourceError, which is what
the caller discriminates on.
NewWebTransportServer took a callback to break the handler/server init cycle;
the caller assigns wt.H3.Handler after construction instead. StartWebTransport
reads Development off the service rather than taking a bool, and returns a nil
stop function where it starts no listener.
Smaller: slices.Sort for sort.Strings, for range for an unused counter, a nil
slice for regs[:0:0], p2c generic over its slice so the call site passes a
method expression rather than allocating a closure per request, and streamCode
deduplicated into wire.StreamCode so both peers map reset codes in one place.
Comments on the touched declarations drop remote behavior, migration narration
and contrastive framing, keeping the constraints and invariants.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: serve webtransport from the node's server set
StartWebTransport built the node's only HTTP/3 listener from inside AgentService:
it minted the TLS config, bound the UDP socket, assembled an h3 mux and returned
a stop closure LivekitServer held in agentWTStop. Every other listener is
constructed in NewLivekitServer and delegates to a service, and agents are the
first consumer of HTTP/3 rather than the last.
webtransport.go now carries the transport with no agent references:
WebTransportTLS, NewWebTransportServer, NewWebTransportHandler,
ListenWebTransport, and WithWebTransportServer/GetWebTransportServer, which put
the serving server in the request context so a route that upgrades is a
conventional http.Handler instead of a closure over the server. The listener is
built in NewLivekitServer beside the prometheus and debug servers, bound in Start
with the other listeners, and stopped once doneChan unblocks, so no stop closure
is held on LivekitServer. /agent is a route on its mux, as it is on the API mux.
UpgradeWebTransport unwraps the ResponseWriter before upgrading. webtransport-go
type-asserts it to http3.Settingser and http3.HTTPStreamer without checking and
negroni wraps the writer on every chain, so the h3 listener could not carry a
middleware chain at all. It now runs the same recovery, api-key auth and path
normalization as the TCP chain.
AgentService splits into AgentWSService and AgentWTService over the shared
AgentHandler, and the endpoint front becomes AgentEndpointService. wire builds
all four and threads one endpoint.Registry into the handler and the front.
NewAgentHandler takes the config and builds its own ServerInfo, so endpointsConfig
and singleAPIKey are set at construction and the psrpc server is registered
against a fully formed handler; it previously bound one whose embedded
*AgentHandler was still nil.
The listener's port and certificate move to a top-level webtransport config
block. The listener no longer reads agents.endpoints.disabled, which continues to
refuse endpoint registrations. ListenWebTransport binds one socket per bind
address, where the h3 listener ignored bind_addresses that every TCP listener
honoured.
NewWorkerRegisterer takes a variadic WorkerRegisterHandler, each handler reading
the request and filling its part of the response and the registration, replacing
the single EndpointSettingsFunc threaded through HandleRegister.
EndpointRegisterHandler is the endpoint declaration's turn, and
endpoint.NegotiateSettings holds the protocol negotiation the OSS and cloud
handshakes each carried a copy of.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: accept webtransport connections rather than racing Serve with Close
ListenWebTransport ran wt.Serve on a goroutine per socket and returned a stop
func calling wt.Close. Serve takes a reference on the server's refCount
WaitGroup and Close waits on it, so the pair breaks the WaitGroup's own rule
that an Add starting from zero must happen before a Wait. The race detector
models that rule as a read of wg.sema in Add against a write in Wait, and
reports it whenever Wait observes a non-zero counter: a listener stopped
before its serve goroutine has run at all, which is every test that builds a
stack and tears it down without a worker connecting. sync is compiled without
instrumentation, so the report names the two ListenWebTransport call sites
with no frame in between. webtransport-go v0.13.0 carries the same code.
The accept loop moves here. quic.ListenEarly builds the listener Serve would
have built, with the datagram and partial-delivery options the session layer
requires, and each connection goes to Server.ServeQUICConn, which touches no
part of that WaitGroup. The stop func keeps the order Close established:
Close first, so every CONNECTION_CLOSE frame is transmitted while the sockets
are still open, then the accept loops are cancelled and drained, then the
listeners and sockets close.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: sequence the endpoint tests on the registry
handleConnection writes the register response from inside HandshakeAgentWorker
and installs the worker's routes after it returns, so a worker learns it is
registered a moment before the node can route to it. The conformance worker's
WaitRegistered unblocks on that response and startWorker returned straight
into the first request, which on a loaded runner arrived while the registry
was still empty and came back 503. TestAgentEndpointsStatusMapping saw that as
503 for the first few mapped statuses, TestAgentEndpointsHOL as an unexpected
EOF, reading 1024 bytes out of a short error body.
The stack keeps the registry it builds and startNamedWorker waits for the
worker's registration to appear in it, which is the condition the requests
actually depend on. TestAgentEndpointsRetrySafety builds its unreachable
worker by hand and waits the same way. The HOL test asserts its status before
reading the body, so a 503 there reports as a status mismatch rather than a
truncated read.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: log the deployment with worker registration
Workers are keyed by deployment alongside agent name, namespace and job type.
The registered and last-deregistered lines named every part of that key except
the deployment.
* agent endpoints: add a path router that matches in declaration order
Manifest matching runs a compiled regex per route over a linear scan, once per
candidate registration, on the serving path. At the 256-route cap that is up to
256 RE2 executions per replica per request.
The router compiles a manifest into a compressed trie instead. Starlette
selects the first route declared that matches, so the trie cannot resolve
static before wildcard the way net/http's does: every node carries the lowest
route index in its subtree, the walk tracks the lowest full match found and
prunes any subtree that cannot improve on it, and inserting in declaration
order leaves edges and leaves sorted by that index with no sort pass.
Params are not segment-aligned - starlette allows /f/{name}.{ext}, a {p:path}
anywhere, and a float whose fraction backtracks - so the walk must search. An
edge is single when nothing below it can start with a byte its own convertor
could have consumed, which is a property of the target node and so is settled
at build time; a single edge takes its greedy run and descends once. Templates
whose params are segment-aligned are entirely single and never search. What
remains is bounded by a step budget, and exhausting it returns ResultOverBudget
rather than a route the cut-short search cannot vouch for.
Templates parse without regexp. Scanning the grammar by hand keeps a brace that
opens nothing well-formed as an ordinary literal, which is what starlette's
finditer does. Anchoring is `\n?$` rather than `$`: python's '$' matches before
one trailing newline where Go's does not, and [^/] accepts a newline where .
refuses one, so the two convertors diverge in opposite directions on a decoded
%0A.
The regex implementation moves into the tests as an oracle, carrying its own
parser so it references the hand-rolled scanner as well as the trie.
FuzzMatchAgainstOracle generates a table and a request and asserts the two
agree on both the winning template and the result. Matching allocates nothing,
asserted by AllocsPerRun on hit, 405, miss and on the searching path.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: route manifests through the path router
ParseManifest builds a router.Router rather than a slice of compiled templates,
Manifest aliases it, and Route keeps the template and the public flag the front
reads. Declared methods become a mask at build time, so a method no route could
declare masks to 0 and matches nothing, as an undeclared one already did.
Three behavior changes ride along.
Templates anchor as `\n?$`. A manifest declaring private /files/{p:path} before
public /files/{p} routed GET /files/x%0A to the public route here and to the
private one on the worker, because the path convertor refuses a newline where
str accepts it. The access gate reads the public flag of the route the front
picked, so it opened on a route starlette would not have selected.
slashAlternatePaths derives the escaped form from the transform it applied
rather than from the result's suffix. A request path of // trimmed to /, the
caller saw a trailing slash on that result, took the append branch, and handed
the worker a request line of /// against a decoded path of /.
The decoded path is capped at MaxPathLength, answering 414 above it. Matching
runs before anything sizes the request head, so net/http's own limit was the
only bound on what the matcher was handed.
A table too ambiguous to decide now dispatches rather than 404s: the worker's
own router still knows, and the preamble carries no route for it to be told
about. Nothing here knows whether that route is public, so the request needs a
grant. Dispatch keys on the match set, since a candidate can now join it
without resolving a route.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: log the manifest as a structured object
Router implements zapcore.ObjectMarshaler, so registration logs the table's
size and the templates whose shape forces the matcher to search under one key
rather than assembling fields at the call site. Which templates those are is
settled when the manifest is parsed, so it is worth reporting before any
request is served; what a single match cost belongs to that request instead.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: key templates by shape and export their ambiguity
Canonical renders a template as its literals and convertor kinds, so two
templates the trie cannot tell apart share a key. Param names reach the trie
only as a kind, so /x/{a} and /x/{b} accept the same paths; anything keying
routes by the declared spelling splits them apart. Literals are length-prefixed
because a brace that opens no well-formed param is literal text: /x/{:str}
matches itself and must not collide with /x/{a}.
ambiguousTemplate becomes Template.Ambiguous. It reads only the template's own
elements, so it answers for a manifest that was validated but never compiled
into a table of its own.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: merge worker manifests into one table per deployment
A deployment owns its route table. A route is one template shape and one method
together with the workers that declared it; registration takes the union of the
table and the arriving worker's manifest, deregistration retracts the worker and
drops the routes nothing is left declaring. Manifest stops being a compiled
router and becomes the ordered list a worker declared, so a fleet holds one
table rather than one per replica, and a request resolves in a single walk.
A mixed fleet is the point. Matching each worker's own table in turn already
served heterogeneous methods, but the route the front reported was the first
candidate's while the worker it dispatched to was any of them, and partial and
denied were accumulated across workers whose manifests disagreed. Both become
properties of one table, so the status a request gets and the template the
worker is told are decided together.
Merge order is the lowest position a route holds in any declaring manifest, tied
by shape then method. A fleet declaring one manifest reproduces that manifest's
order, and the result reads off the current set of manifests, so nodes holding
the same fleet agree. This does not preserve any one worker's internal order: a
worker can be handed a request the merged table resolved to a public route while
its own table routes that path to a private one. Merging the declaration chains
in order would hold the property and is not done here;
TestRouteTableMergeOrderInversion carries the case.
Routes are keyed by shape, so /x/{a} and /x/{b} are one route and a renamed path
param does not halve the fleet serving it. Each worker keeps the spelling it
declared, and the preamble carries that, since it is re-serialized per attempt.
The tree holds routes by pointer and is rebuilt only when the route set or its
order moves, so a worker joining or leaving an existing route swaps nothing. A
supersede runs the removal and the addition as one transaction: an unchanged
manifest would otherwise empty every route, drop it, and build a new one under
the same key, leaving the published tree naming routes nothing can reach and no
change to the key set to show for it. A reconnect can also land on a different
agent name or deployment, so the retiring epoch is retracted from its own table.
Ambiguity is now a property of the merged tree, and one worker's template shape
can carry the whole deployment's matcher over its step budget, which leaves no
route decided and so no public flag to clear an ungranted request.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* agent endpoints: scope the registry, let the embedder own tenancy
The registry was doing three jobs at once: fencing worker epochs by id,
listing a deployment's candidates, and holding its merged route table. The
last two are per-deployment state, so it keyed them on
(api key, agent name, deployment) and grew a tenancy concept that only an
embedder can actually define. Cloud has to lie to it, passing a project id
in a field named APIKey.
Split them. Scope is one deployment's serving state and stores no identity
at all; whoever embeds the package keys a map of scopes however its own
tenancy works, and hands the front a resolved one. Registry keeps only the
worker-id fence, which is genuinely node-wide: worker ids are server-issued,
so an epoch is superseded wherever it was scoped.
The front loses its registry, its SingleKeyFallback and FallbackRequest: the
resolver now returns the scope and a fallback already curried on the
deployment, plus an ok that carries the 401-vs-503 split the empty api key
used to encode. routeTable drops its key and takes the scope's logger, so
identity is curried in rather than stored.
pkg/service takes ownership of the "api key is the tenant" rule, which is
true there and nowhere else, and of releasing a scope once nothing holds it.
Behavior is unchanged, including serving public routes to an unauthenticated
caller when one configured key or one attached tenant makes the key
unambiguous.
* agent endpoints: give the http3 listener a server type
ListenWebTransport returned a stop closure, so every caller had to park it
next to the server it belonged to. In cloud that meant a second field on the
node base, written at start and read at stop, for one listener.
WebTransportServer embeds webtransport.Server and owns the sockets and
accept loops, so Listen and Shutdown are methods and the shape matches
http.Server. Shutdown takes a context: it stops accepting, sends GOAWAY, and
closes whatever has not drained by the deadline, where the closure closed
everything at once with no deadline of its own. It is safe on a server that
never listened, which is what lets a holder key teardown off the field alone.
The embedded Close still releases sessions without releasing the sockets;
that is what the type comment warns about.
* agent endpoints: fold the ident-start test into a helper
staticcheck QF1001 flagged the negated disjunction in scanIdent, which was
failing the lint gate and so keeping the test job from running at all. The
positive helper reads better than the De Morgan form and lets the continue
condition drop its own '_' case.
* go.mod: bump protocol to the merged branch head
* rtc tests: give the logger its own logging config
init() handed the logger &config.DefaultConfig.Logging, so the leveler's
mutex lived inside the global that NewConfig yaml-marshals. Marshalling
reflectively copies the whole struct, mutex included, which raced the
component-level lookups pion makes from its own goroutines and failed
TestPreferMediaCodecForPublisher under -race.
The values are the same; only the storage is now test-local. The integration
helper aliased the same global and gets the same treatment.
* rtc tests: drop the logging config comment
* auth: name the call the query-string read avoids
The token comes from RawQuery so the body reaches a proxied agent endpoint
intact. That only holds as long as nobody reaches for FormValue, which is
the obvious way to read a param and the one that drains the body, so the
comment names it.
* agent endpoints: tunnel WebSocket upgrades to workers
An exchange stream already carries opaque HTTP/1.1, so a WebSocket is an
ordinary GET route: the front forwards an HTTP/1.1 Upgrade: websocket
handshake, the worker's HTTP stack decides, and on a 101 the front hijacks
the client connection and splices it onto the stream. No protocol change.
- Only HTTP/1.x GET with Connection: upgrade and Upgrade: websocket (and no
body) is an upgrade; every other Upgrade token is still stripped. A 101 is
accepted only for such a request and only with Upgrade: websocket;
anything else stays a protocol error.
- A non-101 answer to an upgrade is an ordinary response; the stream then
ends with a FIN rather than a reset.
- Retries are unchanged until the 101; after it the attempt is committed.
- Each worker takes at most MaxTunnelsPerWorker tunnels (default half its
session's MaxStreams) so WebSockets cannot starve its HTTP routes; an
upgrade no worker can take falls to the fallback, then 503.
- Drain: a tunnel outlives its worker starting to drain by
TunnelDrainTimeout (default 30s), and Front.Drain closes tunnels once its
context is done. Both close with a 1001 frame when the client-bound side
is between frames, abruptly otherwise. The server drains tunnels while
participants leave, since http.Server.Shutdown cannot see them.
- access_token is dropped from the target and a LiveKit bearer token from
Authorization for every request: the worker gets identity from the
preamble, and the token must not reach application code.
- Full duplex: enable it before the first flush, so an HTTP/1 client can
keep streaming its body while the response streams back.
Config: agents.endpoints.max_tunnels, agents.endpoints.tunnel_drain_timeout.
* agent endpoints: keep one value per CORS header
The agent chain's CORS middleware writes Access-Control-* and Vary before
the front runs. A worker's head was added on top, duplicating
Access-Control-Allow-Origin, and the clear after a 1xx head dropped the
middleware's headers from the final response.
- A worker's Access-Control-* and Vary values replace the middleware's;
other headers are still added (MergeWorkerHeaders).
- Headers set before the front ran are restored after each 1xx head.
- Preflights stay with the middleware; a plain OPTIONS still reaches a
declared route. NewAgentEndpointCORS holds the agent chain's CORS
middleware so other chains can share it.
* agent endpoints: withhold endpoint settings from sessionless workers
A worker registering over WebSocket has no data plane, yet the handshake
negotiated endpoint settings, telling it its routes were live while every
request 503'd. Negotiate only when a WebTransport session exists, per the
endpoint_settings contract, and keep the worker registered for jobs.
* deps: pin protocol to the agent endpoints branch head
* deps: pin merged protocol
---------
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep one subscription when first subscribes race
SubscribeToTrack and SubscribeToDataTrack looked up the subscription,
then created one and stored it under a separate lock. Two concurrent
first subscribes, or a subscribe and a settings update, could each
store their own entry. The replaced entry could still add the down
track, which nothing owned after that, and the settings could land on
the replaced entry.
Find or create the subscription and set it as desired under one lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: keep early settings and a racing data track resubscribe
A settings or options update before the first subscribe creates an
entry that is not desired. Cleanup removed it on the next pass, so the
subscribe created a new entry without the settings. Keep such an entry
for notFoundTimeout.
After a data track unsubscribe, the entry was deleted even if a
subscribe set it as desired during the removal. Delete it only if it
is still not desired, else reconcile it again.
Cleanup deletes only the entry it checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
RedReceiver kept pointers to the last forwarded packets for its
redundant blocks, but their payload is in the read buffer that
forwardRTP reuses for every packet, so the blocks carried the current
packet's bytes. Copy the history packets into slots RedReceiver owns.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: use the most common recent packet rate in connection quality
The loop that finds the mode packet rate compared a bin's count with the
index of the bin picked so far, so it did not find the most common rate.
Take the bin with the highest count, skipping rates below 10 pps so that
idle, DTX and static content rates do not become the reference. Halve the
counts at each mode update so that the mode follows a lasting change of
rate, and keep the previous mode through a long stretch of low rates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: start the mode timer from the start time
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GenerateCLIFlags created an Int64Flag for every time.Duration config
field. Setting one through its LIVEKIT_* env var or --flag with a
duration string such as "30s" (the format the YAML config and the
generated help text use) made the server exit at startup with a parse
error. A value without a unit was accepted and read as nanoseconds.
Use a DurationFlag for these fields, so they take the same values as in
the YAML config.
* fix: use the fast debounce for the first negotiation
A new transport set lastNegotiate to its creation time. A non-forced
Negotiate soon after creation then waited for the 150 ms debounce
instead of 10 ms, although nothing was negotiated yet. Start from the
zero time so that the first offer uses the fast debounce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check the first negotiation condition without a deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: check that the first offer goes out before a debounced one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix: do not read invalid delta
In TWCC feedback, only the small delta and large delta symbols carry a
receive delta. The send side BWE loop used a delta for every received
symbol, so the reserved symbol read a delta that does not exist. Skip
that symbol and do not read past the last delta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: assert loss count and delta use for the reserved symbol
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test: give mock participants a connection quality
A real participant never returns a nil connection quality. The mock
did, which can crash code that does not expect nil.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: close test rooms when each test ends
Most room tests left their rooms open, so each room's workers kept
running for the rest of the package.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
selectAlternativeVideoCodec walked the enabled codecs and returned the
first one with a video/ prefix. video/rtx and video/flexfec-03 both have
that prefix, so when the requested codec was not enabled the track could
fall back to RTX. RTX is a retransmission stream, not a real codec, so
the track got registered as video/rtx and subscribers saw a blank tile.
Skip RTX and FEC entries so the fallback is always a valid video codec.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* service: start room composite egress configured in a room preset
StandardRoomAllocator.CreateRoom merged a room preset into a copy of the
request and discarded it. RoomInternal has no field for room composite
egress, and getOrCreateRoom started it from the caller's original request,
so a room composite egress from a preset was never started.
RoomAllocator.CreateRoom now also returns the request with the preset
applied, and getOrCreateRoom starts the egress from it.
* service: return ErrEgressNotConnected instead of calling a nil egress launcher
getOrCreateRoom started a room composite egress without checking that an
egress launcher exists. RoomService.CreateRoom rejects egress on the request
when there is none, but not egress from a room preset or a token's roomConfig.
