Bound the size of HTTP request bodies on the main API listener so large
messages cannot exhaust memory. Configurable via limit.max_api_request_body_size
(defaults to 10 MiB, 0 disables).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add configurable read-message size limit on signalling WebSockets
Set a read limit on both the client-facing (/rtc) and agent worker
WebSocket connections so an oversized frame is rejected by the transport
before being buffered. The limits are operator-tunable via
signal_message_size_limit and agent_signal_message_size_limit, both
defaulting to 2 MiB (0 disables).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add tests for signalling WebSocket read-message size limit
Cover the configurable signal_message_size_limit added in the prior
commit:
- config: assert both limits default to 2 MiB and that a YAML override
(including 0 to disable) is parsed correctly.
- full-path integration: a real client connects to /rtc on a single-node
server and an oversized frame is rejected by the transport with a 1009
close; a 0 limit leaves the connection unbounded and signalling
proceeds.
Adds setupSingleNodeTestWithConfig so a single-node server can be started
with config overrides.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bound decompressed size of signalling WebSocket messages
conn.SetReadLimit only accounts for the compressed bytes read off the
wire, and the client-facing /rtc upgrader negotiates permessage-deflate,
so a small compressed frame could still expand into a much larger buffer
once inflated. Enforce the same limit on the decompressed message by
reading through NextReader + io.LimitReader in WSSignalConnection instead
of the unbounded ReadMessage.
The transport-level SetReadLimit is kept as the cheap wire-level guard;
the new check is the decompressed-size backstop.
Adds NextReader to the WebsocketClient interface (regenerated fake) and a
unit test plus permessage-deflate integration tests covering the
amplification case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add per-participant concurrent TURN allocation quota
The embedded TURN server authenticated each Allocate request but placed
no cap on how many relay allocations a single participant credential
could hold. One participant could reuse its credential across many client
5-tuples and open one relay socket/port per request, exhausting the
shared relay-port range for everyone else.
Add a configurable per-participant limit (turn.per_user_relay_allocation_limit,
default 4) wired to Pion's QuotaHandler, keyed by the participant ID from
HandleAuth. Slots are reserved before allocation and released when the
allocation ends, under a single lock, so concurrent Allocate bursts cannot
race past the limit; reservations are keyed by source address so retransmits
are idempotent. Over-quota requests receive 486 (Allocation Quota Reached).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reclaim unconfirmed TURN allocation reservations
Allow reserved a quota slot before Pion built the relay, but the slot was
only released on the allocation-deleted event. An Allocate that passed the
quota check and then failed to create a relay (e.g. relay-port range
exhausted) emits no event, so the reservation leaked: after enough failures
a participant could lock itself out with 486, and the tracking map grew
without bound.
Reservations now start pending and are confirmed on allocation-created; each
pending reservation carries a reclaim timer that frees the slot after a TTL,
so a failed attempt cannot hold a slot forever while concurrent-burst safety
is preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make TURN reservation reclaim identity-aware
The reclaim timer captured only userID+key. Because Timer.Stop cannot cancel
a callback that has already fired and is waiting on the lock, a stale timer
could delete a replacement reservation created for the same userID+key after
the original was released, leaving a live allocation untracked and letting the
participant exceed its cap.
reclaimPending now captures the slot pointer and only removes the entry when
the map still holds that exact slot, so a stale timer is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* Pin protocol for schema metadata
* Pass through schema and frame encoding
* Support custom encoding identifiers
* Rename config key
* Increase default length to 32
* Make log messages more generic
* Use getters with built-in null check
* Do not bump deps
* Rename function
* Use protocol v1.48.1 release
---------
Co-authored-by: boks1971 <raja.gobi@tutanota.com>
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* deps
* atomic
* sctp at 1.9.5
* remove proto clone
---------
Co-authored-by: Jacob Gelman <3182119+ladvoc@users.noreply.github.com>
noticed a config in deploy config while cleaning up some other usused
config. small clean up. probably there is a bunch more that can be
cleaned up, but doing a quick one as I noticed this.
* service: enforce metadata size limit in CreateRoom, bump default to 512 KiB
CreateRoom previously accepted any metadata size; only UpdateRoomMetadata
rejected oversized payloads. Mirror the same CheckMetadataSize check at
the CreateRoom API boundary so both entrypoints are bounded.
Default MaxMetadataSize moves from 64000 to 512 * 1024 to match the
practical needs of customers using room metadata for richer state. The
limit remains configurable via the existing limits.max_metadata_size knob.
* service: split room vs. participant metadata limit, enforce on join + agent dispatch
LimitConfig.MaxMetadataSize was shared between room metadata and
participant metadata. Last commit's bump to 512 KiB lifted both ceilings;
this restores the participant ceiling to 64 KB and introduces a separate
MaxRoomMetadataSize (default 512 KiB) for room metadata.
Additional enforcement:
- RoomManager.StartSession rejects joins whose JWT-grants metadata or
attributes exceed the participant/attributes limits. The check was
missing entirely from this path.
- AgentDispatchService.CreateDispatch and the embedded
CreateRoomRequest.Agents path now validate metadata and attributes
against the common 64 KB ceilings (previously unbounded).
NewAgentDispatchService gains a LimitConfig parameter; the two wire_gen
callsites are updated.
* service: collapse metadata size limit to single 512 KiB knob
Reverts the LimitConfig split introduced in the previous commit:
MaxRoomMetadataSize, CheckRoomMetadataSize, and the max_room_metadata_size
yaml key are removed. MaxMetadataSize moves back to 512 * 1024 and gates
all metadata uniformly — room (CreateRoom, UpdateRoomMetadata), participant
(UpdateParticipant, signal UpdateMetadata, JWT grants on join), and agent
dispatch (CreateDispatch + embedded RoomAgentDispatch).
MaxAttributesSize stays at 64 KB and continues to gate participant and
agent-dispatch attributes separately.
Test cases consolidated under the single knob.
* kb -> kib
* Add TURN permission handler.
- Turn off permissions to private/link local/multicast and internal IPs
- Add a list of CIDRs that can be used for more things to deny
permission to.
* unused
* add config for allowing private IPs, used in testing
* add a TTL to user name and use it to auth
* allow list for restricted peer CIDRs
* Add support for RTP stream restart.
When an unhandled packet is encountered, try a restart sequence.
Restart happens when 5 packets with contiguous sequence numbers and same
or increasing time stamps are received. Note that this does not work for
B-frame type of scenarios, but that is true for receive path handling
even before this. As WebRTC does not use B-frames, it is fine. But,
needs to be looked at again if B-frames are necessary.
It is controlled by a config that is disabled by default.
* clean up
* debug log
* Added optional "Power of Two Random Choices" algorithm for the node selector sort_by feature. The current, default behavior of picking the lowest-valued node remains.
* WIP
* check using protocol version
* revert
* clean up
* sdp cid argument
* WIP
* WIP
* test
* clean up
* clean up
* fixes
* clean up
* clean up
* clean up
* conditional checks
* tests for both dual and single peer connection
* test
* test
* test
* type check
* test
* todo
* munges
* combined config
* populate mid
* limit to receive only
* clean up
* clean up
* clean up
* older test
* clean up
* alternative audio codec
* dtx
* don't need to copy
* Anunay feedback
* use the available peer connection
* publisher check
* WIP
* WIP
* WIP
* no mid
* media sections requirement
* mage generate
* WIP
* WIP
* set data channel receive size for test
* handle early media better
* WIP
* do not do ICERestart if no subscriber
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* start up subscriber RTCP worker
* WIP
* WIP
* clean up
* clean up
* flag to indicate use of single peer connection
* remove unused interface method
* clean up
* clean up
* Jie feedback #1
* deps
* do not access subscriber in one shot mode
* more places for one shot mode
* more one shot fixes
* deps
* deps
* test
* Rework node stats a bit.
Related protocol PR - https://github.com/livekit/protocol/pull/1023
- Make a config for node stats measurements. Wanted to put the config in
`routing` package, but a circular dependency forced me to put in
config.go
- Make rate calculations explicit, i. e. requested via config.
Previously, it had some odd checks to decide when to calculate rate
and it would have been calculating over different windows.
- Report signal/data channel bytes every 5 seconds to stats collection
module. Previously, it was doing it every 30 seconds and that meant
some windows could have had a large spike
NOTE: Still need to think about this for load calculations as a large
number of participants leaving could flush in a small window and that
could report a large spike in bytes/packets. Maybe need to ignore
signal bytes for load calculation?
* deps
* use default node stats config if given config is nil
* split out node stats into a struct for re-use
* update config
Seeing a lot of queuing delay based back offs. Trying a couple of things
1. Accept a bit more queuing.
2. An option to try a different pacer. Would like to try with pass
through. That will produce some out-of-order packets. Remains to be
seen if it will have a negative impact.
* Normalize mime type and add utilities.
An attempt to normalize mime type and avoid string compares remembering
to do case insensitive search.
Not the best solution. Open to ideas. But, define our own mime types
(just in case Pion changes things and Pion also does not have red mime
type defined which should be easy to add though) and tried to use it everywhere.
But, as we get a bunch of callbacks and info from Pion, needed conversion in
more places than I anticipated. And also makes it necessary to carry
that cognitive load of what comes from Pion and needing to process it
properly.
* more locations
* test
* Paul feedback
* MimeType type
* more consolidation
* Remove unused
* test
* test
* mime type as int
* use string method
* Pass error details and timeouts. (#3402)
* go mod tidy (#3408)
* Rename CHANGELOG to CHANGELOG.md (#3391)
Enables markdown features in this otherwise already markdown'ish formatted document
* Update config.go to properly process bool env vars (#3382)
Fixes issue https://github.com/livekit/livekit/issues/3381
* fix(deps): update go deps (#3341)
Generated by renovateBot
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Use a Twirp server hook to send API call details to telemetry. (#3401)
* Use a Twirp server hook to send API call details to telemetry.
* mage generate and clean up
* Add project_id
* deps
* - Redact requests
- Do not store responses
- Extract top level fields room_name, room_id, participant_identity,
participant_id, track_id as appropriate
- Store status as int
* deps
* Update pkg/sfu/mime/mimetype.go
* Fix prefer codec test
* handle down track mime changes
---------
Co-authored-by: Denys Smirnov <dennwc@pm.me>
Co-authored-by: Philzen <Philzen@users.noreply.github.com>
Co-authored-by: Pablo Fuente Pérez <pablofuenteperez@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: cnderrauber <zengjie9004@gmail.com>
* Thottle the publisher data channel sending when subscriber is slow
Avoid the publisher overwhelm the sfu data channel buffer when
the subscriber has lower receive bitrates. It will drop message
if the subscriber is considered too slow to block the entire room.
* Enable nack in mediaengine and disable it in transceiver as need
pion doesn't support per transciver codec configuration, so the nack of this session will be disabled
forever once it is first disabled by a transceiver.
https://github.com/pion/webrtc/pull/2972
