With auto_create on, CreateDispatch creates the room through router.CreateRoom without checking limits.max_room_name_length, so it could create a room that CreateRoom and the join paths reject.
* sfu: replace forward scheduling delay with processing per subscriber
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: describe processing per subscriber as per write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* telemetry: name forward processing metrics per write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pin protocol with NodeStats forward_processing_per_write
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- pkg/service/redisstore.go: after the retry loop, return an error wrapping redis.TxFailedErr instead of nil. CreateIngress now fails instead of returning an ingress that was never written, StoreIngress stops before creating the orphan state key, and a dropped UpdateIngressState reaches the caller's existing error log.
- pkg/service/ingress.go: the Fix nil pointer dereference in IOInfoService.UpdateIngressState on concurrent DeleteIngress #4806 guard applied to the two Twirp handlers, treating a nil State as empty. UpdateIngress then applies and stores the request; DeleteIngress deletes without calling the ingress service.
- pkg/service/redisstore.go: UpdateIngress now fails with ErrIngressNotFound when the ingress no longer exists, checked inside the WATCH on the ingress hash so a later delete fails the EXEC. Without it an update racing a DeleteIngress recreated the deleted ingress and its stream key; this was possible before this PR whenever the delete landed after LoadIngress, and the guard in 2 widened the window to the gap between its two reads.
The guard stays at the call sites to match #4806 and keep API responses unchanged.
* chore: add AttachJob interface on Worker
allows for a worker to take ownership of an existing job
* ensure we can handle stateless jobs
* set initial state in AssignJob
Servers that embed pkg/service and run their own join paths need the
same join limit check. Rename checkConnectLimits to CheckConnectLimits.
No behavior change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* service: enforce participant name length on join
StartSession checked the token's metadata and attributes against the configured limits but not its name, so a participant could join with a name longer than limits.max_participant_name_length, which UpdateParticipant and the signal UpdateMetadata path reject.
* service: consolidate join grant-limit checks on the validate path
Extract checkGrantLimits and run it from ValidateConnectRequest and
WHIPService.validateCreate so the /rtc/validate endpoint and the WHIP
ingest path reject over-limit name, metadata and attributes the same way
a real join does. StartSession keeps its own checks as the backstop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* service: run the full join limit check in StartSession
StartSession now calls checkConnectLimits, so the RTC node checks
identity, room name, participant name, metadata and attributes itself.
ValidateConnectRequest runs the check after the publish suffix, so it
measures the same identity StartSession sees. checkConnectLimits returns
a plain error, and the HTTP callers map it to 400. The name-only
StartSession test is dropped for TestConnectLimits, which gains a
publish-suffix case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* service: test the StartSession limit check directly
TestStartSessionConnectLimits runs the same over-limit cases as
TestConnectLimits through StartSession on the RTC node. The cases move
into an exported ConnectLimitCases table so both test packages share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: ANDROIDHASSAN <84630851+ANDROIDHASSAN@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* sfu: report forward scheduling delay next to forwarding latency
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep full forwarding transit when packing scheduling delay
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: saturate reported forwarding p90s instead of wrapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The WHIP create handler never called ValidateCreateRoom, so with room.auto_create: false a publisher token without roomCreate could create any room over WHIP, while the same join over /rtc gets 404.
* sfu: cap retransmissions per down track (rtc.rtx config, off by default)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: rtx budget review fixes (sequencer undo, sent bytes, newest first, rtx_budget)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: rtx budget second review (order unchanged when off, keep NACK window, metric in down track)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: rtx budget sorts NACKs newest first by extended sequence number
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: wait for the recorded data track subscription before revoking in test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: drop a subscription whose canSubscribe was revoked while it was in flight
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Thread the issuing API key from the /rtc join path to the media node via
StartSession, and include it in the "starting RTC session" log and the
twirp API-request log, so participant activity can be correlated with the
key. Logs the key id only, never the secret.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: measure subscribe pacing delay in shadow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* prometheus: raise the subscribe pace shadow top bucket to 120 s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: pace video subscribes per node in shadow, record the wait only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: subscribe pacer doc wording, make the wiring test independent of timing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: sample shadow pacing at arrival, skip one-shot, keep sub-millisecond waits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: take the shadow pacing token before setup, in arrival order
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: build shadow pacing buckets in one pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add upstream FlexFEC recovery support
* Guard short TWCC extensions in FEC packets
* Release completed FlexFEC states
* Validate FlexFEC payload type range
* Invoke FEC callbacks after unlocking
* Reduce FlexFEC protection state allocations
* Avoid allocations for complete FlexFEC packets
* Optimize ordered FlexFEC packet insertion
* Reuse RTP packet storage for FlexFEC recovery
* Limit pending FlexFEC repair packets
* Lower FlexFEC repair window to two seconds
* Harden upstream FlexFEC handling
* Test FlexFEC with E2EE media
* Fix FlexFEC negotiation with libwebrtc publishers
* Enhance payload length validation in decoder
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* remove the ability to configure payload type for FlexFEC
* moved some helpers into protocol
* Handle big gaps in media sequence numbers
Reset buffers when a big gap in media sequence numbers is detected.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* bump protocol & psrpc deps
* Handle FEC packet recovery failure
Increment FEC packets discarded counter and remove packet.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* do not NACK on fec recovered packet (#4896)
* do not NACK on fec recovered packet
* fix test race in CI
* Fix FEC write return after merging RTX feedback changes
* PR fixes for FlexFEC implementation (#4898)
* do not NACK on fec recovered packet
* refactor common code into unlockAndInvokeFECRecovery
* add FEC stats to RTPStats
* add log for when fecdecoder unavailable after creation
* rename markAsFECRepair to setFECRepairBufferLimit for clarity
* refactor packet generation for tests into util
* fix test race in CI
* move the unlocks back inline for readability
* update RTPstats
* Fix FEC write return after merging RTX feedback changes
* Use shared RTP packet helper in merged FEC NACK test
* Remove redundant return from FEC write handler
* packet post process (#5020)
* packet post process
* remove redundant writeRTX
* send out RTCP (NACK and RR) from packetPostProcess only
* handle FEC decoder on receiver restart (#5032)
* handle FEC decoder on receiver restart
* move onStreamRestartLocked into BufferBaseParams
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Raja Subramanian <raja.gobi@tutanota.com>
Pick up protocol#1879 (round-trip 7s, default response 15s) so a
server-initiated PerformRpc tolerates the same ack and response windows
as the client SDK caller.
Also log an RPC response the v1 caller cannot decode (compressed_payload
or data-stream) instead of dropping it, which otherwise surfaces only as
a response timeout.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
TestDownTrackResumesAfterBandwidthPause called logger.InitFromConfig
(SetLogger) mid-run while the hwstats CPU monitor goroutine, started by
prometheus.Init in forwardstats_test, read the same non-atomic logger
package globals. go test -race flagged the write-vs-read and failed CI.
Set the global logger once in TestMain, before any test starts that
goroutine, and drop the per-test InitFromConfig.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Echo offered video payload types in single peer connection answers
restrictReceiverCodecsToPublishList pinned server payload types on a video
sender that had no media section yet. The later answer then ignored the
payload types the client offered, and Firefox never decoded the track.
Skip transceivers with an empty mid. Fixes#5030.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Shorten the mid guard comment
Point at sendersPendingConfig, which already documents the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
turn_servers already accepts secret_file for the dynamic (static-auth-secret)
mechanism, but the static username/credential pair could only be written inline
in the config, forcing deployments that mount secrets as files to inline them.
Add username_file and credential_file, resolved through a single helper shared
with secret_file: the inline value wins when both are set, file contents are
trimmed, an empty file fails startup, and credential_file must not be readable
by others. username_file carries no permission requirement because the username
is not a secret.
Co-authored-by: Ora Agent <noreply@oratelecom.net>
* Use negotiated data channels in single PC mode
Bump the protocol version to 18. For protocol >= 18 single peer connection
clients, the server creates its publisher data channels as negotiated with
fixed IDs (100/101/102), so it can send on them without waiting for the
client's DCEP OPEN. Dual PC keeps the in-band channels.
* solve comment
* sfu: reset dummy start offset on forwarder restart
Restart() kept dummyStartTSOffset. The next source switch recomputed it
from a refTS that already included the old offset, so every switch after
that was off by the old value and the subscriber's timestamp jumped.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: log extTimestamp in sequencer packet meta, add Chrome case to AV1 config test
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: do not apply dummy start offset without a reference layer
With skipReferenceTS, refTS is the last outgoing timestamp, which is
already on the downtrack's timeline. Computing and adding an offset there
re-added the gap between the last packet and a restart on every later
layer switch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
processSourceSwitch added dummyStartTSOffset to extRefTS in 64 bits. When
the offset is negative and the reference layer's 32-bit RTP timestamp has
wrapped but is still below |offset|, the addition borrows from the upper
half and extRefTS ends up 2^32 below the right value. The wrap correction
that follows only compares the lower 32 bits, so it cannot undo it.
The switch point then looks ~47,722 s (2^32 / 90 kHz) behind and is
rejected with errSwitchPointTooFarBehind on every keyframe, until the
reference layer's timestamp climbs past |offset|, which can take hours.
Apply the offset modulo 2^32 before restoring the cycles from extLastTS.
The lower 32 bits, and so the timestamp sent to subscribers, are unchanged.
RED depacketizing now lives in mediatransportutil so the SFU and
replay export share one implementation. Drop the local copies; the
encoder round-trip tests now exercise the shared package.
Callers passed mono.Now() as a time.Time, which made it easy to pass
time.Now() by mistake and mix clocks. Take mono nanoseconds instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
LastSenderReportTime() returned time.Unix(0, srNewest.At). But At is a
mono timestamp, so the result is not a real wall time. Connection quality
compares it to time.Now() to decide if RTCP is stale. If the wall clock
drifts from mono, for example after an NTP step, streams can look stale
when they are not, or the reverse.
- Convert At to wall time: now minus the mono time since the report.
- Return zero time when At is 0, so the scorer skips the stale check
(before, it got the 1970 epoch and saw the stream as stale).
Follow-up to #5005, which fixed the same mix-up in DLSR.
Tests: rtpstats and connectionquality pass. No new test, because a
test process cannot make the mono and wall clocks drift.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: apply simulcast rids from SDP for WHIP (one-shot) publishers
In one-shot signalling mode HandleRemoteDescription set the remote offer
directly and never called OnSetRemoteDescriptionOffer, so
updateRidsFromSDP did not run for WHIP participants. Every simulcast layer
then mapped to spatial layer 0 and all but the first were rejected with
'duplicate layer'.
* rtc: test that a one-shot simulcast offer applies rids to the pending track
* rtc: use h/q rids in one-shot simulcast test, cover single-layer offer
Name the simulcast test layers medium/h and low/q per convention, and add
TestOneShotSingleLayerOfferUnchanged to check a one-shot offer without rids
still yields a single layer with rid "" at spatial layer 0.
* agent: fix flaky TestAgent job dispatch
Retry JobRequest until the worker subscribes the job topic, because
registration is async and the topic is subscribed after the register response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* agent: report last JobRequest error on dispatch timeout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: reclaim abandoned pending tracks
A pending track is freed only when its first media arrives or the
participant leaves. A client that calls AddTrack and then gives up
before any media flows leaves the slot held for the whole session.
After 20 such slots, every later AddTrack is rejected.
Reclaim these slots when the cap is hit, if no media arrived and the
slot is older than the publish timeout. Keep the supervisor monitor in
step for queued tracks that share a track id. Add a publish_timeout
config value, default 30s.
Fixes: #4970
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: fix publish_timeout comment to not name an internal constant
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: release supervisor-timed-out pending tracks at the limit
Replace the createdAt sweep with the supervisor's publish timeout. The
supervisor counts time only while the publisher is connected and the
track is unmuted, and never reports a live track, so a successor queued
behind a live track is not released.
Restart the publish timer when a queued publish reaches the front of
the queue, so it does not inherit the clock of the track ahead of it.
Write the publication error under the write lock, since it is now read
from another goroutine.
Drop the publish_timeout config value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: skip stale supervisor timeouts and keep live monitors
TimedOutPublications re-checks the monitor, because the cached error is
cleared only on the next tick. A track that published since is not
reported, so its queued successor is not released.
RemovePublication keeps a monitor that still holds a published track.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: drop late media for released pending tracks
Media or an offer with no matching pending track falls back to the first
pending track of the same type. After a release, late media for the
released track could take another pending track's identity.
Record the released track's signal and SDP client ids, the 32 most
recent, and drop media or offers for them. An AddTrack that reuses the
client id clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: clear all ids of a released track when it is re-added
A re-add cleared only the signal client id, so a Firefox republish that
keeps the old SDP id had its offer and media dropped. Keep the ids of
each released track together, and clear the whole track when a new
AddTrack reuses its signal or simulcast codec client id. The limit now
counts released tracks, and a track ages out with all of its ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
An unnamed deployment was addressed as "default" and that spelling leaked
into scope keys, so embedders keyed endpoints differently from job dispatch.
Map "_" back to "" at the URL boundary, as the agent name already does,
and drop NormalizeDeployment.
LoadIngress reads the ingress info and its state from two separate Redis
keys non-atomically, while DeleteIngress removes both keys in a single
transaction. If the delete lands between the two reads, LoadIngress
returns an IngressInfo with a nil State and no error, and
UpdateIngressState panics dereferencing info.State.Status — taking down
the whole server with every room it hosts.
The race fires in practice when an ingress session is being torn down by
DeleteIngress and the ingress service reports the session state change
concurrently: with short-lived per-room WHIP ingresses this crashed
production servers within minutes (observed on v1.13.5 and v1.13.6,
SIGSEGV at ioservice_ingress.go:82, addr=0x8).
Guard the nil State by treating it as an empty state, matching how the
store's UpdateIngressState already tolerates a missing state key.
* turn: refuse TCP relays and limit relays to this host to media ports
Embedded TURN let any participant relay to arbitrary peers (#4924):
- RFC 6062 TCP relays, accepted since pion/turn v5, let a client open TCP
connections to any permitted address, e.g. SSH on the node itself.
WebRTC clients never request them, so they are now refused for both
embedded and standalone TURN.
- Relays to the host's own addresses are delivered locally and bypass any
firewall in front of the host. Embedded TURN now only relays to this
host's RTC media UDP ports.
- Carrier-grade NAT, NAT64 and a few other special-purpose ranges are
added to the restricted peer ranges.
The issue also suggested allowing relays only to the media node's IPs.
That is not done here because, with the changes above, the remaining
relays do not form a security concern: they are UDP only, to public
addresses outside the restricted ranges, and reach a peer exactly as the
client could from its own address (no amplification, no spoofed source).
This is standard TURN behavior. Limiting destinations to media nodes would
also require TURN server to know media node all endpoints, for little benefit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* solve comment
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP9 and AV1 simulcast temporal layers with the dependency descriptor
Every simulcast codec gets an explicit temporal layer selector. With the
dependency descriptor, VP8, VP9 and AV1 filter temporal layers by frame and
forward the descriptor. Without it, a pass-through selector forwards every
temporal layer and reports the target as reached, so a pending target cannot
block allocation. VP8DD becomes the codec-neutral DDTemporal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: VP8-only padding header in DDTemporal, structure gate on presence
DDTemporal returns the one byte VP8 payload descriptor for padding only for
VP8. The descriptor forwarder takes a structure from any packet that carries
one, as the parser accepts it only on the first packet of a frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers across migration
The forwarder state carries the last dependency descriptor frame number
sent, so a migrated subscriber's first key frame continues the numbers
instead of being dropped as older than its last structure. Bumps protocol
for RTPForwarderState.dependency_descriptor_forwarder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: pass on a pending descriptor seed when migrating again
A forwarder seeded before its codec was known returns the pending seed in
its state, so a second migration keeps the frame numbers continuous.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: forward the dependency descriptor for VP8 simulcast
libwebrtc sends VP8 with a one byte payload descriptor when it sends the
dependency descriptor, so a VP8 subscriber gets neither picture IDs nor the
descriptor and finds frame references by sequence number alone. Under loss
with temporal filtering it stalls until a key frame.
DDForwarder rewrites the publisher's descriptor for each forwarded packet:
frame numbers continue across simulcast layer switches and stream restarts,
the structure goes out with the current layer's key frame, and the active
decode targets follow the forwarded temporal layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers from going back
Rebase frame numbers only when a new stream would go back past the last
number sent: on a layer switch, a stream restart (also when it starts with a
delta frame) and a new forwarder on the same downtrack, which now carries
over the last number sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: set VP8 descriptor decode targets from the packet's layer
The active decode targets use the temporal layer the packet is forwarded at,
not the selector's next layer on the last packet before a down switch. They
also go out on key frames, combined with the publisher's own mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: carry the current active decode targets on every DD packet
The parser puts the stream's current mask on every packet, so a mask change
on a dropped packet still reaches the VP8 descriptor forwarder, which no
longer tracks the publisher's mask itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: take the first DD active decode targets mask at any sequence number
A publisher whose first packet has RTP sequence number 0 never had its key
frame's mask applied, so every packet reported no active decode targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A filtered packet that follows a sequence number gap now keeps its slot
and gets padding in it. Rolling it back moved the last outgoing number
below the missing packets' slots, so their retransmissions were rejected
and the subscriber stalled on the hole until a key frame.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: parse VP8 and VP9 payload descriptors without an allocation
Replace ExtPacket.Payload with typed VP8 and VP9 pointer fields that point
at backing structs inside the ExtPacket, so parsing does not allocate and a
wrong codec type is a compile error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: copy the VP8 descriptor in the test packet helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
