refreshToken copied grants field by field and dropped observability, so a
client that adopted the first refresh (sent at join) lost the grant within
a second, and one that kept its join token lost it at expiry. Keep the join
token's grant on every refresh; never add one.
Pick up protocol#1879 (round-trip 7s, default response 15s) so a
server-initiated PerformRpc tolerates the same ack and response windows
as the client SDK caller.
Also log an RPC response the v1 caller cannot decode (compressed_payload
or data-stream) instead of dropping it, which otherwise surfaces only as
a response timeout.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
TestDownTrackResumesAfterBandwidthPause called logger.InitFromConfig
(SetLogger) mid-run while the hwstats CPU monitor goroutine, started by
prometheus.Init in forwardstats_test, read the same non-atomic logger
package globals. go test -race flagged the write-vs-read and failed CI.
Set the global logger once in TestMain, before any test starts that
goroutine, and drop the per-test InitFromConfig.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Echo offered video payload types in single peer connection answers
restrictReceiverCodecsToPublishList pinned server payload types on a video
sender that had no media section yet. The later answer then ignored the
payload types the client offered, and Firefox never decoded the track.
Skip transceivers with an empty mid. Fixes#5030.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Shorten the mid guard comment
Point at sendersPendingConfig, which already documents the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
turn_servers already accepts secret_file for the dynamic (static-auth-secret)
mechanism, but the static username/credential pair could only be written inline
in the config, forcing deployments that mount secrets as files to inline them.
Add username_file and credential_file, resolved through a single helper shared
with secret_file: the inline value wins when both are set, file contents are
trimmed, an empty file fails startup, and credential_file must not be readable
by others. username_file carries no permission requirement because the username
is not a secret.
Co-authored-by: Ora Agent <noreply@oratelecom.net>
* Use negotiated data channels in single PC mode
Bump the protocol version to 18. For protocol >= 18 single peer connection
clients, the server creates its publisher data channels as negotiated with
fixed IDs (100/101/102), so it can send on them without waiting for the
client's DCEP OPEN. Dual PC keeps the in-band channels.
* solve comment
* sfu: reset dummy start offset on forwarder restart
Restart() kept dummyStartTSOffset. The next source switch recomputed it
from a refTS that already included the old offset, so every switch after
that was off by the old value and the subscriber's timestamp jumped.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: log extTimestamp in sequencer packet meta, add Chrome case to AV1 config test
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: do not apply dummy start offset without a reference layer
With skipReferenceTS, refTS is the last outgoing timestamp, which is
already on the downtrack's timeline. Computing and adding an offset there
re-added the gap between the last packet and a restart on every later
layer switch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
processSourceSwitch added dummyStartTSOffset to extRefTS in 64 bits. When
the offset is negative and the reference layer's 32-bit RTP timestamp has
wrapped but is still below |offset|, the addition borrows from the upper
half and extRefTS ends up 2^32 below the right value. The wrap correction
that follows only compares the lower 32 bits, so it cannot undo it.
The switch point then looks ~47,722 s (2^32 / 90 kHz) behind and is
rejected with errSwitchPointTooFarBehind on every keyframe, until the
reference layer's timestamp climbs past |offset|, which can take hours.
Apply the offset modulo 2^32 before restoring the cycles from extLastTS.
The lower 32 bits, and so the timestamp sent to subscribers, are unchanged.
RED depacketizing now lives in mediatransportutil so the SFU and
replay export share one implementation. Drop the local copies; the
encoder round-trip tests now exercise the shared package.
Callers passed mono.Now() as a time.Time, which made it easy to pass
time.Now() by mistake and mix clocks. Take mono nanoseconds instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
LastSenderReportTime() returned time.Unix(0, srNewest.At). But At is a
mono timestamp, so the result is not a real wall time. Connection quality
compares it to time.Now() to decide if RTCP is stale. If the wall clock
drifts from mono, for example after an NTP step, streams can look stale
when they are not, or the reverse.
- Convert At to wall time: now minus the mono time since the report.
- Return zero time when At is 0, so the scorer skips the stale check
(before, it got the 1970 epoch and saw the stream as stale).
Follow-up to #5005, which fixed the same mix-up in DLSR.
Tests: rtpstats and connectionquality pass. No new test, because a
test process cannot make the mono and wall clocks drift.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: apply simulcast rids from SDP for WHIP (one-shot) publishers
In one-shot signalling mode HandleRemoteDescription set the remote offer
directly and never called OnSetRemoteDescriptionOffer, so
updateRidsFromSDP did not run for WHIP participants. Every simulcast layer
then mapped to spatial layer 0 and all but the first were rejected with
'duplicate layer'.
* rtc: test that a one-shot simulcast offer applies rids to the pending track
* rtc: use h/q rids in one-shot simulcast test, cover single-layer offer
Name the simulcast test layers medium/h and low/q per convention, and add
TestOneShotSingleLayerOfferUnchanged to check a one-shot offer without rids
still yields a single layer with rid "" at spatial layer 0.
* agent: fix flaky TestAgent job dispatch
Retry JobRequest until the worker subscribes the job topic, because
registration is async and the topic is subscribed after the register response.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* agent: report last JobRequest error on dispatch timeout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: reclaim abandoned pending tracks
A pending track is freed only when its first media arrives or the
participant leaves. A client that calls AddTrack and then gives up
before any media flows leaves the slot held for the whole session.
After 20 such slots, every later AddTrack is rejected.
Reclaim these slots when the cap is hit, if no media arrived and the
slot is older than the publish timeout. Keep the supervisor monitor in
step for queued tracks that share a track id. Add a publish_timeout
config value, default 30s.
Fixes: #4970
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: fix publish_timeout comment to not name an internal constant
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* rtc: release supervisor-timed-out pending tracks at the limit
Replace the createdAt sweep with the supervisor's publish timeout. The
supervisor counts time only while the publisher is connected and the
track is unmuted, and never reports a live track, so a successor queued
behind a live track is not released.
Restart the publish timer when a queued publish reaches the front of
the queue, so it does not inherit the clock of the track ahead of it.
Write the publication error under the write lock, since it is now read
from another goroutine.
Drop the publish_timeout config value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: skip stale supervisor timeouts and keep live monitors
TimedOutPublications re-checks the monitor, because the cached error is
cleared only on the next tick. A track that published since is not
reported, so its queued successor is not released.
RemovePublication keeps a monitor that still holds a published track.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: drop late media for released pending tracks
Media or an offer with no matching pending track falls back to the first
pending track of the same type. After a release, late media for the
released track could take another pending track's identity.
Record the released track's signal and SDP client ids, the 32 most
recent, and drop media or offers for them. An AddTrack that reuses the
client id clears it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rtc: clear all ids of a released track when it is re-added
A re-add cleared only the signal client id, so a Firefox republish that
keeps the old SDP id had its offer and media dropped. Keep the ids of
each released track together, and clear the whole track when a new
AddTrack reuses its signal or simulcast codec client id. The limit now
counts released tracks, and a track ages out with all of its ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
An unnamed deployment was addressed as "default" and that spelling leaked
into scope keys, so embedders keyed endpoints differently from job dispatch.
Map "_" back to "" at the URL boundary, as the agent name already does,
and drop NormalizeDeployment.
LoadIngress reads the ingress info and its state from two separate Redis
keys non-atomically, while DeleteIngress removes both keys in a single
transaction. If the delete lands between the two reads, LoadIngress
returns an IngressInfo with a nil State and no error, and
UpdateIngressState panics dereferencing info.State.Status — taking down
the whole server with every room it hosts.
The race fires in practice when an ingress session is being torn down by
DeleteIngress and the ingress service reports the session state change
concurrently: with short-lived per-room WHIP ingresses this crashed
production servers within minutes (observed on v1.13.5 and v1.13.6,
SIGSEGV at ioservice_ingress.go:82, addr=0x8).
Guard the nil State by treating it as an empty state, matching how the
store's UpdateIngressState already tolerates a missing state key.
* turn: refuse TCP relays and limit relays to this host to media ports
Embedded TURN let any participant relay to arbitrary peers (#4924):
- RFC 6062 TCP relays, accepted since pion/turn v5, let a client open TCP
connections to any permitted address, e.g. SSH on the node itself.
WebRTC clients never request them, so they are now refused for both
embedded and standalone TURN.
- Relays to the host's own addresses are delivered locally and bypass any
firewall in front of the host. Embedded TURN now only relays to this
host's RTC media UDP ports.
- Carrier-grade NAT, NAT64 and a few other special-purpose ranges are
added to the restricted peer ranges.
The issue also suggested allowing relays only to the media node's IPs.
That is not done here because, with the changes above, the remaining
relays do not form a security concern: they are UDP only, to public
addresses outside the restricted ranges, and reach a peer exactly as the
client could from its own address (no amplification, no spoofed source).
This is standard TURN behavior. Limiting destinations to media nodes would
also require TURN server to know media node all endpoints, for little benefit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* solve comment
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP9 and AV1 simulcast temporal layers with the dependency descriptor
Every simulcast codec gets an explicit temporal layer selector. With the
dependency descriptor, VP8, VP9 and AV1 filter temporal layers by frame and
forward the descriptor. Without it, a pass-through selector forwards every
temporal layer and reports the target as reached, so a pending target cannot
block allocation. VP8DD becomes the codec-neutral DDTemporal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: VP8-only padding header in DDTemporal, structure gate on presence
DDTemporal returns the one byte VP8 payload descriptor for padding only for
VP8. The descriptor forwarder takes a structure from any packet that carries
one, as the parser accepts it only on the first packet of a frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers across migration
The forwarder state carries the last dependency descriptor frame number
sent, so a migrated subscriber's first key frame continues the numbers
instead of being dropped as older than its last structure. Bumps protocol
for RTPForwarderState.dependency_descriptor_forwarder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: pass on a pending descriptor seed when migrating again
A forwarder seeded before its codec was known returns the pending seed in
its state, so a second migration keeps the frame numbers continuous.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: forward the dependency descriptor for VP8 simulcast
libwebrtc sends VP8 with a one byte payload descriptor when it sends the
dependency descriptor, so a VP8 subscriber gets neither picture IDs nor the
descriptor and finds frame references by sequence number alone. Under loss
with temporal filtering it stalls until a key frame.
DDForwarder rewrites the publisher's descriptor for each forwarded packet:
frame numbers continue across simulcast layer switches and stream restarts,
the structure goes out with the current layer's key frame, and the active
decode targets follow the forwarded temporal layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: keep VP8 descriptor frame numbers from going back
Rebase frame numbers only when a new stream would go back past the last
number sent: on a layer switch, a stream restart (also when it starts with a
delta frame) and a new forwarder on the same downtrack, which now carries
over the last number sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: set VP8 descriptor decode targets from the packet's layer
The active decode targets use the temporal layer the packet is forwarded at,
not the selector's next layer on the last packet before a down switch. They
also go out on key frames, combined with the publisher's own mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: carry the current active decode targets on every DD packet
The parser puts the stream's current mask on every packet, so a mask change
on a dropped packet still reaches the VP8 descriptor forwarder, which no
longer tracks the publisher's mask itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: take the first DD active decode targets mask at any sequence number
A publisher whose first packet has RTP sequence number 0 never had its key
frame's mask applied, so every packet reported no active decode targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A filtered packet that follows a sequence number gap now keeps its slot
and gets padding in it. Rolling it back moved the last outgoing number
below the missing packets' slots, so their retransmissions were rejected
and the subscriber stalled on the hole until a key frame.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: parse VP8 and VP9 payload descriptors without an allocation
Replace ExtPacket.Payload with typed VP8 and VP9 pointer fields that point
at backing structs inside the ExtPacket, so parsing does not allocate and a
wrong codec type is a compile error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: copy the VP8 descriptor in the test packet helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The key frame requester re-armed its timer on every loop, so every video down
track woke up every 200 ms to 1 s for its whole life, even when the layer was
locked and there was nothing to do.
Re-arm the timer only while the requested layer is not locked, or while the
lock can change without an event: an acquisition grace is armed, or layers are
selected with the dependency descriptor, which can lose the lock on the packet
path. Post the event on resync, mute, max published layer change and packet
path layer switch, since those can change the lock without an allocation.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On a gap the RTP munger gives the missing sequence numbers outgoing slots.
If a late packet then arrives and is dropped, nothing is ever sent in its
slot, and a subscriber that finds frame references by sequence number
stalls until a key frame. The forwarder now flags such drops, and the
downtrack sends a one byte padding only packet in the slot.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: filter VP8 temporal layers by frame when the dependency descriptor is sent
With the dependency descriptor, libwebrtc sends a one byte VP8 payload
descriptor without a picture ID, so the VP8 munger's picture ID based
temporal drop logic stops filtering after the first gap. A new VP8DD codec
munger decides once per frame from the descriptor's frame number and
temporal ID and applies that decision to every packet of the frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* sfu: use the VP8 munger when picture IDs come with the dependency descriptor
VP8DD needs the one byte VP8 payload descriptor that libwebrtc sends with
the dependency descriptor. If a publisher keeps picture IDs next to the
descriptor, they need rewriting, so the forwarder switches to the VP8
munger and keeps switching temporal layers with the descriptor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GetRecentlyDisconnectedParticipants walked the participant update cache
with Get, which moves each entry to the front of the LRU. Walking from
newest to oldest left the cache in reverse order, so after a reconnect
the next updates evicted the most recently updated participants first.
In a room where the subscriber has tracked 128 or more participants, a
participant that had just disconnected was then dropped from the cache
and a later reconnect did not report it as disconnected, and the
out-of-order version check was lost for recently updated participants.
Use Peek, which reads an entry without changing its recency.
