- Count a publish attempt on a migrating in tarck as there is no
AddTrack for that.
- Add cancel publish only if the participant connection is canceled
- Do not add publish counter for synthetic publish attempts which
happens for migrating in tracks. It will be counted on migrating in
node when the track is actually published, i. e. negotiated/packets
flowing.
* Do not call telemetry listener under pending track lock.
Fix the TrackPublishRequested call of telemetry listener.
Audited other callbacks to ensure that it is not under lock.
* missed some paths of recording it, thanks Devin
* Record subscribe stream start time in prometheus.
Adjust for mutes, i. e. take the last unmute time as the start point and
calculate time till the first byte is sent.
* close the tiny window of race
* Prevent long tail sample when publisher glitches.
Thanks to @milos-lk for this.
Publisher restarting would have reset the layer and would have caused a
sample with very high stream start time. We only need to capture when we
do a dummy start or when the state is seeded to a different node upon
migration.
* reduce a diff
* test
* changed the wrong thing, thank you Devin
In single-PC and one-shot signalling modes the subscriber PCTransport is
never created, so t.subscriber is nil. Every other TransportManager method
that touches the subscriber nil-checks it first; HandleAnswer did not, so a
client sending an SDP answer in those modes crashed the process with a nil
pointer dereference. Guard it and log, matching AddICECandidate/Negotiate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Was doing it on participant close. But, that misaligns `rtc_init`
(number of attempts) a bit if the participant sessions are long.
Record it as soon as RTC connects so that it is more time aligned with
`rtc_init`.
* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
Leave out canceled attempts. Should make it easier to do percentages.
Not putting these in node stats yet. Will observe in prom before using
it in node stats.
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* Pin protocol for schema metadata
* Pass through schema and frame encoding
* Support custom encoding identifiers
* Rename config key
* Increase default length to 32
* Make log messages more generic
* Use getters with built-in null check
* Do not bump deps
* Rename function
* Use protocol v1.48.1 release
---------
Co-authored-by: boks1971 <raja.gobi@tutanota.com>
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* deps
* atomic
* sctp at 1.9.5
* remove proto clone
---------
Co-authored-by: Jacob Gelman <3182119+ladvoc@users.noreply.github.com>
In single peer connection mode, when the server answers a subscriber's
offer, configureSenderAudio set the sender codec preferences from the
server MediaEngine's payload types. The answer could therefore advertise
Opus on a payload type the offerer never offered (server PT 111 vs
offered PT 109). Chrome tolerates this; Firefox decodes 0 samples
(silence) -- packets are received but never decoded. The forwarded RTP
already uses the offered PT, so only the answer SDP was inconsistent.
This regressed in v1.12.0 once the single-PC MediaEngine became a union
of publish+subscribe codecs.
Parse the remote offer's audio rtpmap and remap the sender audio codec
preferences to echo the offered payload types (RFC 3264 6.1) before
SetCodecPreferences.
Fixes#4599
Co-authored-by: laosun <14806343+cnvipstar@users.noreply.github.com>
Previously it was anchored to participant transitioning to `ACTIVE` if
the add track request happened before that. But, that has a few issues
1.`ACTIVE` is for primary peer connection which could be subscriber peer
connection.
2. `ACTIVE` also include data channel establishment.
Switch to first connected time of publisher peer connection for that to
get a more accurate measure of track publish time.
* feat: acquire requested video layer directly at HIGH quality by default
Two changes that together remove the visible low->high quality ramp for a new
subscriber (both publisher-first and subscriber-first join orders):
1. Default a subscriber's initial video quality to HIGH on bind instead of LOW
for adaptive stream, so the subscribed max layer is the top layer. Adaptive
stream clients can still scale down afterwards based on viewport.
2. On initial layer acquisition the forwarder/selector latch directly onto the
allocator's target (the requested top layer) instead of opportunistically
latching onto the first lower key frame that arrives. A short
initial-acquisition grace aims the target at the requested layer; if it does
not show up in time, the target falls back to the highest layer seen so
acquisition never stalls.
Always on - no configuration flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: gate start-at-desired-quality behind EnableStartAtDesiredQuality flag
Put the "acquire requested video layer directly at HIGH quality" behavior
behind a per-subscriber EnableStartAtDesiredQuality flag (default off, so
the original low->high ramp-up is restored unless enabled).
Plumbed from config.RTC.EnableStartAtDesiredQuality through ParticipantParams
-> SubscribedTrack/DownTrack -> Forwarder -> simulcast selector, gating all
three behavior changes: the HIGH default on bind, the forwarder's
initial-acquisition grace, and the selector's direct-latch-onto-target.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* remove config.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There are several places the participant can drop off after initiating a
connection attempt. Count those places as cancellation including when
participant is closed due to specific reasons.
Cancels should be discounted when determining RTC/ICE connectivity
success/failure percentage.
* agent: thread simulation flag from dispatch to job
Reads simulation from AgentDispatch / RoomAgentDispatch and copies it
onto Job in agent.LaunchJob and the inline room-agent path so workers
see the flag.
Stacked on top of livekit/protocol#1629.
* agent: replace simulation bool with attributes map
Threads the renamed attributes map (was bool simulation) from dispatch
to job and bumps the protocol pseudo-version.
* deps
* rtc: add RestartSessionTimer to re-anchor participant session duration
Exposes ParticipantImpl.RestartSessionTimer so the session timer can be
re-anchored to the actual join time. Duration is only ever emitted once
the participant becomes active, so re-anchoring at join keeps pre-join
wall-clock out of the reported/billed duration. Adds the method to the
LocalParticipant interface (fake regenerated) and a local protocol
replace to pick up SessionTimer.Reset.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* tidy
* update protocol
* report ended at for inactive sessions
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
* Add prom metrics for peer connectino state.
By direction (PUBLISHER vs SUBSCRIBER) and state ("started" ->
"connected"). This gives a way to track peer connections failing to
finish establishment.
The RTC active count can be useful for primary peer connection, but not
for non-primary. This counter can be used to track any and can generally
be used to understand success/failure rate of peer connection
establishment.
* add a couple of more states
* clean up and avoid duplicate reporting fully established
* staticcheck
MoveToRoom resets the participant reporter resolver to receive new
(room, participant_session) keys for the destination, but the source
room's participant_session row never gets an end_time — the periodic
duration scrape only emits one once disconnectedAt is set, and a move
doesn't transition the participant to DISCONNECTED. Report end_time
immediately before the reset so the row is closed out cleanly.
Data tracks (the new _data_track datachannel) previously only updated a
private dataTrackStats that logged a single summary at Close. Bytes never
reached the OnTrackStats -> TelemetryService.TrackStats pipeline that
media tracks and signal channels feed.
Wire DataTrack (UPSTREAM, publisher-home) and DataDownTrack (DOWNSTREAM,
per-subscriber) into BytesTrackStats on the same 5s cadence, mirroring
the media-track convention: subscriber's country and ID with publisher's
track ID for DOWNSTREAM. Cross-region proxy DataTracks leave the stats
pointer nil (no publisher reporter on that node, and relayed bytes would
double-count). Legacy dataTrackStats packet-loss/frame counters are
preserved.
Shared helper for callers that need to distinguish intentional/expected
participant closures (client leave, admin action, room teardown, migration)
from connection failures. Extracted from cloud's IsClosedIntentionally
switch so cloud-side code paths can share a single source of truth.
Reverting
- https://github.com/livekit/livekit/pull/4521
- https://github.com/livekit/livekit/pull/4525
There are TWCC feedback packets that are larger than MTU. Seems to
happen under a couple of conditions
1. Bad client data, i. e. severely out-of-order packets, bad sequence
numbers, etc.
2. On an ICE restart - this is rare, but it seemed to be flaky network
with some packets arriving and some not and causing a lot of gaps.
Either case, not much to do. If fargmentation/re-assembly back to
publisher works, the feedback will make it through. If not, feedbacks
will be missed and clients have to work with some missing data which is
not unexpected and the protocol is designed to handle.
However, filed pion/interceptor issue just in case - https://github.com/pion/interceptor/issues/416
* Don't require media sections when joining
Client except browser (rust/libwebrtc is known) could have problem
to fire ontrack event when reuses extra media section to subscribe
track, so disable this feature in server side and let client determine
if extra media sections are needed.
* lint
* rtc: report participant kind code and details
Plumb ParticipantKind and KindDetails through MediaTrack and
BytesTrackStats so track-level reporting can record the numeric kind
code plus details codes on every participant_session aggregation,
alongside the existing Kind string. Also picks up the new kind fields
on resolved BytesSignalStats participants.
Adds deployment/agentID/version to the agent worker logger.
* Log details of RTCP packets.
Seeing large (> MTU) packets on publisher peer connection RTCP. The
four types there are
- RTCP Receiver Reports
- NACK
- TWCC
- PLI
Can't think of what would be blowing up in size.
RTCP Receiver Report and PLI are fixed in size
NACKs vary, but the limit is 100 NACKs which should fit in 400 bytes
even if all of them are spread apart in the sequence number space.
TWCC varies, but a feedback packet is sent every 100ms or when it holds
100 packets. So, that also should not be too big.
Logging packet details to understand this better.
* revert debug
* Log large packets receive/send.
Seeing cases of servers reporting need for segmentation/re-assembly of
packets. So, logging packet receive/send for RTP/RTCP to check if
anything is seeing more than 1400 byte packets.
* log downtrack RTCP too
* Fix publish-only limitations being incorrectly applied receive-side in a single PC
* `StaticConfigurations` disabled some codecs for publish only, which worked in dual PC
* In single PC, the server incorrectly disabled these codecs in both directions
* Dual PC mode is unchanged; single PC handles per-direction filtering correctly
* Filter recv-side codecs to publish list in single-PC SDP answer
* Confirm H264 is present in offer in test
* Legacy TrackInfo.Simulcast flag.
When AddTrack did not send SimulcastCodecs, the legacy `Simulcast` flag
was not set. Fix it by setting the flag when a second layer is
published.
* staticcheck
* use the existing PrimaryReceiver function
