- Count a publish attempt on a migrating in tarck as there is no
AddTrack for that.
- Add cancel publish only if the participant connection is canceled
- Do not add publish counter for synthetic publish attempts which
happens for migrating in tracks. It will be counted on migrating in
node when the track is actually published, i. e. negotiated/packets
flowing.
* Do not call telemetry listener under pending track lock.
Fix the TrackPublishRequested call of telemetry listener.
Audited other callbacks to ensure that it is not under lock.
* missed some paths of recording it, thanks Devin
* Record subscribe stream start time in prometheus.
Adjust for mutes, i. e. take the last unmute time as the start point and
calculate time till the first byte is sent.
* close the tiny window of race
* Prevent long tail sample when publisher glitches.
Thanks to @milos-lk for this.
Publisher restarting would have reset the layer and would have caused a
sample with very high stream start time. We only need to capture when we
do a dummy start or when the state is seeded to a different node upon
migration.
* reduce a diff
* test
* changed the wrong thing, thank you Devin
* Tests for down stream packet push.
A recent issue (padding bit in RTP header) surfaced a gap which slipped
through due to lack of tests. Changes in pion/rtp were not adopted
properly.
So, adding some tests (thank you Claude for the heavy lifting) to test
the down stream packet path using the whole pion chain.
Split out some interfaces so it is easier to have it all in one place
and create fakes.
Will help adding more tests, for example include the upstream path also
in the integration test. May have to create more interfaces and make
things testable, but this is a start.
* missed file
* rtx specific test
Addresses https://github.com/livekit/livekit/issues/4689
We were probably missing a couple of bits with this
1. Not affected for regular traffic like from browsers as it does not
add padding, but special clients were affected.
2. Probe packets were probably using wrong last byte as pion/rtp would
have overwrriten with 0 because the header.PaddingSize for the newer
versions were not set. That could have affected bandwidth estimation
catch up.
Following up on this PR (https://github.com/livekit/livekit/pull/4678 -
great catch btw, thank you), propagating the spreader threshold to RED
receivers also.
Removing the test as it was a simple case and that would not have caught
the original issue.
In single-PC and one-shot signalling modes the subscriber PCTransport is
never created, so t.subscriber is nil. Every other TransportManager method
that touches the subscriber nil-checks it first; HandleAnswer did not, so a
client sending an SDP answer in those modes crashed the process with a nil
pointer dereference. Guard it and log, matching AddICECandidate/Negotiate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Was doing it on participant close. But, that misaligns `rtc_init`
(number of attempts) a bit if the participant sessions are long.
Record it as soon as RTC connects so that it is more time aligned with
`rtc_init`.
* fix: don't undercount data channel bitrate over idle gaps
BitrateCalculator divided drained bytes by wall-clock time, including
idle gaps between sparse writes. A fast channel fed sparsely (e.g. 1 KB
every second, acked in a few ms) therefore reported its low offered load
(~8 kbps) instead of its drain capacity (~800 kbps). In the unreliable
writer that shrinks targetLatencyLimit and drops bursts on a channel
that is actually keeping up.
Measure the rate over backlogged ("busy") time only, so idle time no
longer dilutes the estimate. Backlog is detected from the buffer
occupancy just before a write (bufferedAmount - bytesWritten): since no
bytes are added between writes the buffer only drains, so a non-zero
pre-write level means it never emptied and the interval was genuinely
busy. Gating on the post-write buffered amount would be wrong -- it
always includes the just-enqueued (not yet SACKed) bytes and is ~never
zero, collapsing the estimate back to wall-clock. When no backlog is
ever observed the calculator reports no estimate (ok=false) instead of a
confidently-low number.
Introduce BitrateMode:
- BusyOnly (writer default): busy denominator, all drained bytes counted.
- ExcludeIdleDrain: also drops bytes drained across non-backlogged
intervals for a clean drain-capacity estimate free of idle
contamination.
- WallClock: elapsed-time denominator, all bytes; for the test-client
reader which has no send buffer to observe.
Call sites keep the conservative behaviour (writers BusyOnly, reader
WallClock). Tests exercise both writer modes and the mixed idle-then-
backlog window where they diverge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Trim comments in data channel bitrate calculator
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make BitrateMode.String a method on the enum
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix goroutine leak from orphaned signal relay streams
signalService.RelaySignal blocks on the first `<-stream.Channel()` waiting for the StartSession message. psrpc's streamHandler.handleOpenRequest only closes the stream after the handler returns, so if a stream is opened but the client goes away before sending StartSession, the channel is never fed and never closed, and this goroutine blocks forever. Under mass reconnects this leaks one goroutine (and its retained objects) per orphaned stream; they only clear on process restart.
Wrap the initial receive in a select that also returns when the stream context is cancelled or after config.SignalRelay.RetryTimeout, so an orphaned stream returns before Hijack() and psrpc closes it.
Signed-off-by: SKaterinenko <skaterinenko@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Some data track related tests are failing due to setting a target
latency of 100ms. Would be good to understand if taget latency or min
buffer is causing the drops.
* fix: bound data-track buffering under downlink congestion
The SFU data-track down-channel writer was constructed with targetLatency=0,
which disables the buffered-amount/latency-based drop in writeUnreliable. Under
subscriber downlink congestion, frames were queued unbounded and in order in the
per-subscriber SCTP send buffer instead of being dropped, so end-to-end latency
grew without limit and only drained once congestion cleared. This contradicts
data tracks being a low-latency, lossy transport.
Wire a dedicated `datachannel_data_track_target_latency` config through to both
data-track writer call sites (mirroring how `datachannel_lossy_target_latency`
controls the lossy publishData channel), reusing lossyDataChannelMinBufferedAmount
as the drop floor. Defaults to 100ms so data tracks are latency-bounded out of
the box; set to 0 to restore the previous unbounded behavior.
Fixes#4666
* fix: track drained bytes on successful unreliable data channel writes
writeUnreliable only called BitrateCalculator.AddBytes when a write failed,
so on the common success path the calculator never received samples. As a
result Bitrate() had no data and the latency-based drop threshold collapsed to
the static minBufferedAmount floor, making the per-subscriber latency control
(datachannel_lossy_target_latency and the new datachannel_data_track_target_latency)
effectively inert. Call AddBytes after every write, matching writeReliable.
Newer kernels report qdisc stats via TCA_STATS2 (Stats2) while older
kernels only populate the legacy TCA_STATS attribute (Stats). Whichever
is unused is left nil, so unconditionally dereferencing Stats crashed
with a SIGSEGV during telemetry init. Prefer Stats2 and fall back to
Stats, skipping when both are nil.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use by cloud simulated tracks.
Introduces a lock, but it is used only in the background worker (where
it will be almost always uncontested) and when API GetStats is accessed.
Does not touch the per-packet Update path.
* Move ForwardStats aggregation off the packet forwarding path
ForwardStats is a process-wide singleton and its Update runs for every
forwarded packet. It previously took a shared mutex, updated a windowed
aggregate, and observed into a global Prometheus histogram on every call.
Update now only buffers the transit sample into a sharded lock-free ring
(one atomic add to reserve a slot, one atomic store to publish). A
background worker drains the ring every summary interval, observes each
sample into the histogram (per-packet fidelity retained) and folds the
interval summary into a window ring for the latency/jitter gauges. Under
sustained overload the oldest excess samples are dropped and counted, and
the count is logged.
Ring slots are atomic.Int64 so producer store / consumer load are
synchronized (race-clean). Capacity is numShards*shardCap = 131072
samples; at the default 50ms summary interval that sustains ~2.6M
samples/s before dropping.
Benchmark: BenchmarkForwardStatsUpdate (0 allocs/op both), Update per call:
cores before after speedup
1 ~8.6 ns ~1.5 ns ~6x
8 ~175 ns ~22 ns ~8x
The before path (mutex + windowed Welford aggregate + per-packet histogram
observe) degrades ~20x from 1 to 8 cores under contention; the after path
does not block and stays an order of magnitude lower under load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix ring publish race and report-path double-flush
Addresses two review findings on the forward-stats sample buffer.
1. Publish race (reserve-before-store): push reserved a ring slot by
advancing writeIdx and stored the value afterwards, so drain could read a
slot the producer had reserved but not yet written, getting a stale/zero
value; the producer's later store then landed behind the read cursor and
was lost. Each slot now carries a publish epoch. push stores the value and
then publishes seq = index+1. drain reads a slot only when seq == r+1; a
reserved-but-unpublished slot at the cursor stops the drain and is picked
up on the next call, so no sample is read stale or lost. A slot overwritten
during the read is detected on re-check and counted as dropped.
The windowed latency/jitter stats did not permanently drift even before
this change (report recomputes from a fixed-size ring that overwrites, not
discounts, old buckets, so any error aged out within the report window),
but these values feed the capacity manager, so the buffer is made exact.
2. Report-path double-flush: run() flushed on both the summary tick and the
report tick, and every flush advances the window ring, so the ring cycled
faster than intended and the effective window was shorter than configured
(~5% at 50ms/1s/1m). The report tick no longer flushes; the summary ticker
keeps the ring current to within one summary interval.
Benchmark, Update per call (0 allocs/op), 1 and 8 cores:
after fixes: ~5.5 ns / ~29 ns
before fixes (this branch): ~1.5 ns / ~22 ns
baseline (mutex + per-packet histogram): ~8.6 ns / ~175 ns
The publish epoch adds one atomic store to push; the path stays non-blocking,
zero-allocation, and well below the baseline under contention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Leave out canceled attempts. Should make it easier to do percentages.
Not putting these in node stats yet. Will observe in prom before using
it in node stats.
The WHIP connection-notify loop kept issuing RPCs after the participant
had left the room. Guard sendConnectionNotify against a closed
participant (returning ErrParticipantNotFound) and treat that error as a
clean loop exit. Also reorder DeleteSession so the participant is removed
before its WHIP OnClose entry is cleared.
Adds tests covering loop termination on participant leave, context
cancellation, and the closed-participant guard.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* Pin protocol for schema metadata
* Pass through schema and frame encoding
* Support custom encoding identifiers
* Rename config key
* Increase default length to 32
* Make log messages more generic
* Use getters with built-in null check
* Do not bump deps
* Rename function
* Use protocol v1.48.1 release
---------
Co-authored-by: boks1971 <raja.gobi@tutanota.com>
* Async attributes on participant.
How it is different from existing participant attributes?
1. Async attribute can be added one at a time.
2. These are not included in `ParticipantInfo`.
3. Get an attribute bt participant identity and async attribute ID as
and when needed.
* clean up
* get full definitions, not just ids
* listener OnDataTrackSchema
* name length config
* data blob
* deps
* static check
* Add missing request ID
* Update protocol commit
* Wire up StoreDataBlobResponse
* Pass request ID through in GetDataBlobResponse
* deps
* atomic
* sctp at 1.9.5
* remove proto clone
---------
Co-authored-by: Jacob Gelman <3182119+ladvoc@users.noreply.github.com>
In single peer connection mode, when the server answers a subscriber's
offer, configureSenderAudio set the sender codec preferences from the
server MediaEngine's payload types. The answer could therefore advertise
Opus on a payload type the offerer never offered (server PT 111 vs
offered PT 109). Chrome tolerates this; Firefox decodes 0 samples
(silence) -- packets are received but never decoded. The forwarded RTP
already uses the offered PT, so only the answer SDP was inconsistent.
This regressed in v1.12.0 once the single-PC MediaEngine became a union
of publish+subscribe codecs.
Parse the remote offer's audio rtpmap and remap the sender audio codec
preferences to echo the offered payload types (RFC 3264 6.1) before
SetCodecPreferences.
Fixes#4599
Co-authored-by: laosun <14806343+cnvipstar@users.noreply.github.com>
* Prometheus metric for join latency.
Also including a couple of other failures in the signal connection path
and moving the signal connected to after all that.
Not doing counters for the new signal failure paths. I should not have
done for the other two I added a little while ago also (
validation failure and start participant failure) as those are not
scalable to keep adding to node stats. Will probably remove those two
from node stats later. Can add those counters if they are useful.
* deprecate signal failed counters
Previously it was anchored to participant transitioning to `ACTIVE` if
the add track request happened before that. But, that has a few issues
1.`ACTIVE` is for primary peer connection which could be subscriber peer
connection.
2. `ACTIVE` also include data channel establishment.
Switch to first connected time of publisher peer connection for that to
get a more accurate measure of track publish time.
noticed a config in deploy config while cleaning up some other usused
config. small clean up. probably there is a bunch more that can be
cleaned up, but doing a quick one as I noticed this.
* feat: acquire requested video layer directly at HIGH quality by default
Two changes that together remove the visible low->high quality ramp for a new
subscriber (both publisher-first and subscriber-first join orders):
1. Default a subscriber's initial video quality to HIGH on bind instead of LOW
for adaptive stream, so the subscribed max layer is the top layer. Adaptive
stream clients can still scale down afterwards based on viewport.
2. On initial layer acquisition the forwarder/selector latch directly onto the
allocator's target (the requested top layer) instead of opportunistically
latching onto the first lower key frame that arrives. A short
initial-acquisition grace aims the target at the requested layer; if it does
not show up in time, the target falls back to the highest layer seen so
acquisition never stalls.
Always on - no configuration flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: gate start-at-desired-quality behind EnableStartAtDesiredQuality flag
Put the "acquire requested video layer directly at HIGH quality" behavior
behind a per-subscriber EnableStartAtDesiredQuality flag (default off, so
the original low->high ramp-up is restored unless enabled).
Plumbed from config.RTC.EnableStartAtDesiredQuality through ParticipantParams
-> SubscribedTrack/DownTrack -> Forwarder -> simulcast selector, gating all
three behavior changes: the HIGH default on bind, the forwarder's
initial-acquisition grace, and the selector's direct-latch-onto-target.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* remove config.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix skipped packets accounting.
No need to copy unskipped packet RTP header to skipped packet.
That was causing padding bytes to be counted.
Also use Header.PaddingSize as base PaddingSize is deprecated.
* PaddingSize in header in utils
* service: enforce metadata size limit in CreateRoom, bump default to 512 KiB
CreateRoom previously accepted any metadata size; only UpdateRoomMetadata
rejected oversized payloads. Mirror the same CheckMetadataSize check at
the CreateRoom API boundary so both entrypoints are bounded.
Default MaxMetadataSize moves from 64000 to 512 * 1024 to match the
practical needs of customers using room metadata for richer state. The
limit remains configurable via the existing limits.max_metadata_size knob.
* service: split room vs. participant metadata limit, enforce on join + agent dispatch
LimitConfig.MaxMetadataSize was shared between room metadata and
participant metadata. Last commit's bump to 512 KiB lifted both ceilings;
this restores the participant ceiling to 64 KB and introduces a separate
MaxRoomMetadataSize (default 512 KiB) for room metadata.
Additional enforcement:
- RoomManager.StartSession rejects joins whose JWT-grants metadata or
attributes exceed the participant/attributes limits. The check was
missing entirely from this path.
- AgentDispatchService.CreateDispatch and the embedded
CreateRoomRequest.Agents path now validate metadata and attributes
against the common 64 KB ceilings (previously unbounded).
NewAgentDispatchService gains a LimitConfig parameter; the two wire_gen
callsites are updated.
* service: collapse metadata size limit to single 512 KiB knob
Reverts the LimitConfig split introduced in the previous commit:
MaxRoomMetadataSize, CheckRoomMetadataSize, and the max_room_metadata_size
yaml key are removed. MaxMetadataSize moves back to 512 * 1024 and gates
all metadata uniformly — room (CreateRoom, UpdateRoomMetadata), participant
(UpdateParticipant, signal UpdateMetadata, JWT grants on join), and agent
dispatch (CreateDispatch + embedded RoomAgentDispatch).
MaxAttributesSize stays at 64 KB and continues to gate participant and
agent-dispatch attributes separately.
Test cases consolidated under the single knob.
* kb -> kib
