Subscription can switch between remote track and local track or
vice-versa. When that happens, closing the subscribed track of one or
the other asynchronously means the re-subscribe could race with
subscribed track closing.
Keeping the case of `isExpectedToResume` sync to prevent the race.
Would be good to support multiple subscribed tracks per subscription.
So, when subscribed track closes, subscription manager can check and
close the correct subscribed track. But, it gets complex to clearly
determine if a subccription is pending or not and other events. So,
keeping it sync.
Release to Docker / docker (push) Failing after 3m42s
* Key telemetry stats work using combination of roomID, participantID
With forwarded participant, the same participantID can existing in two
rooms.
NOTE: This does not yet allow a participant session to report its
events/track stats into multiple rooms. That would require regitering
multiple listeners (from rooms a participant is forwarded to).
* missed file
* data channel stats
* PR comments + pass in room name so that telemetry events have proper room name also
Even when encrypted, can set up opus as the second codec to support the
case of RED interspersed with Opus packets when the RED packet is too
big to fit in one packet.
The change here is to not go through all up stream codecs when trying to
find a match in DownTrack.Bind when source is encrypted. When encrypted,
the down track codec should match the primary upstream codec, i. e. the
codec at index 0.
The following scenario produced silence audio in egress
1. JS SDK publishing audio/red
2. Egress joins
3. Egress picks the RED primary receiver -> this converts RED to Opus in SFU
4. At the same time, codec regression is triggered back to the publisher. Publisher switches to publishing audio/opus and stops audio/red.
5. But, egress is still attached to audio/red and pulls down only silence.
Fix by checking if the negotiated codec needs publish before sending
subscribed codec udpate.
- Move downTrack instantiation to SubscribedTrack as it should own that
DownTrack. Still more to do here as `DownTrack` is fetched from
`SubscribedTrack` in a few places and used. Would like to avoid that,
but doing this initially.
- Use an interface from sfu.Downtrack and replace a bunch of callbacks.
SubscribedTrack is the implementation for DownTrackListener.
* Normalize mime type and add utilities.
An attempt to normalize mime type and avoid string compares remembering
to do case insensitive search.
Not the best solution. Open to ideas. But, define our own mime types
(just in case Pion changes things and Pion also does not have red mime
type defined which should be easy to add though) and tried to use it everywhere.
But, as we get a bunch of callbacks and info from Pion, needed conversion in
more places than I anticipated. And also makes it necessary to carry
that cognitive load of what comes from Pion and needing to process it
properly.
* more locations
* test
* Paul feedback
* MimeType type
* more consolidation
* Remove unused
* test
* test
* mime type as int
* use string method
* Pass error details and timeouts. (#3402)
* go mod tidy (#3408)
* Rename CHANGELOG to CHANGELOG.md (#3391)
Enables markdown features in this otherwise already markdown'ish formatted document
* Update config.go to properly process bool env vars (#3382)
Fixes issue https://github.com/livekit/livekit/issues/3381
* fix(deps): update go deps (#3341)
Generated by renovateBot
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Use a Twirp server hook to send API call details to telemetry. (#3401)
* Use a Twirp server hook to send API call details to telemetry.
* mage generate and clean up
* Add project_id
* deps
* - Redact requests
- Do not store responses
- Extract top level fields room_name, room_id, participant_identity,
participant_id, track_id as appropriate
- Store status as int
* deps
* Update pkg/sfu/mime/mimetype.go
* Fix prefer codec test
* handle down track mime changes
---------
Co-authored-by: Denys Smirnov <dennwc@pm.me>
Co-authored-by: Philzen <Philzen@users.noreply.github.com>
Co-authored-by: Pablo Fuente Pérez <pablofuenteperez@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: cnderrauber <zengjie9004@gmail.com>
That is the main change. Changed variable name to `isExpectedToResume`
everywhere to be consistent.
Planning to use the callback value in relays to determine if the down
track should be closed or switched to a different up track.
* Ignore `disabled` when adpative stream is enabled.
Due to interplay of adaptive stream/visibility/dynacast, when adaptive
stream is enabled, subscribed track forces visibility and starts
streaming at low quality. This would trigger a render on client and
trigger a visibility update.
So, even if a migration disables a track, upon migration complete and
subscription bind, ignore disable and stream.
* don't hold lock during callback
* don't need to store pubMuted
* don't need to hold settings lock for pub muted
* Integrate logger components
Dividing into the following components
* pub - publisher
* pub.sfu
* sub - subscriber
* transport
* transport.pion
* transport.cc
* api
* webhook
* update go modules
Previous change to check for non-zero width caused test failures
as subscribed track settings can use the quality field and not
necessarily width/height.
* Remove parked layer feature.
Not worth the added complexity.
Several reasons
- Not seeing black frames on pub mute always.
- If they are there, it can consume more than 30kbps if the parked layer
is high res. That is wasted bandwidth downstream when pub is muted.
- On resume, client some time sends PLI and that triggers a key frame
request.
But, leaving the separate `PubMuted` flag in forwarder in case we can
use it for better handling.
* need the request spatial
* Avoid reconnect loop for unsupported downtrack
If the client subscribes to a track which codec is unsupported by the
client, sfu will trigger negotiation failed and issue a full reconnect
after received client answer. If the client try to subscribe that track
then it will got full reconnect again. That will cause a infinite
reconnect loop until the client don't subscribe that track. This PR
will unsubscribe the error track for the client and send a
SubscriptionResponse that contain the reason to indicates the track's
codec is not supported to avoid the reconnect loop.
* Notes on wht to do
- Should targetLayers be altered while doing opportunistic locking
- Should targetLayers be altered in any other path than stream allocator path?
- Lock to layer as long as it is <= opportunistic layer
- When not congested, opportunistic can be highest
- When congested, opportunistic could be nil or lowest if paused is not allowed
- When muting, can we hold on to current layers (or keep it as previous) and
restore on unmute.
- Store current/target in forwarder state and restore on seeding
- Watch for looking for targetLayers, etc. when looking to insert padding
packets. There may be an assumption about restarting on key frame and hence
okay to insert padding when target layers are invalid. This may not be true
any more when doing opportunistic forwarding.
- Can we distinguish between publisher mute or dynacast (i. e. publisher side
stopping) vs subscriber mute and do something useful? Publisher side mute
could mean continuity in sequence numbers on a restart (might be able to
catch it with opportunistic forwarding). But, there is the challenge of
unmute from publisher via signalling channel vs media. If media is arriving,
should subscribers do opportunistic forwarding before publisher mute state
update happens?
- Maybe introduce a mode where forwarding continues to a frame end (of course
with a time limit just in case the end of frame packet is lost) and then
insert silence/padding packets?
- Ensure that audio blank frame insertion does not suffer from frame boundary
issues.
* pub/sub mute separate + more notes on things to check
* WIP commit, more notes
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* clean up
* slightly better comments
* Do not stop on unmute
* do not inject blank frames when pub muted
* do not forward on audio publisher mute
When AdaptiveStream is enabled, default the subscriber to LOW quality stream
we would want LOW instead of OFF for a couple of reasons
1. when a subscriber unsubscribes from a track, we would forget their previously defined settings
depending on client implementation, subscription on/off is kept separately from adaptive stream
So when there are no changes to desired resolution, but the user re-subscribes, we may leave stream at OFF
2. when interacting with dynacast *and* adaptive stream. If the publisher was not publishing at the
time of subscription, we might not be able to trigger adaptive stream updates on the client side
(since there aren't any video frames coming through). this will leave the stream "stuck" on off, without
a trigger to re-enable it
Added a new manager to handle all subscription needs. Implemented using reconciler pattern. The goals are:
improve subscription resilience by separating desired state and current state
reduce complexity of synchronous processing
better detect failures with the ability to trigger full reconnect
* Move subscribe/unsubscribe queue to participant.
As subscribe/unsubscribe operation can come from both
local media track or remote media track, participant
needs to have it.
* Remove comment
* Stop reneg timer on close
* address comments
* Use rtcscore-go to calculate audio/video score
Signed-off-by: shishir gowda <shishir@livekit.io>
* Get max expected layer and find max actual layer from stream
Signed-off-by: shishir gowda <shishir@livekit.io>
* Cleanup unused methods
Signed-off-by: shishir gowda <shishir@livekit.io>
* Cleanup code - address review comments
Signed-off-by: shishir gowda <shishir@livekit.io>
* get expected layer info instead of just quality
Signed-off-by: shishir gowda <shishir@livekit.io>
* Move SpatialLayerForQuality to utils/helpers
method is required in rtc,sfu and connectionstats pkg
Moved to utils/helpers.go to remove cyclic deps
Signed-off-by: shishir gowda <shishir@livekit.io>
* update tests
Signed-off-by: shishir gowda <shishir@livekit.io>
* Pick stream stats with max layer
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update rtcscore-go pkg to make rtt/jitter optional
when passing 0, rtcscore-go was setting default values
Signed-off-by: shishir gowda <shishir@livekit.io>
* update score to rating
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update rtcscore-go pkg to use simulcast layer info for score
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update score ratings to reflect rtcscore range
Signed-off-by: shishir gowda <shishir@livekit.io>
* update test params for new rtcscore
Signed-off-by: shishir gowda <shishir@livekit.io>
* Delay sending scores to connections only till full data is available
first interval can have partial data leading to lower scores
Signed-off-by: shishir gowda <shishir@livekit.io>
* Check for inf values in quality params
Signed-off-by: shishir gowda <shishir@livekit.io>
* Clean up initial score calculation. Default to 5
Signed-off-by: shishir gowda <shishir@livekit.io>
Co-authored-by: David Zhao <dz@livekit.io>
* Support participant identity in permissions
It is harder for clients to update permissions by SID as remote
reconnecting means a new SID for that participant. Using participant
identity is a better option.
For now, participant SID is also supported. Internally, it will
get mapped to identity. Server code uses identity throughout after
doing any necessary conversion from SID -> Identity.
* Address comments
* Refactor media track subscriptions
- To enable re-use of common bits
- Add max quality from other nodes
* Lock close handlers slice
* Reverting multiple on close handlers of downtrack, unclear if it is needed yet
* Make Logger a pointer
* audio level in MediaTrack like remote media track
* Cleanup
* Add a no subscribers callback
* Add method to update subscribed quality from another node
* loss proxying from remote node
* Address comments from David
* create subscriber node quality map
David Chen