* Normalize mime type and add utilities.
An attempt to normalize mime type and avoid string compares remembering
to do case insensitive search.
Not the best solution. Open to ideas. But, define our own mime types
(just in case Pion changes things and Pion also does not have red mime
type defined which should be easy to add though) and tried to use it everywhere.
But, as we get a bunch of callbacks and info from Pion, needed conversion in
more places than I anticipated. And also makes it necessary to carry
that cognitive load of what comes from Pion and needing to process it
properly.
* more locations
* test
* Paul feedback
* MimeType type
* more consolidation
* Remove unused
* test
* test
* mime type as int
* use string method
* Pass error details and timeouts. (#3402)
* go mod tidy (#3408)
* Rename CHANGELOG to CHANGELOG.md (#3391)
Enables markdown features in this otherwise already markdown'ish formatted document
* Update config.go to properly process bool env vars (#3382)
Fixes issue https://github.com/livekit/livekit/issues/3381
* fix(deps): update go deps (#3341)
Generated by renovateBot
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Use a Twirp server hook to send API call details to telemetry. (#3401)
* Use a Twirp server hook to send API call details to telemetry.
* mage generate and clean up
* Add project_id
* deps
* - Redact requests
- Do not store responses
- Extract top level fields room_name, room_id, participant_identity,
participant_id, track_id as appropriate
- Store status as int
* deps
* Update pkg/sfu/mime/mimetype.go
* Fix prefer codec test
* handle down track mime changes
---------
Co-authored-by: Denys Smirnov <dennwc@pm.me>
Co-authored-by: Philzen <Philzen@users.noreply.github.com>
Co-authored-by: Pablo Fuente Pérez <pablofuenteperez@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: cnderrauber <zengjie9004@gmail.com>
That is the main change. Changed variable name to `isExpectedToResume`
everywhere to be consistent.
Planning to use the callback value in relays to determine if the down
track should be closed or switched to a different up track.
* Ignore `disabled` when adpative stream is enabled.
Due to interplay of adaptive stream/visibility/dynacast, when adaptive
stream is enabled, subscribed track forces visibility and starts
streaming at low quality. This would trigger a render on client and
trigger a visibility update.
So, even if a migration disables a track, upon migration complete and
subscription bind, ignore disable and stream.
* don't hold lock during callback
* don't need to store pubMuted
* don't need to hold settings lock for pub muted
* Integrate logger components
Dividing into the following components
* pub - publisher
* pub.sfu
* sub - subscriber
* transport
* transport.pion
* transport.cc
* api
* webhook
* update go modules
Previous change to check for non-zero width caused test failures
as subscribed track settings can use the quality field and not
necessarily width/height.
* Remove parked layer feature.
Not worth the added complexity.
Several reasons
- Not seeing black frames on pub mute always.
- If they are there, it can consume more than 30kbps if the parked layer
is high res. That is wasted bandwidth downstream when pub is muted.
- On resume, client some time sends PLI and that triggers a key frame
request.
But, leaving the separate `PubMuted` flag in forwarder in case we can
use it for better handling.
* need the request spatial
* Avoid reconnect loop for unsupported downtrack
If the client subscribes to a track which codec is unsupported by the
client, sfu will trigger negotiation failed and issue a full reconnect
after received client answer. If the client try to subscribe that track
then it will got full reconnect again. That will cause a infinite
reconnect loop until the client don't subscribe that track. This PR
will unsubscribe the error track for the client and send a
SubscriptionResponse that contain the reason to indicates the track's
codec is not supported to avoid the reconnect loop.
* Notes on wht to do
- Should targetLayers be altered while doing opportunistic locking
- Should targetLayers be altered in any other path than stream allocator path?
- Lock to layer as long as it is <= opportunistic layer
- When not congested, opportunistic can be highest
- When congested, opportunistic could be nil or lowest if paused is not allowed
- When muting, can we hold on to current layers (or keep it as previous) and
restore on unmute.
- Store current/target in forwarder state and restore on seeding
- Watch for looking for targetLayers, etc. when looking to insert padding
packets. There may be an assumption about restarting on key frame and hence
okay to insert padding when target layers are invalid. This may not be true
any more when doing opportunistic forwarding.
- Can we distinguish between publisher mute or dynacast (i. e. publisher side
stopping) vs subscriber mute and do something useful? Publisher side mute
could mean continuity in sequence numbers on a restart (might be able to
catch it with opportunistic forwarding). But, there is the challenge of
unmute from publisher via signalling channel vs media. If media is arriving,
should subscribers do opportunistic forwarding before publisher mute state
update happens?
- Maybe introduce a mode where forwarding continues to a frame end (of course
with a time limit just in case the end of frame packet is lost) and then
insert silence/padding packets?
- Ensure that audio blank frame insertion does not suffer from frame boundary
issues.
* pub/sub mute separate + more notes on things to check
* WIP commit, more notes
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* WIP commit
* clean up
* slightly better comments
* Do not stop on unmute
* do not inject blank frames when pub muted
* do not forward on audio publisher mute
When AdaptiveStream is enabled, default the subscriber to LOW quality stream
we would want LOW instead of OFF for a couple of reasons
1. when a subscriber unsubscribes from a track, we would forget their previously defined settings
depending on client implementation, subscription on/off is kept separately from adaptive stream
So when there are no changes to desired resolution, but the user re-subscribes, we may leave stream at OFF
2. when interacting with dynacast *and* adaptive stream. If the publisher was not publishing at the
time of subscription, we might not be able to trigger adaptive stream updates on the client side
(since there aren't any video frames coming through). this will leave the stream "stuck" on off, without
a trigger to re-enable it
Added a new manager to handle all subscription needs. Implemented using reconciler pattern. The goals are:
improve subscription resilience by separating desired state and current state
reduce complexity of synchronous processing
better detect failures with the ability to trigger full reconnect
* Move subscribe/unsubscribe queue to participant.
As subscribe/unsubscribe operation can come from both
local media track or remote media track, participant
needs to have it.
* Remove comment
* Stop reneg timer on close
* address comments
* Use rtcscore-go to calculate audio/video score
Signed-off-by: shishir gowda <shishir@livekit.io>
* Get max expected layer and find max actual layer from stream
Signed-off-by: shishir gowda <shishir@livekit.io>
* Cleanup unused methods
Signed-off-by: shishir gowda <shishir@livekit.io>
* Cleanup code - address review comments
Signed-off-by: shishir gowda <shishir@livekit.io>
* get expected layer info instead of just quality
Signed-off-by: shishir gowda <shishir@livekit.io>
* Move SpatialLayerForQuality to utils/helpers
method is required in rtc,sfu and connectionstats pkg
Moved to utils/helpers.go to remove cyclic deps
Signed-off-by: shishir gowda <shishir@livekit.io>
* update tests
Signed-off-by: shishir gowda <shishir@livekit.io>
* Pick stream stats with max layer
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update rtcscore-go pkg to make rtt/jitter optional
when passing 0, rtcscore-go was setting default values
Signed-off-by: shishir gowda <shishir@livekit.io>
* update score to rating
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update rtcscore-go pkg to use simulcast layer info for score
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update score ratings to reflect rtcscore range
Signed-off-by: shishir gowda <shishir@livekit.io>
* update test params for new rtcscore
Signed-off-by: shishir gowda <shishir@livekit.io>
* Delay sending scores to connections only till full data is available
first interval can have partial data leading to lower scores
Signed-off-by: shishir gowda <shishir@livekit.io>
* Check for inf values in quality params
Signed-off-by: shishir gowda <shishir@livekit.io>
* Clean up initial score calculation. Default to 5
Signed-off-by: shishir gowda <shishir@livekit.io>
Co-authored-by: David Zhao <dz@livekit.io>
* Support participant identity in permissions
It is harder for clients to update permissions by SID as remote
reconnecting means a new SID for that participant. Using participant
identity is a better option.
For now, participant SID is also supported. Internally, it will
get mapped to identity. Server code uses identity throughout after
doing any necessary conversion from SID -> Identity.
* Address comments
* Refactor media track subscriptions
- To enable re-use of common bits
- Add max quality from other nodes
* Lock close handlers slice
* Reverting multiple on close handlers of downtrack, unclear if it is needed yet
* Make Logger a pointer
* audio level in MediaTrack like remote media track
* Cleanup
* Add a no subscribers callback
* Add method to update subscribed quality from another node
* loss proxying from remote node
* Address comments from David
* create subscriber node quality map
* Adaptive publishing improvements
* fix tests
* proper comment sentence
* Address comments from David
* proper locking
* fix crash
* Do not start quality timer if not video
* Scoped speaker update
Include only participants a participant is subscribed to.
NOTE: Not doing this for active speaker changes for Protocol < 3.
* correct comment spelling
* audio connection quality mos for publisher stats
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update tests
Signed-off-by: shishir gowda <shishir@livekit.io>
* Change ratings range, increase default rtt to 80
Signed-off-by: shishir gowda <shishir@livekit.io>
* Use stats worker to get total packets to find %lost in window
Signed-off-by: shishir gowda <shishir@livekit.io>
* Update go dep
Signed-off-by: shishir gowda <shishir@livekit.io>
* Increase interval of score cal to 5 seconds
Signed-off-by: shishir gowda <shishir@livekit.io>
* use lastSequenceNumber in reports to find total packets
Signed-off-by: shishir gowda <shishir@livekit.io>
* Account for delay while calculating scores
Signed-off-by: shishir gowda <shishir@livekit.io>
* Fix minor typo
Signed-off-by: shishir gowda <shishir@livekit.io>
* Add connection stats/score to subscribed audio tracks
Signed-off-by: shishir gowda <shishir@livekit.io>
* Cleanup
Signed-off-by: shishir gowda <shishir@livekit.io>
* Ignore duplicate LastSequenceNumbers in rtcp reports
Ignore if sequence number is less than what was recieved
Signed-off-by: shishir gowda <shishir@livekit.io>
* Move video track score calc to media/downtracks
Signed-off-by: shishir gowda <shishir@livekit.io>
* Deprecate SubscribeLossPercentage() as score calc is now handled downstream
Signed-off-by: shishir gowda <shishir@livekit.io>
* Initialize connection score to excellent
score is calc at 5sec interval. Client fetches score before first
score is computed
* Update test cases for connection quality
Signed-off-by: shishir gowda <shishir@livekit.io>
* WIP commit
* SubscribedQualityUpdate message to send list of currently subscribed
qualities for a simulcast video publisher
* Correct subscriberID
* goimports
* Do quality update on add/remove of subscribed track
* do not update quality when admin mute is active
* update quality on admin unmute
* Update protocol version
* Simplify max subscribed quality loop per David's suggestion
* WIP commit
* deficient handling
* Add missing ProvisionalAllocatePrepare
* adjust state on track removal
* Increase test timeout
* - Add comments about cooperative routines
- Take down transition if available in cooperative scheme
- Use layer comparison when taking down transition. Because of when the
bitrate is measured, it is not always guaranteed bandwidthDelta is -ve
when moving down.
- Do not add track to stream allocator till bind.
* make comment better
* a bit more clear comments
* Use OnBind on subscribed track
* Stream Allocator Try 3
Making an intermediate PR to do
- Special treatment for screen share tracks
- When allocating all tracks,
o try to stream all tracks by starting with the lowest layer
o multi-pass across tracks to get a more even distribution
Not yet done:
-------------
In deficient state,
o Allocate a specific track on a change
o Steal from other tracks
* Correct sense of managed track
* have to range to copy
* generate
* fix VideoLayers compare
* Use t.simulcasted
* WIP branch to hash out down stream allocator.
* Plug more bits of stream allocator
* update protocol
* remove SignalRequest_Simulcast (#154)
* Plug more bits of stream allocator
* Handle simulcast track available layers change
* WIP branch to hash out down stream allocator.
* Plug more bits of stream allocator
* Handle simulcast track available layers change
* Adopt signature of call to AdjustAllocation
* Move StreamAllocator to PCTransport and allocate only for subscriber.
* Move streamallocator to ion-sfu
* Start/Stop of streamallocator
* Use StreamAllocator for subscriber bandwidth management.
* Do not allocate in ADD_TRACK
* Set payload in constructor
* - Add some logging
- Protocol message to notify clients of paused/resumed streams
* named return
* oops correct sense of isPausing
* Update pkg/sfu/streamallocator.go
Committing David's suggestion.
Co-authored-by: David Zhao <david@davidzhao.com>
* - Log estimate changes/commits.
- Catch more than epsilon change oscillating for long time.
Co-authored-by: David Colburn <xero73@gmail.com>
Co-authored-by: David Zhao <david@davidzhao.com>
Raja Subramanian