* WIP commit
* SubscribedQualityUpdate message to send list of currently subscribed
qualities for a simulcast video publisher
* Correct subscriberID
* goimports
* Do quality update on add/remove of subscribed track
* do not update quality when admin mute is active
* update quality on admin unmute
* Update protocol version
* Simplify max subscribed quality loop per David's suggestion
* Fix forwarding status deduction
- When muted OR when there are no available layers, declare optimal
- When target layer is the maximum it can achieve taking available
layers into account even if they are not the maximum subscribed layer,
it is still optimal as there is nothing better available.
* Fix and add more tests for forwarding status
* WIP commit
* deficient handling
* Add missing ProvisionalAllocatePrepare
* adjust state on track removal
* Increase test timeout
* - Add comments about cooperative routines
- Take down transition if available in cooperative scheme
- Use layer comparison when taking down transition. Because of when the
bitrate is measured, it is not always guaranteed bandwidthDelta is -ve
when moving down.
- Do not add track to stream allocator till bind.
* make comment better
* a bit more clear comments
* Use OnBind on subscribed track
* Stream Allocator Try 3
Making an intermediate PR to do
- Special treatment for screen share tracks
- When allocating all tracks,
o try to stream all tracks by starting with the lowest layer
o multi-pass across tracks to get a more even distribution
Not yet done:
-------------
In deficient state,
o Allocate a specific track on a change
o Steal from other tracks
* Correct sense of managed track
* have to range to copy
* generate
* fix VideoLayers compare
* Use t.simulcasted
* WIP commit
* test padding only with a gap scenario
* Debug
* Fix video corruption, need buffer to include payload and translated header size
* Revert incorrect change
* Fix VP8 translation to return buffer with proper length
* Restore 7-bit mode
* WIP commit
* Clean up
* More clean up and tests compiling again
* Fix tests
* Make VP8 packet translation thread-safe.
Was using one packet from pool for all VP8 translation which was not thread safe.
Grab packets from the pool when needed for VP8 translation and return to pool after done.
Do not grab packet from pool if the header size between incoming and translated matches.
That also saves copying the packet payload.
* Keep Get/Put in the same function.
* Use protocol friendly StreamedTracksUpdate
* WIP commit
* Stream allocator update
* subtract the requested bandwidth as delta from Allocate could be adding to bandwidth
* Calculate delta correctly
* correct comment
* Simplify eventCh per David's suggestion
* Simplifying (hopefully) sfu.DownTrack
* Remove unnecessary check as pdding only packets are dropped before that check
* Temporal filtering max layer
* - Split out forwarder bits into a separate structure
- Address comments from Jie and David
* Remove debug and unneeded stuff
* Fix test
* Remove unneeded default initialization
* WIP branch to hash out down stream allocator.
* Plug more bits of stream allocator
* update protocol
* remove SignalRequest_Simulcast (#154)
* Plug more bits of stream allocator
* Handle simulcast track available layers change
* WIP branch to hash out down stream allocator.
* Plug more bits of stream allocator
* Handle simulcast track available layers change
* Adopt signature of call to AdjustAllocation
* Move StreamAllocator to PCTransport and allocate only for subscriber.
* Move streamallocator to ion-sfu
* Start/Stop of streamallocator
* Use StreamAllocator for subscriber bandwidth management.
* Do not allocate in ADD_TRACK
* Set payload in constructor
* - Add some logging
- Protocol message to notify clients of paused/resumed streams
* named return
* oops correct sense of isPausing
* Update pkg/sfu/streamallocator.go
Committing David's suggestion.
Co-authored-by: David Zhao <david@davidzhao.com>
* - Log estimate changes/commits.
- Catch more than epsilon change oscillating for long time.
Co-authored-by: David Colburn <xero73@gmail.com>
Co-authored-by: David Zhao <david@davidzhao.com>
Use max layer subscription of down tracks to
prioritize bandwidth allocation. The ordering is
- Highest spatial layer
- Highest temporal layer, if spatial layer matches
- Down track id if both layers match (this is to
prevent unnecessary re-ordering)
Testing:
--------
(Really need to make a DownTrack interface so that
we can mock it and use it in tests)
For now, just a sanity check.
Small step on the way to making StreamAllocator prioritization of tracks.
With the new callback into StreamAllocator, the idea is to use the
max layer information to do track prioritization.
Testing:
--------
Sanity check that sample app works
* Handle multiple codecs in renegotiation
update pion to v3.1.9 for answer same order of codec as publisher.
register enable codecs in subscriber peerconnectin created.
add codec parameter to buffer.bind
buffer should use the codec of TrackRemote as it's codec mime.
sent h264blankframe when DownTrack closing
When we RLock during write cycles, the mutex spends the majority of its time
staying locked. As new participants join, they have to acquire the WLock
before downtracks could be add it.
In load test scenarios (25 participants joining together), it's common to see
goroutine dump showing MediaTrack.AddSubscriber -> DownTrack.storeDownTrack trying to acquire mutex, and never able to acquire it.
* Fixing edge cases in picture id munging.
Changes
1. Check the RTP sequence number order before VP8 temporal layer
filtering and use that ordering result while doing temporal
layer filtering.
In a sequence like below
o Packet 10 -> Picture ID 10
o Packet 11 -> missing
o Packet 12 -> Picture ID 11
it is not known if packet 11 will belong to Picture ID 10 or
Picture ID 11. The problem becomes a lot more tricky if there
is a burst loss and there is a larger hole in the picture id
space also as a result.
So, in the event of a packet loss, forward even if the current
packet belongs to a layer that can be dropped. More comments
in code.
2. Use result of sequence number ordering check while doing VP8 picture id munging.
3. When adding to missing picture id cache, have to include picture ids including
both ends. As a picture can span multiple packets and it is not known which
picture the packet belongs to, have to include both ends also in missing
picture id cache in the event of a gap.
4. As a picture can span multiple packets, it is not possible to have a simple
map of missing picture ids as an entry cannot be deleted if an out-of-order
picture id is received. There may be more missing packets belonging to that
picture id that is yet to be received.
So, have to use an ordered map and truncate the map if it grows too large.
Picked this for ordered map - https://github.com/elliotchance/orderedmap.
Has a simple API, had the highest number of stars of all the ones I checked.
And there are benchmarks.
The author also wrote a medium post at https://medium.com/swlh/an-ordered-map-in-go-436634692381
Another one which I looked at is - https://github.com/wk8/go-ordered-map.
The author of that wrote at https://morioh.com/p/990229f32171 and has a
bunch of other options at the end of that post (but does not include the
one I picked above). None of those have that many stars.
Testing:
--------
- Set max temporal layers to 0 so that temporal filtering happens and run for
an hour on sample app.
* do not let padding packets through VP8
* Correct comment
* fix comment
* Review comments from Jie
* golang naming convention
* Try sending small key frames to clear decoded buffer
Problem:
--------
With transceiver re-use, client disables/enables tracks.
With video, this retains the last picture and when a new track
starts, there is a brief moment when the old stream is displayed
till there is data from new stream to decode and display.
Fix:
---
Send small key frames before closing DownTrack to try and clear
the decoder display buffer.
Testing:
--------
Tried with Chrome/Safari/Firefox and they worked. But, very rarely,
the last frames do not seem to show up. In fact, 6 frames are sent
and webrtc internals (in Firefox) reports anywhere from 1 - 6 frames
at the small resolution. Unclear as to why it does not get all the
frames or why it reports less than 6. A not so small percentage of
times (maybe 1 in 15 - 20), have seen no small frame reported at all.
TODO:
----
- Have to support more video codecs
- Would this be an issue for audio also? Should we send something to handle that?
Probably not necessary as video is more jarring.
* Make VP8 Key Frame a const
* Need a packet factory buffer for simple tracks too
as we are using the VP8 munger for simple tracks too because
of the need to send blank frames at the end.
Also, making the writeBlankFrameRTP a private function.
And adding a check to not send blank frames if nothing has been sent
on that DownTrack.
* Separate from ion-sfu
changes:
1. extract pkg/buffer, twcc, sfu, relay, stats, logger
2. to solve cycle import, move ion-sfu/pkg/logger to pkg/sfu/logger
3. replace pion/ion-sfu => ./
reason: will change import pion/ion-sfu/pkg/* to livekit-server/pkg/*
after this pr merged. Just not change any code in this pr, because it
will confused with the separate code from ion-sfu in review.
* Move code from ion-sfu to pkg/sfu
* fix build error for resovle conflict
Co-authored-by: cnderrauber <zengjie9004@gmail.com>
In order to avoid race conditions with WebRTC, where either side could initiate an offer when tracks have changes, we'll always initiate them from the SFU side.
Artur Shellunts