* agent: release JobTerminate handler when job ends before registration
JobRequest registered the job's JobTerminate handler after AssignJob
returned, so an ended UpdateJobStatus or worker disconnect handled in
between deregistered nothing and the handler leaked. AssignJob also sent
the assignment before recording the job as running, so an immediate
ended update was dropped.
Record the job before sending the assignment, and after registering the
handler recheck under h.mu that the job is still running and its worker
still registered, deregistering otherwise.
Fixes#4901
* replace utils.CloneProto with proto.CloneOf
* agent: wait on worker readiness and poll for released handlers in test
* sfu: broadcast RTP to down tracks without a per-packet closure
Every forwarded packet allocated a closure capturing the packet and layer,
and an atomic write counter that escaped with it, even for tracks with a
single subscriber. Add BroadcastRTP, which carries the packet and layer in
the worker state and sums the writes per worker. The serial path no longer
allocates; the parallel path allocates the shared state and the worker
funcval only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: reuse the forwarded packet copy in the RED receivers
Both RED receivers copied the ExtPacket (and the opus one the rtp.Packet)
per forwarded packet, and the copies moved to the heap because the
broadcast takes their address. Keep them on the receiver like the existing
redPayloadBuf: ForwardRTP runs on one goroutine and down tracks do not keep
the packet past WriteRTP.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: skip BroadcastRTP allocation assertion under the race detector
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* sfu: group BroadcastRTP and its interfaces after the spreader methods
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Tests for down stream packet push.
A recent issue (padding bit in RTP header) surfaced a gap which slipped
through due to lack of tests. Changes in pion/rtp were not adopted
properly.
So, adding some tests (thank you Claude for the heavy lifting) to test
the down stream packet path using the whole pion chain.
Split out some interfaces so it is easier to have it all in one place
and create fakes.
Will help adding more tests, for example include the upstream path also
in the integration test. May have to create more interfaces and make
things testable, but this is a start.
* missed file
* rtx specific test
Following up on this PR (https://github.com/livekit/livekit/pull/4678 -
great catch btw, thank you), propagating the spreader threshold to RED
receivers also.
Removing the test as it was a simple case and that would not have caught
the original issue.
* Use Muted in TrackInfo to propagated published track muted.
When the track is muted as a receiver is created, the receiver
potentially was not getting the muted property. That would result in
quality scorer expecting packets.
Use TrackInfo consistently for mute and apply the mute on start up of a
receiver.
* update mute of subscriptions
* Add debug for receiver restart.
Have a suspicion that something is deadlocking between restart receiver
and buffer bind during replay. Adding debug to get a better picture of
state of receiver restart.
* consistent logging
In some paths, it is better to preserve pre-recorded time. So, make the
base implementations preserve the RTCP Sender Report receive time.
Also, add a method to enable forwarding packet arrival time. Could be
used across relay.
* Rework receiver restart.
- Protect against concurrent restarts
- Clean up and consolidate code around restarts
- Use `RestartStream` of buffer rather than creating new buffers.
* fix test
* Minor refactor in buffer base and audio level
- Make a function for `restartStream`. Will be useful
when external signal needs to restart a stream. Also restart all the
bits (audio level, dd parser and frame rate calculator)
- make an audio level mode with RTP timestamp so that some state can be
moved out of buffer base
* clean up
* log restart
* Refactor receiver and buffer into Base and higher layer.
To be able to share code/functionality with relay.
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* clean up
* deps
* fix test
* fix test
* Store buffer after creating it.
Also changing signature of creator function as it could call TrackInfo()
and get into a deadlock.
* fix double unlock
* add some more debug logging
* Refactor receiver and buffer into Base and higher layer.
To be able to share code/functionality with relay.
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* clean up
* deps
* fix test
* fix test
Paul Wells