Commit Graph

692 Commits

Author SHA1 Message Date
Raja Subramanian d0128b19cd Reset sender reports before measuring clock skew. (#2437) 2024-02-02 21:52:43 +05:30
Raja Subramanian 174e69c81d Restore min score to 30. (#2435)
Was at 20 when LOST was introduced, but was going to 20 even when under
not LOST conditions. When there are packets, want the min to be at 30.
Going down to 20 resulted in reporting LOST quality even when packets
were flowing (although they were experiencing heavy loss and quality
would have been very bad, yet they are not lost).

Also, sample warning about adding packet to bucket even more.
2024-02-02 08:52:52 +05:30
Raja Subramanian ff69c2aa11 Add debug to understand VP9 freezes. (#2434)
* Add debug to understand VP9 freezes.

Have reports of VP9 freezing in some rooms.
Some data indicates that NACKs are received by SFU, but cannot get RTP
packet when that happens. It is possible that the NACKs are all from
dropped packets. Adding some debug to understand drops/NACKs better.

* enable DD debug

* comment out DD debug

* markers

* add back log about diff length mismatch

* add back key frame mismatch logging

* log skipped drops also
2024-01-31 15:33:39 +05:30
Raja Subramanian b71d373f4a Use Deque in ops queue. (#2418)
* Use Seque in ops queue.

Standardizing some uses
- Change OpsQueue to use Deque so that it can grow/shrink as necessary and
  need not worry about channel getting full and dropping events.
- Change StreamAllocator and TelemetryService to use OpsQueue so that
  they also need not worry about channel size and overflows.

* Address feedback

* delete obvious comment

* clean up
2024-01-28 13:48:30 +05:30
cnderrauber 9b4ba2d41d use default max playout delay as chrome (#2411) 2024-01-26 13:32:54 +08:00
cnderrauber 995fddbaf9 Add dynamic playout delay if PlayoutDelay enabled in the room (#2403)
* Add dynamic playout delay

* type for state
2024-01-26 09:33:35 +08:00
Raja Subramanian f29a28611b Prevent writable race. (#2388)
It is possible that onBindAndConnectedChanged gets executed in such a
way that `writable` does not have the correct value in some very rare
timing case (i. e. case like two executions of the function is racing
and one atomic was read on first exeuction and second execution runs and
sets `writable` and then first execution completes the sets `writable`
to incorrect value based on stale read of first execution).

Prevent it by executing under bind lock.
2024-01-16 12:15:13 +05:30
cnderrauber 5b4848e772 remove dd debug logs (#2387) 2024-01-16 12:07:29 +08:00
Raja Subramanian a1ca41a4e1 Skip reporting skew for out-of-order reports (#2369) 2024-01-08 13:11:04 +05:30
Raja Subramanian 73af5da956 Notify TrackInfo available from red receivers. (#2354)
* Notify TrackInfo available from red receivers.

That kicks off the down track scorer.

* test
2023-12-28 15:50:51 +05:30
cnderrauber fcee6edcb2 change svc frame number log to debug (#2347) 2023-12-26 11:40:34 +08:00
Raja Subramanian bdcd142c0d Adding some logs in subscribe path. (#2343)
Trying to chase down an older client failing to subscribe some times.
2023-12-25 14:12:08 +05:30
Raja Subramanian f5e2c1da15 Debug log downtrack life cycle a bit (#2342) 2023-12-24 14:03:51 +05:30
Raja Subramanian 6cac17affe Add some debug logs around track publish (#2340) 2023-12-23 18:55:24 +05:30
Raja Subramanian faff67162b Consolidate TrackInfo. (#2331)
* Consolidate TrackInfo.

TrackInfo was spread across a bit. Consolidating it.

* TODO comments

* test

* update TrackInfo on SSRC change

* further consolidation

* log mimes only

* update receivers on SSRC set

* clone proto on return

* feedback: break loop on mime match

* prevent data race
2023-12-21 09:56:54 +05:30
Raja Subramanian 5ee307952e Reduce a couple of logs to Debugw. Small saving. (#2322) 2023-12-18 14:27:55 +05:30
cnderrauber a150eaf697 Fix mid info lost when migrating multi-codec simulcast track (#2315)
* Fix mid info lost when migrating multi-codec simulcast track

* update pion
2023-12-15 00:02:27 +08:00
cnderrauber b0b7489b0e Fix svc freeze caused by key frame id wrap around (#2302)
* Fix svc freeze caused by key frame id wrap around

* rename
2023-12-07 12:53:17 +08:00
Raja Subramanian 1f335dd564 Convert to formatter string for lazy evaluation. (#2298) 2023-12-06 18:11:05 +05:30
Raja Subramanian 83efa9258e Bump up protocol for connection quality LOST. (#2297)
Also log trackID/trackInfo in layer mapping.
2023-12-06 16:59:05 +05:30
cnderrauber e1cc9d6b3c Fix log marshal error (#2295) 2023-12-06 00:08:48 +08:00
David Zhao 3fe124c87f Log cleanup pass (#2285)
* Log cleanup pass

Demoted a bunch of logs to DEBUG, consolidated logs.

* use context logger and fix context var usage

* moved common error types, fixed tests
2023-12-02 15:07:31 -08:00
Raja Subramanian d866b5110f Restrict scope of negotiation time out error logs (#2283)
* Restrict scope of negotiation time out error logs

1. Log "negotiation failed" only if signal channel was active
within half window of negotiation timeout. Negotiation timeout currently
is at 15 seconds. Signal pings are every 10 seconds.
2. In transport.go, do not report negotiation timed out and do not
callback negotiation failure if the peer connection state is not
connected. Goal of negotiation failure tracker is to take remedial
action when an in-session negotiation fails. Seeing a bunch of cases
of the case hitting even without ICE connection forming. Negotiation
timer is not intended for those cases.

* fix test
2023-12-02 12:44:37 +05:30
Raja Subramanian 2299a493de Throttle DD parse logs (#2281) 2023-12-01 12:42:12 +05:30
Raja Subramanian 7b778c50eb Group SDES items for one SSRC in the same chunk. (#2280) 2023-12-01 11:37:14 +05:30
Raja Subramanian 2ee5aa7c98 Add optional supervisor disable. (#2277)
* Add optional supervisor disable.

Used `DisableSupervisor` so that default can be enabled and
it can be disabled explicity. But, open to defaulting to disable
(i. e. change param to `EnableSupervisor`).

* Move nil check to call site
2023-11-30 13:04:31 +05:30
cnderrauber 0f1c1ec224 clean dd log (#2275)
* clean dd log

* Implemented Raja's feedback

---------

Co-authored-by: David Zhao <dz@livekit.io>
2023-11-29 12:12:29 -08:00
Raja Subramanian a2053dfd94 ConnectionQuality DISCONNECTED -> LOST (#2276) 2023-11-29 23:17:17 +05:30
Raja Subramanian bfc4f19c74 Guard against bad quality in trackInfo (#2271) 2023-11-28 22:28:30 +05:30
Raja Subramanian 396371312b Use variables for score -> quality mapping (#2268)
* Use variables for score -> quality mapping

* spelling
2023-11-28 11:51:21 +05:30
Raja Subramanian 5f76d1adcc Introduce DISCONNECTED connection quality. (#2265)
* Introduce `DISCONNECTED` connection quality.

Currently, this state happens when any up stream track does not
send any packets in an analysis window when it is expected to send
packets.

This can be used by participants to know the quality of a potentially
disconnected participant. Previously, it took 20 - 30 seconds for
the stale timeout to kick in and disconnect the limbo participant which
triggered a participant update through which other participants knew
about it.

Previously, `POOR` quality was also overloaded to denote that the
up stream is not sending any packets. With this change, that is a
separate indicator, i. e. `DISCONNECTED`.

* clean up

* Update deps

* spelling
2023-11-27 23:06:53 +05:30
Raja Subramanian 53542b09a0 Participant traffic load. (#2262)
* Participant traffic load.

Capturing information about participant traffic
- Upstream/Downstream
- Audio/Video/Data
- Packets/Bytes

This captures a notion of how much traffic load a participant is
generating.

Can be used to make allocation decisions.

* Clean up

* SIP patches

* reporter goroutine

* unlock

* move traffic stats from protocol

* check type
2023-11-26 23:05:00 +05:30
Raja Subramanian 8c3ec742e6 Use now for end time (#2248)
* Use now for end time

* less arithmetic
2023-11-17 12:00:47 +05:30
Raja Subramanian c62382c76e Clean up restart a bit. (#2247) 2023-11-17 00:40:00 +05:30
Raja Subramanian 49ca1066cd Reduce logging (#2243)
* Reduce logging

1. Do not print rtp stats if nil. Means that some subscribed tracks may
   not have any logs (very short subscriptions which end before any
   packet is sent).
2. Log ICE candidates only at the end, not when ICE connects. That logs
   the selected ICE candidate pair.
3. Log ICE candidates only if not empty.

* Update some deps
2023-11-15 12:57:03 +05:30
Raja Subramanian 440f00bcac Declare audio inactive if stale. (#2229)
* Declare audio inactive if stale.

Stale samples were used to declare audio active.
Maintain last update time and declare inactive if samples are stale.

* correct comment

* spelling

* check level in test
2023-11-08 11:13:39 +05:30
Raja Subramanian 12a9d74acb Do not restart on receiver side. (#2224)
* Do not restart on receiver side.

Restart with wrap back causes issues in the forwarding path
as the subscriber assumes the extended type from receiver side does
not restart.

Restart was an attempt to include as many packets as possible, but
in practice is not super useful. So, taking it out. Can clean up
a bit more stuff, but want to run this first and check for any oddities.

* fix test
2023-11-06 10:41:56 +05:30
cnderrauber f5047ab653 Check request layer for DD selector sync (#2219) 2023-11-03 18:40:36 +08:00
cnderrauber f247b68ed6 Make sure dd selector uses correct keyframe to select packets (#2218)
* Make sure dd selector uses correct keyframe to select packets

* Fix test case

* remove unsed field
2023-11-03 17:49:02 +08:00
Raja Subramanian a7a227709a Prevent out-of-bounds access. (#2216)
* Prevent out-of-bounds access.

Don't know which codec causes a spatial layer three access.
Returning nil and also logging so that we know the trackID of offending
track.

* spelling
2023-11-02 12:33:02 +05:30
Raja Subramanian 0bdfdb0c49 Squelching DD reader error. (#2215)
Squelching Structure is nil error as it can happen on packets
received before a key frame is received.
2023-11-02 11:10:28 +05:30
Paul Wells a6ede46adc add bounds check to dependency descriptor loop (#2214) 2023-11-01 18:04:04 -07:00
Raja Subramanian 45346d7c76 Clean up condition that is not happening (#2207) 2023-11-01 15:17:09 +05:30
Raja Subramanian c93a88bd9b Log starts on metadata cache overflow. (#2206) 2023-11-01 10:55:07 +05:30
Raja Subramanian 33a629065d Fix pre-extended value on wrap back restart. (#2202)
When wrapping back on a restart, was not setting pre-extended
values properly. It was missing a cycle. That caused rare large
sequence number gap.
2023-10-31 01:10:47 +05:30
Raja Subramanian 7578f9cbf5 Mark packet relevant only when current layer is valid. (#2200)
* Mark packet relevant only when current layer is valid.

Else, it introduces a large sequence number gap.

* fix tests
2023-10-30 00:34:17 +05:30
Raja Subramanian b5dd020ff7 Forgot to commit change from last PR discussion (#2198)
* Forgot to commit change from last PR discussion

* fix test case

---------

Co-authored-by: cnderrauber <zengjie9004@gmail.com>
2023-10-27 15:36:09 +05:30
Raja Subramanian a64bd23b6d Do server PLI when sync is required. (#2197)
* Do server PLI when sync is required.

A few changes
- Run key frame requester goroutine always. Runs every 200 ms which is
  not bad.
- Post a key frame request when server knows it needs one, like after an
  allocation. This ensures that the initial request is not delayed.
- Periodic check will ensure PLI for cases like all frame chains of a
  dependency descriptor being broken.

* simplify
2023-10-27 15:16:39 +05:30
cnderrauber 1f0ba21854 Fix svc: Drop frame is earlier than current keyframe (#2196)
* Fix svc: Drop frame is earlier than current keyframe

* Log detail of dependencydescriptor
2023-10-27 13:57:03 +08:00
Raja Subramanian 8b16db2270 Log PLI requests. (#2194)
A few things
- Log PLI requests from client.
- Pass in marker to RTP munger as SVC can insert marker.
- Adjusting first packet time should be aware of SVC as there is single
  stream in SVC
2023-10-26 21:07:36 +05:30