* Prometheus metric for join latency.
Also including a couple of other failures in the signal connection path
and moving the signal connected to after all that.
Not doing counters for the new signal failure paths. I should not have
done for the other two I added a little while ago also (
validation failure and start participant failure) as those are not
scalable to keep adding to node stats. Will probably remove those two
from node stats later. Can add those counters if they are useful.
* deprecate signal failed counters
* Metrics for participant active, i. e. fully established.
- Egress stub for v2 API
- Fix the participant canceled counter 🤦
- Add active counter -> this is increment when a participant becomes
active, i. e. primary peer connection established. Can be used to
monitor node wise connection establishment issues.
- Add singnalling validation fail counter.
With this, we have
- signalling validation fail
- signalling failed --> this is when the `startSession` fails
- signalling connected -> signalling is succesful and can send back
joinResponse to client
on media connection side
- rtc_init -> start
- rtc_connected -> participant session created (joined)
- rtc_active -> primay peer connection established
- rtc_canceled -> could not proceed with RTC connection due to not being
able to resume.
* signalling counters deps
* revert pion/webrtc to 4.2.12 to get SCTP without interleaving
* go back to pion/webrtc 4.2.11 and sctp 1.9.5
When a client hits /rtc/v[01]/validate with a base64 WrappedJoinRequest
whose embedded JoinRequest.ClientInfo is unset, validateInternal called
AugmentClientInfo with a nil *ClientInfo and panicked at ci.Address =
GetClientIP(req). The non-wrapped branch already allocates via
ParseClientInfo; do the same here so pi.Client always gets at least the
resolved client Address.
* Log join duration.
Also revert the "unresolved" init. Defeated the purpose of log resolver
as it was resolving with those values even if not forced. Instead set it
to "unresolved" if not set when forced.
Join duration is not reset if resolver is reset as that happens on
moving a participant and there is no new join duration in that case.
* explode
Release to Docker / docker (push) Failing after 3m42s
* Key telemetry stats work using combination of roomID, participantID
With forwarded participant, the same participantID can existing in two
rooms.
NOTE: This does not yet allow a participant session to report its
events/track stats into multiple rooms. That would require regitering
multiple listeners (from rooms a participant is forwarded to).
* missed file
* data channel stats
* PR comments + pass in room name so that telemetry events have proper room name also
* Make new path for signalling v1.5 support.
To be able to support newer clients to interact with older servers, move
signalling v1.5 to new path (`/rtc1`). On the new path, `join_request`
is required and single peer connection is used.
With the existing path `/rtc`, single peer connection is still supported
if `join_request` is used.
Newer clients connecting to old server should follow
1. Try new path WebSocket
2. If that fails, try new path validate at `/rtc1/validate`.
3. If the above gets a 404 which will happen with older server, revert
back to old path and signalling 1.0.
Open to suggestions on path name.
* test on both paths
* change path from /rtc1 -> /rtc/v1
* test all rtc service path combinations
* WIP
* check using protocol version
* revert
* clean up
* sdp cid argument
* WIP
* WIP
* test
* clean up
* clean up
* fixes
* clean up
* clean up
* clean up
* conditional checks
* tests for both dual and single peer connection
* test
* test
* test
* type check
* test
* todo
* munges
* combined config
* populate mid
* limit to receive only
* clean up
* clean up
* clean up
* older test
* clean up
* alternative audio codec
* dtx
* don't need to copy
* Anunay feedback
* use the available peer connection
* publisher check
* WIP
* WIP
* WIP
* no mid
* media sections requirement
* mage generate
* WIP
* WIP
* set data channel receive size for test
* handle early media better
* WIP
* do not do ICERestart if no subscriber
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* start up subscriber RTCP worker
* WIP
* WIP
* clean up
* clean up
* flag to indicate use of single peer connection
* remove unused interface method
* clean up
* clean up
* Jie feedback #1
* deps
* do not access subscriber in one shot mode
* more places for one shot mode
* more one shot fixes
* deps
* deps
* test
* Add Moving participant to another room
it is implemented in cloud only since the destination
room can exist in different node with the source room
* Update pkg/service/errors.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* rename
* test panic
* fake LocalParticipantHelper
* revert delete line
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Rework node stats a bit.
Related protocol PR - https://github.com/livekit/protocol/pull/1023
- Make a config for node stats measurements. Wanted to put the config in
`routing` package, but a circular dependency forced me to put in
config.go
- Make rate calculations explicit, i. e. requested via config.
Previously, it had some odd checks to decide when to calculate rate
and it would have been calculating over different windows.
- Report signal/data channel bytes every 5 seconds to stats collection
module. Previously, it was doing it every 30 seconds and that meant
some windows could have had a large spike
NOTE: Still need to think about this for load calculations as a large
number of participants leaving could flush in a small window and that
could report a large spike in bytes/packets. Maybe need to ignore
signal bytes for load calculation?
* deps
* use default node stats config if given config is nil
* split out node stats into a struct for re-use
* update config
While working on a different service, noticed delegation of setting up
routes to the service itself. So, making that change and making some
methods internal only as there is no need to export them outside the
service.
* Debug FF TWCC
* - TURN off TWCC for Firefox. Seems to fail with VP9 send, i.e. there are
no TWCC feedback packets when sending VP9.
- Relax thresholds for congestion as staging data is showing
oscillations.
- Clean up some logging.
* debug log a few more signal messages
* revert config
* revert config
* clean up
* Handle room configuration that's set in the grant itself
* ensure refresh token contains updates
* deps
* dep
---------
Co-authored-by: Paul Wells <paulwe@gmail.com>
* Ref count the stats worker.
NOTE: Don't liek this much, but wanted to open this get some 👀 on
this and get feedback.
There are two entities, one for counting signal bytes and another for
media stats. They both send `ParticipantJoined` and `ParticipantLeft`
event.
In the case of a participant resume, as the old web socket
connection is closed, that triggers a signal stats counter close. That
would call `ParticipantLeft` and that would close the stats worker.
The closed stats worker got reaped in `FlushStats` after three minutes.
So, all events after that did not have a worker and hence went
unreported including missing participant_left webhook because it relied
on checking if a participant was ever connected and that needed to check
the worker state.
Using a ref count to keep track of join/leaves. And not close the worker
until ref count goes down to 0.
* create a stats worker on resume
* revert incorrect changes
* transfer connected state
* transfer connected state when creating worker
* resolve participant on a resume
* exponential backoff when calling CreateRoom
* dz+bb review: check ctx cancellation, remove retry max
* bb review: fix loop condition
* raja review: use timer, bring back max tries
- Store agent dispaches independently of room agents on rtc.Room
- Serialize agent dispatches in rtc.Room
- Support for agent dispatch and job serialization in redis
The agent Job object references denormalized Room and ParticipantInfo object. When storing Jobs, this sets the Room to nil, and only stores the Participant identity field. When read back, these fields need to be set to their current value.
