Reverts #4816. Since livekit/cloud#4747 a room re-key closes every
participant's stats worker under the old room id and opens a new one
under the new id (LeaveSession/JoinSession, #4917), so RoomIDChanged
and reKeyRoom have nothing left to move, and with them go the sealed
stats batches, SetRoom, ForceClose and the reference hand-over that
only the re-key collision case needed. The worker map goes back to a
flat (room, participant) key.
The guard fixes that landed after #4816 (#4817, #4856, #4860) are kept.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* telemetry: support roomID change for a participant
A room can get a new id while participants are connected. Key stats
workers as map[roomID]map[participantID] so moving a room is a single
map splice, and add reKeyRoom/RoomIDChanged to do the move.
Stats collected before the change are sealed off with the room they
were collected in so they stay attributed to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* telemetry: close superseded worker on re-key collision
Only one worker can be keyed at (room, participant). If a re-key lands
on a room that already has a worker for the same participant, keep the
one already filed there and close the superseded one so it drains and
is reaped instead of lingering in the flush list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* telemetry: hand references to the successor on force close
A ReferenceGuard records that it activated some worker, not which one,
so a superseded worker cannot just drop its references - the survivor
would be left with references it never sees released and would never
close. Hand them over instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* report a reason on room-ended telemetry
Room.Close already took a ParticipantCloseReason, but OnClose was func()
with no arguments, so the reason was structurally dropped before it could
reach RoomEnded.
Add types.RoomCloseReason and carry it through Close -> OnClose -> RoomEnded,
where it lands on both the analytics event and the room_finished webhook.
Room.Close now takes the room reason and derives the participant reason from
it, so the two can never disagree. ToParticipantCloseReason maps each reason
to exactly what its call site passed before, so no participant-facing
behaviour changes; a table test pins that mapping.
* fix(test): update webhook test for RoomCloseReason
test/ was outside the packages checked before pushing, so this Close call
site was missed. Also assert the reason reaches the room_finished webhook,
which the unit test cannot cover since it runs with a nil notifier.
* Fix publish track count on migration in.
https://github.com/livekit/livekit/pull/4707 addressed the case of
publish tracks overcounting due to synthesised track publish on migrate
in. But, it introduced an issue where published tracks count could go
negative because unpublish subtracted the counter irrespective of the
track actually migrated in or not.
Fix it by keeping track of local publish.
Also, the older code was skipping publisher track count increase if the
synthesised publish was handled first. Address it by checking if the
track is actually new (i. e. fresh local publish) when the track was
already created in the migrate in path.
* fix pub time for tracks published after migration
* test
* prevent multiple track egresses
* pass room proto directly to telemetry events
* Keep telemetry analytics events on a minimal room, gate full room in webhooks
---------
Co-authored-by: Simon Beeli <simon.beeli@gmx.ch>
- Count a publish attempt on a migrating in tarck as there is no
AddTrack for that.
- Add cancel publish only if the participant connection is canceled
- Do not add publish counter for synthetic publish attempts which
happens for migrating in tracks. It will be counted on migrating in
node when the track is actually published, i. e. negotiated/packets
flowing.
* Metrics for participant active, i. e. fully established.
- Egress stub for v2 API
- Fix the participant canceled counter 🤦
- Add active counter -> this is increment when a participant becomes
active, i. e. primary peer connection established. Can be used to
monitor node wise connection establishment issues.
- Add singnalling validation fail counter.
With this, we have
- signalling validation fail
- signalling failed --> this is when the `startSession` fails
- signalling connected -> signalling is succesful and can send back
joinResponse to client
on media connection side
- rtc_init -> start
- rtc_connected -> participant session created (joined)
- rtc_active -> primay peer connection established
- rtc_canceled -> could not proceed with RTC connection due to not being
able to resume.
* signalling counters deps
* revert pion/webrtc to 4.2.12 to get SCTP without interleaving
* go back to pion/webrtc 4.2.11 and sctp 1.9.5
Release to Docker / docker (push) Failing after 3m42s
* Key telemetry stats work using combination of roomID, participantID
With forwarded participant, the same participantID can existing in two
rooms.
NOTE: This does not yet allow a participant session to report its
events/track stats into multiple rooms. That would require regitering
multiple listeners (from rooms a participant is forwarded to).
* missed file
* data channel stats
* PR comments + pass in room name so that telemetry events have proper room name also
Published counter was bumped up only when not migrating in, but it was
decremented when a migrating participant leaves without expectation to
resume. That could have resulted in negative counts.
Always change counters irrespective of migration or expected to resume
on leave. Control events send based on migration/resume.
* Send `participant_connection_aborted` when participant session is closed
without becoming `ACTIVE`.
There is one sticky case. If there is a migration and the migration
fails, this will send `participant_connection_aborted` even though the
participant may have connected properly on the previous node.
* depsg
* Normalize mime type and add utilities.
An attempt to normalize mime type and avoid string compares remembering
to do case insensitive search.
Not the best solution. Open to ideas. But, define our own mime types
(just in case Pion changes things and Pion also does not have red mime
type defined which should be easy to add though) and tried to use it everywhere.
But, as we get a bunch of callbacks and info from Pion, needed conversion in
more places than I anticipated. And also makes it necessary to carry
that cognitive load of what comes from Pion and needing to process it
properly.
* more locations
* test
* Paul feedback
* MimeType type
* more consolidation
* Remove unused
* test
* test
* mime type as int
* use string method
* Pass error details and timeouts. (#3402)
* go mod tidy (#3408)
* Rename CHANGELOG to CHANGELOG.md (#3391)
Enables markdown features in this otherwise already markdown'ish formatted document
* Update config.go to properly process bool env vars (#3382)
Fixes issue https://github.com/livekit/livekit/issues/3381
* fix(deps): update go deps (#3341)
Generated by renovateBot
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Use a Twirp server hook to send API call details to telemetry. (#3401)
* Use a Twirp server hook to send API call details to telemetry.
* mage generate and clean up
* Add project_id
* deps
* - Redact requests
- Do not store responses
- Extract top level fields room_name, room_id, participant_identity,
participant_id, track_id as appropriate
- Store status as int
* deps
* Update pkg/sfu/mime/mimetype.go
* Fix prefer codec test
* handle down track mime changes
---------
Co-authored-by: Denys Smirnov <dennwc@pm.me>
Co-authored-by: Philzen <Philzen@users.noreply.github.com>
Co-authored-by: Pablo Fuente Pérez <pablofuenteperez@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Paul Wells <paulwe@gmail.com>
Co-authored-by: cnderrauber <zengjie9004@gmail.com>
* Use a Twirp server hook to send API call details to telemetry.
* mage generate and clean up
* Add project_id
* deps
* - Redact requests
- Do not store responses
- Extract top level fields room_name, room_id, participant_identity,
participant_id, track_id as appropriate
- Store status as int
* deps
On a resume, the signal stats will call `ParticipantLeft`. Although, it
explicity says not to send events, it could still close the stats
worker.
To handle that, we created a stats worker if needed in
`ParticipantResume` notification in this PR
(https://github.com/livekit/livekit/pull/2982), but that is not enough
as that event could happen before previous signal connection closes the
stats worker.
A new stats worker does get created when `ParticipantJoined` is called
by the new signal connection, but it does not transfer connected state.
So, when the client leaves, `ParticipantLeft` is not sent.
I am not seeing why we should not transfer connected state always given
that it is the same participant SID/session. But, I have a feeling that
I am missing some corner case. Please let me know if I am missing
something here.
* Ref count the stats worker.
NOTE: Don't liek this much, but wanted to open this get some 👀 on
this and get feedback.
There are two entities, one for counting signal bytes and another for
media stats. They both send `ParticipantJoined` and `ParticipantLeft`
event.
In the case of a participant resume, as the old web socket
connection is closed, that triggers a signal stats counter close. That
would call `ParticipantLeft` and that would close the stats worker.
The closed stats worker got reaped in `FlushStats` after three minutes.
So, all events after that did not have a worker and hence went
unreported including missing participant_left webhook because it relied
on checking if a participant was ever connected and that needed to check
the worker state.
Using a ref count to keep track of join/leaves. And not close the worker
until ref count goes down to 0.
* create a stats worker on resume
* revert incorrect changes
* transfer connected state
* transfer connected state when creating worker
* resolve participant on a resume
UpdateSubscription had a shortcoming where when it couldn't find the
participant, it ignored the request.
This PR further removes the reliance of current publisher state from
subscribers.
- SubscribeToTrack only takes in a trackID
- Introduced RoomTrackManager to maintain all published tracks to a room
- Added TrackUnpublished event to clearly indicate when a track has been removed
- SubscribeRequested event no longer include information about the publisher
Added a new manager to handle all subscription needs. Implemented using reconciler pattern. The goals are:
improve subscription resilience by separating desired state and current state
reduce complexity of synchronous processing
better detect failures with the ability to trigger full reconnect
Related to livekit/protocol#273
This PR adds:
- ParticipantResumed - for when ICE restart or migration had occurred
- TrackPublishRequested - when we initiate a publication
- TrackSubscribeRequested - when we initiate a subscription
- TrackMuted - publisher muted track
- TrackUnmuted - publisher unmuted track
- TrackPublish/TrackSubcribe events will indicate when those actions have been successful, to differentiate.
* initial commit
* add correct label
* clean up
* more cleanup on adding stats
* cleanup
* move things to pub and sub monitors, ensure stats are correctly updated
* fix merge conflict
* Fix panic on MacOS (#1296)
* fixing last feedback
Co-authored-by: Raja Subramanian <raja.gobi@tutanota.com>
In certain scenarios such as migration, we do not want a duplicate event
to be sent when the participant is reconnecting. The Prometheus metric
should still be updated though.
