* agent: release JobTerminate handler when job ends before registration
JobRequest registered the job's JobTerminate handler after AssignJob
returned, so an ended UpdateJobStatus or worker disconnect handled in
between deregistered nothing and the handler leaked. AssignJob also sent
the assignment before recording the job as running, so an immediate
ended update was dropped.
Record the job before sending the assignment, and after registering the
handler recheck under h.mu that the job is still running and its worker
still registered, deregistering otherwise.
Fixes#4901
* replace utils.CloneProto with proto.CloneOf
* agent: wait on worker readiness and poll for released handlers in test
* Rework node stats a bit.
Related protocol PR - https://github.com/livekit/protocol/pull/1023
- Make a config for node stats measurements. Wanted to put the config in
`routing` package, but a circular dependency forced me to put in
config.go
- Make rate calculations explicit, i. e. requested via config.
Previously, it had some odd checks to decide when to calculate rate
and it would have been calculating over different windows.
- Report signal/data channel bytes every 5 seconds to stats collection
module. Previously, it was doing it every 30 seconds and that meant
some windows could have had a large spike
NOTE: Still need to think about this for load calculations as a large
number of participants leaving could flush in a small window and that
could report a large spike in bytes/packets. Maybe need to ignore
signal bytes for load calculation?
* deps
* use default node stats config if given config is nil
* split out node stats into a struct for re-use
* update config
* cli: Allow setting the current node region with flag or env variable
Also add region to "starting LiveKit server" log.
* routing: Add region to node registration
Register the node's region on the selected router so it can be used for
region aware node selection.
Also add the region to the list-nodes output.
* regionaware: Set minDist to zero for the current node
If you don't set the minDist when leaving the loop early for a node that
matches the current region, the minDist value with still be at max. This
causes the the wrong node to be selected if the current node is the
first one the loop passes through.
Add a test that validates this change. The new test fails if this new
change is not in place.
* prestop and hasparticipants in interface
* add prestop function to existing routers
* fakerouter prestop
* update protocol version
* read lock
* redis router graceful stop
* test fix
* force stop
Paul Wells