* Deregister agent job when the server terminates it
JobTerminate ended the job on the worker but left its JobTerminate
handler and jobToWorker entry registered. Those were only cleaned up
when the worker later sent an ended job status, or on worker disconnect
for jobs still running. Workers that do not report job status after a
termination (agents-js never sends UpdateJobStatus) leaked a psrpc
handler for every job terminated by the server, e.g. each time an agent
participant left its room.
* Wait for worker registration in JobTerminate test
The worker's registration response is sent before its job request topic
is registered, so wait for the WorkerRegistered event before requesting
jobs. Also cover a worker reporting an ended status after the server
terminated the job, and clarify the JobTerminate comment.
* agent: release JobTerminate handler when job ends before registration
JobRequest registered the job's JobTerminate handler after AssignJob
returned, so an ended UpdateJobStatus or worker disconnect handled in
between deregistered nothing and the handler leaked. AssignJob also sent
the assignment before recording the job as running, so an immediate
ended update was dropped.
Record the job before sending the assignment, and after registering the
handler recheck under h.mu that the job is still running and its worker
still registered, deregistering otherwise.
Fixes#4901
* replace utils.CloneProto with proto.CloneOf
* agent: wait on worker readiness and poll for released handlers in test
This is not all of it as it is not possible (or at least I do not know
of a way) to get all suggestions for a repo/project. Did this via loop
searching mainly and taking the modernize suggestions.
* select the least loaded agent worker for job dispatch
* update to load balance using inverse load
* remove unused file
* adding unit tests for worker job distribution
Martin Njalale