* Log cleanup pass
Demoted a bunch of logs to DEBUG, consolidated logs.
* use context logger and fix context var usage
* moved common error types, fixed tests
* Restrict scope of negotiation time out error logs
1. Log "negotiation failed" only if signal channel was active
within half window of negotiation timeout. Negotiation timeout currently
is at 15 seconds. Signal pings are every 10 seconds.
2. In transport.go, do not report negotiation timed out and do not
callback negotiation failure if the peer connection state is not
connected. Goal of negotiation failure tracker is to take remedial
action when an in-session negotiation fails. Seeing a bunch of cases
of the case hitting even without ICE connection forming. Negotiation
timer is not intended for those cases.
* fix test
Not super useful. It does happen a bunch of times especially at lower
end of estimate where the next layer up is high. We have to probe
anyway. Effects of large jumps have been mitigated by doing it for short
time.
1. When re-allocating for a track in DEFICIENT state, try to use
available headroom to accommodate change before trying to steal
bits from other tracks.
2. If the changing track gives back bits (because of muting or
moving to a lower layer subscription), use the returned bits
to try and boost deficient track(s).
* Plug a couple of holes in stream transitions.
1. Missed negative sign meant stealing bits from other tracks was not
working.
2. When a track change (mute, unmute, subscription change) cannot be
allocated, explicitly pause so that stream state update happens.
Refactor stream state update a bit to make it a bit cleaner.
* correct comment
* Check for request layer lock only in the goroutine
* check before sending PLI
* max layer notifier worker
* test cleanup
* clean up
* do notification in the callback
* Make congestion controller probe config
* Wait for enough estimate samples
* fixes
* format
* limit number of times a packet is ACKed
* ramp up probe duration
* go format
* correct comment
* restore default
* add float64 type to generated CLI
- Increase max interval between probes to 2 minutes.
- Use a minimum probe rate of 200 kbps. This is to ensure that
the probe rate is decent and can produce a stronger signal.
* Simplify sliding window collapse.
Keep the same value collapsing simple.
Add it to sliding window as long as same value is received for longer
than collapse threshold.
But, add a prune with three conditions to process the siliding window
to ensure only valid samples are kept.
* flip the order of validity window and same value pruning
* increase collapse threshold to 0.5 seconds during non-probe
1. Probe end time needs to include the probe cluster running time also.
2. Apply collapse window only within the sliding window. This is to
prevent cases of some old data declaring congestion. For example,
an estimate could have fallen 15 seconds ago and there might have
been a bunch of estimates at that fallen value. And the whole
sliding window could have that value at some point. But, a further
drop may trigger congestion detection. But, that might be acting too
fast, i. e. on one instance of value fall. Change it so that we
detect if there is a fall within the sliding window and apply
collapse based on that.
On a state change, it was possible an aborted probe was pending
finalize. When probe controller is reset, the probe channel
observer was not reset. Create a new non-probe channel observer
on state change to get a fresh start.
Also limit probe finalize wait to 10 seconds max. It is possible
that the estimate is very low and we have sent a bunch of probes.
Calculating wait based on that could lead to finalize waiting for
a long time (could be minutes).
* Simplify probe done handling.
Seeing a case where the channel abserver is not re-created after
an aborted probe. Simplifying probe done (no callbacks, making it
synchronous).
* log more
* Split probe controller from StreamAllocator.
With TWCC, there is a need to check for probe status
in a separate goroutine. So, probe specific stuff need
locking. Split out the probe controller to make that cleaner.
* remove defer
* A coupke of stream allocator tweaks
- Do not overshoot on catch up. It so happens that during probe
the next higher layer is at some bit rate which is much lower
than normal bit rate for that layer. But, by the time the probe
ends, publisher has climbed up to normal bit rate.
So, the probe goal although achieved is not enough.
Allowing overshoot latches on the next layer which might be more
than the channel capacity.
- Use a collapse window to record values in case of a only one
or two changes in an evaluation window. Some times it happens
that the estimate falls once or twice and stays there. By collapsing
repeated values, it could be a long time before that fall in estimate
is processed. Introduce a collapse window and record duplicate value
if a value was not recorded for collapse window duration. This allows
delayed processing of those isolated falls in estimate.
* minor clean up
* add a probe max rate
* fix max
* use max of committed, expected for max limiting
* have to probe at goal
When detecting congestion based on loss, it is possible that
the loss based signal triggers earlier and the estimate based
signal is lagging. In those cases, check against last received
estimate and if that is lower than loss based throttling, use that.
Without this, it was possible that the current usage high.
Loss based throttling may not dial things back far enough to pause
the stream. Ideally, congestion should hit again and it should be dialled
down further and eventually pause, but there are situations it never
dials back far enough to pause.
* Support simualting subscriber bandwidth.
When non-zero, a full allocation is triggered.
Also, probes are stopped.
When set to zero, normal probing mechanism should catch up.
Adding `allowPause` override which can be a connection option.
* fix log
* allowPause in participant params
