Use by cloud simulated tracks.
Introduces a lock, but it is used only in the background worker (where
it will be almost always uncontested) and when API GetStats is accessed.
Does not touch the per-packet Update path.
* Move ForwardStats aggregation off the packet forwarding path
ForwardStats is a process-wide singleton and its Update runs for every
forwarded packet. It previously took a shared mutex, updated a windowed
aggregate, and observed into a global Prometheus histogram on every call.
Update now only buffers the transit sample into a sharded lock-free ring
(one atomic add to reserve a slot, one atomic store to publish). A
background worker drains the ring every summary interval, observes each
sample into the histogram (per-packet fidelity retained) and folds the
interval summary into a window ring for the latency/jitter gauges. Under
sustained overload the oldest excess samples are dropped and counted, and
the count is logged.
Ring slots are atomic.Int64 so producer store / consumer load are
synchronized (race-clean). Capacity is numShards*shardCap = 131072
samples; at the default 50ms summary interval that sustains ~2.6M
samples/s before dropping.
Benchmark: BenchmarkForwardStatsUpdate (0 allocs/op both), Update per call:
cores before after speedup
1 ~8.6 ns ~1.5 ns ~6x
8 ~175 ns ~22 ns ~8x
The before path (mutex + windowed Welford aggregate + per-packet histogram
observe) degrades ~20x from 1 to 8 cores under contention; the after path
does not block and stays an order of magnitude lower under load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix ring publish race and report-path double-flush
Addresses two review findings on the forward-stats sample buffer.
1. Publish race (reserve-before-store): push reserved a ring slot by
advancing writeIdx and stored the value afterwards, so drain could read a
slot the producer had reserved but not yet written, getting a stale/zero
value; the producer's later store then landed behind the read cursor and
was lost. Each slot now carries a publish epoch. push stores the value and
then publishes seq = index+1. drain reads a slot only when seq == r+1; a
reserved-but-unpublished slot at the cursor stops the drain and is picked
up on the next call, so no sample is read stale or lost. A slot overwritten
during the read is detected on re-check and counted as dropped.
The windowed latency/jitter stats did not permanently drift even before
this change (report recomputes from a fixed-size ring that overwrites, not
discounts, old buckets, so any error aged out within the report window),
but these values feed the capacity manager, so the buffer is made exact.
2. Report-path double-flush: run() flushed on both the summary tick and the
report tick, and every flush advances the window ring, so the ring cycled
faster than intended and the effective window was shorter than configured
(~5% at 50ms/1s/1m). The report tick no longer flushes; the summary ticker
keeps the ring current to within one summary interval.
Benchmark, Update per call (0 allocs/op), 1 and 8 cores:
after fixes: ~5.5 ns / ~29 ns
before fixes (this branch): ~1.5 ns / ~22 ns
baseline (mutex + per-packet histogram): ~8.6 ns / ~175 ns
The publish epoch adds one atomic store to push; the path stays non-blocking,
zero-allocation, and well below the baseline under contention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Forwarding latency measurement tweaks.
- prom transmission type public
- do not measure short term values as it is not used and saves some lock
contention time in packet path potentially. Adding a separate method
for that.
- Change latency/jitter summary reporting to `ns` also to match the
histogram.
* add GetShortStats
* Higher resolution forwarding latency histogram.
Was using the average latency/jitter of last second to populate
forwarding latency/jitter histogram. But, it is too coarse, i. e. the
average value of latency/jitter is very low and those summarised samples
end up in the lowest bucket always.
A few things to address it
- record per packet forwarding latency in histogram
- adjust histogram bins to include smaller values
- Drop jitter histogram
This is a per packet call, but prometheus histogram is supposedly
fast/light weight. Would be good to get better resolution histograms.
Hence doing this. Please let me know if there are performance concerns.
* typo
* one more typo
* Tweaks tresholds for logging high forwarding latency/jitter.
Previous attempt showed skewed jitter (i. e. more than 10x latency),
But, no large latency.
So, reducing the latency treshold to declare high latency.
And also keeping track of lowest/highest per reporting window and
logging those along with short term and long term measurements.
NOTE: previously short term and long term were separate calls with locks
acquired. Now, it is all in one lock. So, it does increase the lock
duration a bit, but hopefully not by too much as the welford merge for
short term would go over 20 samples (at 50 ms sampling interval and 1 s
reporting window).
* revert skew factor
* Log some information around high forwarding latency.
Latency is not 0 after switching to microseconds resolution.
But, still seeing high jitter. Logging a bit more to understand under
what conditions it happens.
More notes inline.
* compact
Latency is always 0, but jitter is high.
Not sure how that happens as latency is the welford mean and jitter is
welford standard deviation. Feels like some mis-labeling.
Anyhow, switching to microseconds units to get better resolution.
* Handle cases of long mute/rollover of time stamp.
There are cases where the track is muted for long enough for timestamp
roll over to happen. There are no packets in that window (typically
there should be black frames (for video) or silence (for audio)). But,
maybe the pause based implementation of mute is causing this.
Anyhow, use time since last packet to gauge how much roll over should
have happened and use that to update time stamp. There will be really
edge cases where this could also fail (for e. g. packet time is affected
by propagation delay, so it could theoretically happen that mute/unmute
+ packet reception could happen exactly around that rollover point and
miscalculate, but should be rare).
As this happen per packet on receive side, changing time to `UnixNano()`
to make it more efficient to check this.
* spelling
* tests
* test util
* tests
Raja Subramanian