mirror of
https://github.com/livekit/livekit.git
synced 2026-08-27 22:34:25 +00:00
add flexFEC publisher to SFU support & test scripts
This commit is contained in:
David Chen
@@ -0,0 +1,2 @@
|
||||
# FEC test run artifacts (logs, CSVs, prom samples, reports, built binaries)
|
||||
out/
|
||||
@@ -0,0 +1,135 @@
|
||||
# FlexFEC test harness (publisher → SFU)
|
||||
|
||||
Validates that FlexFEC-03 sent by a publisher recovers lost packets at the SFU
|
||||
under cellular-like loss (continuous base loss plus periodic bursts), the kind
|
||||
of uplink a robot on LTE/5G sees. The harness runs everything on loopback:
|
||||
|
||||
```
|
||||
publisher (rust-sdks local_video, --test-pattern --flex-fec)
|
||||
│ UDP → 127.0.0.1:7882 ← traffic shaper drops packets here
|
||||
▼
|
||||
livekit-server (enable_flexfec: true, prometheus on :6789)
|
||||
│
|
||||
▼
|
||||
subscriber (rust-sdks local_video, --headless --log-frames)
|
||||
```
|
||||
|
||||
The publisher attaches a wall-clock timestamp and a frame id to every frame
|
||||
via the packet trailer feature; the subscriber logs them per received frame.
|
||||
That gives ground truth for end-to-end frame latency and frame loss, while
|
||||
the SFU's prometheus counters (`livekit_flexfec_packet_total{state=...}`,
|
||||
`livekit_nack_total`, `livekit_packet_loss_total`) show what FEC did.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `go`, `cargo`, `curl`, `python3` with `matplotlib` (`pip3 install matplotlib`)
|
||||
- a `rust-sdks` checkout with the local_video example
|
||||
(default `../rust-sdks` relative to this repo, override with `RUST_SDKS_DIR`)
|
||||
- `sudo` for traffic shaping:
|
||||
- **macOS**: dummynet (`dnctl` + `pfctl`). If dummynet is unavailable on
|
||||
your macOS build, run with `--no-shaping` and use Network Link Conditioner
|
||||
manually, or test on Linux.
|
||||
- **Linux**: `tc` with the `netem` qdisc (`iproute2`).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# A/B comparison: baseline (NACK only) vs FlexFEC, 2 minutes each
|
||||
./run_fec_test.sh --mode ab --duration 120
|
||||
|
||||
# single FEC run with a harsher profile
|
||||
./run_fec_test.sh --mode fec --duration 60 --base-loss 0.05 --burst-loss 0.4
|
||||
|
||||
# sanity check without shaping (expect ~0 loss, ~0 recoveries)
|
||||
./run_fec_test.sh --mode fec --duration 30 --no-shaping
|
||||
|
||||
# no sudo available: drop 4% of received packets inside the SFU instead of
|
||||
# OS shaping (uniform loss only, also useful for CI)
|
||||
./run_fec_test.sh --mode ab --duration 60 --debug-drop 4
|
||||
```
|
||||
|
||||
Outputs land in `scripts/fec/out/<timestamp>/`:
|
||||
|
||||
- `fec_report.png` — stacked time series: per-frame latency with frame-gap
|
||||
markers and burst shading, FEC received/recovered/failed rates, NACK and
|
||||
packet-loss rates, delivered fps
|
||||
- `summary.txt` — per-run table (frames lost, latency percentiles, FEC
|
||||
counters, NACK totals) and the A/B comparison
|
||||
- per run (`baseline/`, `fec/`): `server.log`, `publisher.log`,
|
||||
`subscriber.log`, `frames.csv`, `prom.tsv`, `events.csv`, `meta.env`
|
||||
|
||||
## Loss profile
|
||||
|
||||
Defaults simulate a robot uplink over cellular: 2% continuous loss
|
||||
(Gilbert-Elliott on Linux for realistic correlation, uniform on macOS) with a
|
||||
3 s burst of 25% loss every 15 s. Tune via `--base-loss`, `--burst-loss`,
|
||||
`--burst-every`, `--burst-len`. The shaper matches **UDP destined to port
|
||||
7882** on loopback, which is the publisher→SFU media leg; the subscriber's
|
||||
upstream RTCP shares that port and is shaped too, which is acceptable for A/B
|
||||
comparisons since both runs see identical conditions.
|
||||
|
||||
The publisher defaults to a fixed 30% FEC protection rate with the bursty
|
||||
mask (`--fec-rate`, `--fec-mask-type`); pass `--fec-rate 0` to let libwebrtc
|
||||
adapt the rate to its loss estimate instead.
|
||||
|
||||
`--debug-drop PCT` is an unprivileged alternative to OS shaping: the SFU
|
||||
drops PCT% of received packets before processing (uniform, all SSRCs,
|
||||
enabled via the `LIVEKIT_DEBUG_RX_DROP_PCT` env var). Use the OS shapers for
|
||||
the cellular burst profile, this knob for quick checks and CI. Note that with
|
||||
uniform loss and a fixed protection rate, blocks losing two or more packets
|
||||
are not FEC-recoverable and fall back to NACK/RTX, so expect partial
|
||||
recovery; bursty loss with the bursty mask is the scenario FlexFEC targets.
|
||||
|
||||
## What to expect
|
||||
|
||||
- Sanity run (no shaping): `state="received"` grows, `recovered` ≈ 0, no
|
||||
frame gaps.
|
||||
- FEC run under bursts: `recovered` spikes inside burst windows, the
|
||||
subscriber sees few or no frame-id gaps, latency stays near baseline.
|
||||
- Baseline under the same bursts: frame gaps and latency spikes during
|
||||
bursts (NACK/RTX needs a round trip per loss; FEC repairs immediately).
|
||||
- Publisher bitrate should not collapse in the FEC run — FEC packets carry
|
||||
transport-wide CC sequence numbers and the SFU reports them, so the
|
||||
publisher's bandwidth estimate stays intact.
|
||||
|
||||
## Parameter sweep
|
||||
|
||||
`sweep_fec.sh` runs `run_fec_test.sh` across a matrix of loss levels and FEC
|
||||
configurations (one baseline plus one FEC run per `(rate, mask)` at each loss
|
||||
level), builds the binaries once, then `aggregate_fec.py` combines all cells
|
||||
into a single comparison: `sweep_report.png` (frame loss, p99 latency, FEC
|
||||
recovery rate, and recovered-packet count, each vs loss level, baseline vs
|
||||
every FEC config), `sweep_summary.csv`, and a markdown table.
|
||||
|
||||
```bash
|
||||
# uniform-loss sweep, no sudo: 4 loss levels x 2 FEC rates x 1 mask
|
||||
./sweep_fec.sh --loss-mode debug --loss-list "2 5 10 15" --fec-rate-list "20 50"
|
||||
|
||||
# cellular-burst sweep: vary burst intensity, compare the two masks at 30%
|
||||
sudo -v && ./sweep_fec.sh --loss-mode shaped --loss-list "0.15 0.30 0.50" \
|
||||
--fec-rate-list "30" --mask-list "random bursty" --duration 90
|
||||
|
||||
# re-aggregate an existing sweep without re-running it
|
||||
python3 aggregate_fec.py --sweep out/sweep_<ts>
|
||||
```
|
||||
|
||||
Runtime ≈ cells × (~20s setup + `--duration`). Each cell is a directory under
|
||||
the sweep output; `manifest.tsv` maps cell → parameters. In `debug` mode
|
||||
`--loss-list` is packet-drop percentages; in `shaped` mode it is the burst
|
||||
loss fraction (base loss and cadence fixed by `--base-loss`/`--burst-every`/
|
||||
`--burst-len`). Note the caveat above: uniform `debug` loss with a fixed FEC
|
||||
rate only recovers blocks that lose a single packet, so `shaped` bursts with
|
||||
the `bursty` mask show FlexFEC at its best.
|
||||
|
||||
## Direct script use
|
||||
|
||||
```bash
|
||||
sudo ./shape_macos.sh start --port 7882 --base-loss 0.02 --burst-loss 0.25 \
|
||||
--burst-every 15 --burst-len 3 --events /tmp/events.csv
|
||||
sudo ./shape_macos.sh stop # force cleanup (also: shape_linux.sh)
|
||||
|
||||
./prom_poll.sh 6789 /tmp/prom.tsv
|
||||
|
||||
# single run / A/B pair
|
||||
python3 plot_fec.py --run out/<ts>/baseline --run out/<ts>/fec --out out/<ts>
|
||||
```
|
||||
Executable
+212
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate a FlexFEC parameter sweep into a single comparison report.
|
||||
|
||||
Reads the cells produced by sweep_fec.sh (a manifest.tsv plus one run directory
|
||||
per cell) and emits:
|
||||
sweep_report.png - loss/latency/recovery vs loss level, baseline vs each config
|
||||
sweep_summary.csv - one row per cell with the key metrics
|
||||
a markdown table on stdout
|
||||
|
||||
Usage:
|
||||
aggregate_fec.py --sweep <sweep_dir>
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import sys
|
||||
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt # noqa: E402
|
||||
|
||||
# Run lives in plot_fec.py next to this script
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from plot_fec import Run # noqa: E402
|
||||
|
||||
|
||||
def read_manifest(sweep_dir):
|
||||
rows = []
|
||||
with open(os.path.join(sweep_dir, "manifest.tsv")) as f:
|
||||
for row in csv.DictReader(f, delimiter="\t"):
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def collect(sweep_dir):
|
||||
records = []
|
||||
for entry in read_manifest(sweep_dir):
|
||||
run_dir = os.path.join(sweep_dir, entry["leaf"])
|
||||
if not os.path.isdir(run_dir):
|
||||
print(f"warning: missing run dir {run_dir}", file=sys.stderr)
|
||||
continue
|
||||
run = Run(run_dir)
|
||||
if not run.frames:
|
||||
print(f"warning: no frames for {entry['leaf']}, skipping", file=sys.stderr)
|
||||
continue
|
||||
s = run.summary()
|
||||
try:
|
||||
loss = float(entry["loss"])
|
||||
except ValueError:
|
||||
loss = 0.0
|
||||
expected = s["frames_expected"] or 1
|
||||
rec = {
|
||||
"leaf": entry["leaf"],
|
||||
"mode": entry["mode"],
|
||||
"loss": loss,
|
||||
"loss_label": entry["loss"],
|
||||
"fec_rate": entry["fec_rate"],
|
||||
"fec_mask": entry["fec_mask"],
|
||||
"config": "baseline"
|
||||
if entry["mode"] == "baseline"
|
||||
else f"FEC r{entry['fec_rate']}/{entry['fec_mask']}",
|
||||
"frame_loss_pct": 100.0 * s["frames_lost"] / expected,
|
||||
"recovery_rate_pct": (100.0 * s["fec_recovered"] / s["fec_received"]) if s["fec_received"] else 0.0,
|
||||
"debug_drop": run.meta.get("DEBUG_DROP", "0"),
|
||||
**s,
|
||||
}
|
||||
records.append(rec)
|
||||
return records
|
||||
|
||||
|
||||
def x_unit(records):
|
||||
if records and records[0]["debug_drop"] not in ("0", ""):
|
||||
return "injected packet drop (%)"
|
||||
return "burst loss fraction"
|
||||
|
||||
|
||||
def write_csv(records, path):
|
||||
cols = [
|
||||
"config", "mode", "loss_label", "fec_rate", "fec_mask",
|
||||
"frames_received", "frames_expected", "frames_lost", "frame_loss_pct",
|
||||
"latency_p50_ms", "latency_p95_ms", "latency_p99_ms",
|
||||
"fec_received", "fec_recovery_attempts", "fec_recovered",
|
||||
"fec_recovery_failed", "recovery_rate_pct", "nack_total",
|
||||
]
|
||||
with open(path, "w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(cols)
|
||||
for r in sorted(records, key=lambda r: (r["loss"], r["config"])):
|
||||
w.writerow([_fmt(r.get(c)) for c in cols])
|
||||
|
||||
|
||||
def _fmt(v):
|
||||
if isinstance(v, float):
|
||||
return f"{v:.2f}"
|
||||
return v
|
||||
|
||||
|
||||
def series_by_config(records):
|
||||
"""config label -> sorted [(loss, record)]"""
|
||||
by_cfg = {}
|
||||
for r in records:
|
||||
by_cfg.setdefault(r["config"], []).append(r)
|
||||
for cfg in by_cfg:
|
||||
by_cfg[cfg].sort(key=lambda r: r["loss"])
|
||||
return by_cfg
|
||||
|
||||
|
||||
def plot(records, out_path):
|
||||
by_cfg = series_by_config(records)
|
||||
unit = x_unit(records)
|
||||
# baseline first (gray), FEC configs in a color cycle
|
||||
order = sorted(by_cfg, key=lambda c: (c != "baseline", c))
|
||||
cmap = plt.get_cmap("viridis")
|
||||
fec_cfgs = [c for c in order if c != "baseline"]
|
||||
colors = {"baseline": "#888888"}
|
||||
for i, c in enumerate(fec_cfgs):
|
||||
colors[c] = cmap(0.15 + 0.7 * (i / max(1, len(fec_cfgs) - 1)))
|
||||
|
||||
fig, axes = plt.subplots(2, 2, figsize=(15, 11))
|
||||
ax_loss, ax_lat, ax_rec, ax_recn = axes.flat
|
||||
|
||||
def line(ax, key, cfgs):
|
||||
for cfg in cfgs:
|
||||
pts = by_cfg[cfg]
|
||||
xs = [r["loss"] for r in pts]
|
||||
ys = [r[key] for r in pts]
|
||||
ax.plot(xs, ys, marker="o", label=cfg, color=colors[cfg],
|
||||
lw=2 if cfg != "baseline" else 1.5,
|
||||
ls="-" if cfg != "baseline" else "--")
|
||||
|
||||
line(ax_loss, "frame_loss_pct", order)
|
||||
ax_loss.set_title("Frame loss vs loss level")
|
||||
ax_loss.set_ylabel("frames lost (%)")
|
||||
ax_loss.set_xlabel(unit)
|
||||
ax_loss.legend(fontsize=8)
|
||||
ax_loss.grid(alpha=0.3)
|
||||
|
||||
line(ax_lat, "latency_p99_ms", order)
|
||||
ax_lat.set_title("Tail latency (p99) vs loss level")
|
||||
ax_lat.set_ylabel("capture→receive p99 (ms)")
|
||||
ax_lat.set_xlabel(unit)
|
||||
ax_lat.legend(fontsize=8)
|
||||
ax_lat.grid(alpha=0.3)
|
||||
|
||||
line(ax_rec, "recovery_rate_pct", fec_cfgs)
|
||||
ax_rec.set_title("FEC recovery rate (recovered / FEC packets received)")
|
||||
ax_rec.set_ylabel("recovery rate (%)")
|
||||
ax_rec.set_xlabel(unit)
|
||||
ax_rec.legend(fontsize=8)
|
||||
ax_rec.grid(alpha=0.3)
|
||||
|
||||
line(ax_recn, "fec_recovered", fec_cfgs)
|
||||
ax_recn.set_title("Packets recovered by FEC")
|
||||
ax_recn.set_ylabel("recovered packets")
|
||||
ax_recn.set_xlabel(unit)
|
||||
ax_recn.legend(fontsize=8)
|
||||
ax_recn.grid(alpha=0.3)
|
||||
|
||||
fig.suptitle("FlexFEC parameter sweep — baseline (no FEC) vs FEC configs", fontsize=13)
|
||||
fig.tight_layout(rect=(0, 0, 1, 0.97))
|
||||
fig.savefig(out_path, dpi=130)
|
||||
|
||||
|
||||
def print_table(records):
|
||||
cols = [
|
||||
("config", "config", "{}"),
|
||||
("loss_label", "loss", "{}"),
|
||||
("frames_lost", "lost", "{:.0f}"),
|
||||
("frame_loss_pct", "lost%", "{:.1f}"),
|
||||
("latency_p95_ms", "p95ms", "{:.1f}"),
|
||||
("latency_p99_ms", "p99ms", "{:.1f}"),
|
||||
("fec_received", "fecRx", "{:.0f}"),
|
||||
("fec_recovered", "recov", "{:.0f}"),
|
||||
("recovery_rate_pct", "recov%", "{:.1f}"),
|
||||
("nack_total", "nacks", "{:.0f}"),
|
||||
]
|
||||
widths = [max(len(h), 8) for _, h, _ in cols]
|
||||
print("| " + " | ".join(h.ljust(w) for (_, h, _), w in zip(cols, widths)) + " |")
|
||||
print("|" + "|".join("-" * (w + 2) for w in widths) + "|")
|
||||
for r in sorted(records, key=lambda r: (r["loss"], r["config"] != "baseline", r["config"])):
|
||||
cells = []
|
||||
for (key, _, fmt), w in zip(cols, widths):
|
||||
v = r.get(key)
|
||||
cells.append((fmt.format(v) if v is not None else "-").ljust(w))
|
||||
print("| " + " | ".join(cells) + " |")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--sweep", required=True, help="sweep output directory (contains manifest.tsv)")
|
||||
args = parser.parse_args()
|
||||
|
||||
records = collect(args.sweep)
|
||||
if not records:
|
||||
print("no usable cells found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
csv_path = os.path.join(args.sweep, "sweep_summary.csv")
|
||||
png_path = os.path.join(args.sweep, "sweep_report.png")
|
||||
write_csv(records, csv_path)
|
||||
plot(records, png_path)
|
||||
|
||||
print(f"report: {png_path}")
|
||||
print(f"summary: {csv_path}")
|
||||
print()
|
||||
print_table(records)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+373
@@ -0,0 +1,373 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render time-series plots and a summary for FlexFEC test harness runs.
|
||||
|
||||
Inputs are run directories produced by run_fec_test.sh, each containing:
|
||||
frames.csv - per received frame: recv_wall_us,frame_id,user_timestamp_us,width,height
|
||||
prom.tsv - 1s prometheus samples: wall_us<TAB>metric{labels}<TAB>value
|
||||
events.csv - shaper events: wall_us,event (burst_on / burst_off / ...)
|
||||
meta.env - run parameters incl. T0_US/T1_US measurement window
|
||||
|
||||
Usage:
|
||||
plot_fec.py --run <dir> [--run <dir2>] --out <dir>
|
||||
|
||||
With two runs the first is treated as the baseline and the second as the FEC
|
||||
run, both are overlaid and a comparison table is printed.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt # noqa: E402
|
||||
|
||||
FLEXFEC_METRIC = "livekit_flexfec_packet_total"
|
||||
METRIC_RE = re.compile(r"^(?P<name>[a-zA-Z0-9_]+)(?:\{(?P<labels>.*)\})?$")
|
||||
|
||||
|
||||
class Run:
|
||||
def __init__(self, path):
|
||||
self.path = path
|
||||
self.name = os.path.basename(os.path.normpath(path))
|
||||
self.meta = self._read_meta()
|
||||
self.t0 = int(self.meta.get("T0_US", 0))
|
||||
self.t1 = int(self.meta.get("T1_US", 1 << 62))
|
||||
self.frames = self._read_frames()
|
||||
self.prom = self._read_prom()
|
||||
self.bursts = self._read_bursts()
|
||||
|
||||
def _read_meta(self):
|
||||
meta = {}
|
||||
path = os.path.join(self.path, "meta.env")
|
||||
if os.path.exists(path):
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
if "=" in line:
|
||||
k, v = line.strip().split("=", 1)
|
||||
meta[k] = v
|
||||
return meta
|
||||
|
||||
def _in_window(self, ts):
|
||||
return self.t0 <= ts <= self.t1
|
||||
|
||||
def rel(self, ts):
|
||||
return (ts - self.t0) / 1e6
|
||||
|
||||
def _read_frames(self):
|
||||
frames = []
|
||||
path = os.path.join(self.path, "frames.csv")
|
||||
if not os.path.exists(path):
|
||||
return frames
|
||||
with open(path) as f:
|
||||
for row in csv.DictReader(f):
|
||||
try:
|
||||
recv = int(row["recv_wall_us"])
|
||||
except (KeyError, ValueError):
|
||||
continue
|
||||
if not self._in_window(recv):
|
||||
continue
|
||||
frames.append(
|
||||
{
|
||||
"recv": recv,
|
||||
"frame_id": int(row["frame_id"]) if row.get("frame_id") else None,
|
||||
"user_ts": int(row["user_timestamp_us"]) if row.get("user_timestamp_us") else None,
|
||||
}
|
||||
)
|
||||
frames.sort(key=lambda fr: fr["recv"])
|
||||
return frames
|
||||
|
||||
def _read_prom(self):
|
||||
"""metric family -> label key -> [(wall_us, value)]"""
|
||||
series = {}
|
||||
path = os.path.join(self.path, "prom.tsv")
|
||||
if not os.path.exists(path):
|
||||
return series
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.rstrip("\n").split("\t")
|
||||
if len(parts) != 3:
|
||||
continue
|
||||
ts, metric, value = parts
|
||||
m = METRIC_RE.match(metric)
|
||||
if not m:
|
||||
continue
|
||||
try:
|
||||
ts, value = int(ts), float(value)
|
||||
except ValueError:
|
||||
continue
|
||||
family = m.group("name")
|
||||
labels = m.group("labels") or ""
|
||||
series.setdefault(family, {}).setdefault(labels, []).append((ts, value))
|
||||
return series
|
||||
|
||||
def _read_bursts(self):
|
||||
"""[(start_rel_s, end_rel_s)] of shaper burst windows"""
|
||||
bursts, start = [], None
|
||||
path = os.path.join(self.path, "events.csv")
|
||||
if not os.path.exists(path):
|
||||
return bursts
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split(",", 1)
|
||||
if len(parts) != 2:
|
||||
continue
|
||||
ts, event = int(parts[0]), parts[1]
|
||||
if event == "burst_on":
|
||||
start = self.rel(ts)
|
||||
elif event == "burst_off" and start is not None:
|
||||
bursts.append((start, self.rel(ts)))
|
||||
start = None
|
||||
if start is not None:
|
||||
bursts.append((start, self.rel(self.t1 if self.t1 < (1 << 62) else start)))
|
||||
return bursts
|
||||
|
||||
# ---------- derived series ----------
|
||||
|
||||
def latency_series(self):
|
||||
xs, ys = [], []
|
||||
for fr in self.frames:
|
||||
if fr["user_ts"]:
|
||||
xs.append(self.rel(fr["recv"]))
|
||||
ys.append((fr["recv"] - fr["user_ts"]) / 1e3)
|
||||
return xs, ys
|
||||
|
||||
def frame_gaps(self):
|
||||
"""[(rel_s, missing_count)] where frame ids skipped between consecutive frames"""
|
||||
gaps, prev = [], None
|
||||
for fr in self.frames:
|
||||
fid = fr["frame_id"]
|
||||
if fid is None:
|
||||
continue
|
||||
if prev is not None and fid > prev + 1:
|
||||
gaps.append((self.rel(fr["recv"]), fid - prev - 1))
|
||||
prev = fid
|
||||
return gaps
|
||||
|
||||
def fps_series(self):
|
||||
buckets = {}
|
||||
for fr in self.frames:
|
||||
buckets[int(self.rel(fr["recv"]))] = buckets.get(int(self.rel(fr["recv"])), 0) + 1
|
||||
xs = sorted(buckets)
|
||||
return xs, [buckets[x] for x in xs]
|
||||
|
||||
def counter_rate(self, family, label_filter=None):
|
||||
"""summed per-second rate across label sets of a counter family"""
|
||||
fam = self.prom.get(family, {})
|
||||
per_ts = {}
|
||||
for labels, samples in fam.items():
|
||||
if label_filter and label_filter not in labels:
|
||||
continue
|
||||
samples = [s for s in samples if self._in_window(s[0])]
|
||||
for (t_a, v_a), (t_b, v_b) in zip(samples, samples[1:]):
|
||||
dt = (t_b - t_a) / 1e6
|
||||
if dt <= 0:
|
||||
continue
|
||||
key = int(self.rel(t_b))
|
||||
per_ts[key] = per_ts.get(key, 0.0) + max(0.0, v_b - v_a) / dt
|
||||
xs = sorted(per_ts)
|
||||
return xs, [per_ts[x] for x in xs]
|
||||
|
||||
def counter_total(self, family, label_filter=None):
|
||||
"""delta of a summed counter family over the measurement window"""
|
||||
total = 0.0
|
||||
for labels, samples in self.prom.get(family, {}).items():
|
||||
if label_filter and label_filter not in labels:
|
||||
continue
|
||||
samples = [s for s in samples if self._in_window(s[0])]
|
||||
if len(samples) >= 2:
|
||||
total += samples[-1][1] - samples[0][1]
|
||||
return total
|
||||
|
||||
def flexfec_state_rate(self, state):
|
||||
return self.counter_rate(FLEXFEC_METRIC, f'state="{state}"')
|
||||
|
||||
def flexfec_state_total(self, state):
|
||||
return self.counter_total(FLEXFEC_METRIC, f'state="{state}"')
|
||||
|
||||
# ---------- summary ----------
|
||||
|
||||
def summary(self):
|
||||
_, lat = self.latency_series()
|
||||
gaps = self.frame_gaps()
|
||||
frame_ids = [fr["frame_id"] for fr in self.frames if fr["frame_id"] is not None]
|
||||
expected = (max(frame_ids) - min(frame_ids) + 1) if frame_ids else 0
|
||||
lost = sum(n for _, n in gaps)
|
||||
|
||||
def pct(p):
|
||||
if not lat:
|
||||
return float("nan")
|
||||
data = sorted(lat)
|
||||
return data[min(len(data) - 1, int(len(data) * p))]
|
||||
|
||||
return {
|
||||
"frames_received": len(self.frames),
|
||||
"frames_expected": expected,
|
||||
"frames_lost": lost,
|
||||
"gap_events": len(gaps),
|
||||
"latency_mean_ms": statistics.fmean(lat) if lat else float("nan"),
|
||||
"latency_p50_ms": pct(0.50),
|
||||
"latency_p95_ms": pct(0.95),
|
||||
"latency_p99_ms": pct(0.99),
|
||||
"fec_received": self.flexfec_state_total("received"),
|
||||
"fec_recovery_attempts": self.flexfec_state_total("recovery_attempt"),
|
||||
"fec_recovered": self.flexfec_state_total("recovered"),
|
||||
"fec_recovery_failed": self.flexfec_state_total("recovery_failed"),
|
||||
"fec_unused": self.flexfec_state_total("unused"),
|
||||
"fec_invalid": self.flexfec_state_total("invalid"),
|
||||
"nack_total": self.counter_total("livekit_nack_total"),
|
||||
"packet_loss_total": self.counter_total("livekit_packet_loss_total"),
|
||||
}
|
||||
|
||||
|
||||
def shade_bursts(ax, bursts):
|
||||
for start, end in bursts:
|
||||
ax.axvspan(start, end, color="red", alpha=0.08, lw=0)
|
||||
|
||||
|
||||
def plot(runs, out_dir):
|
||||
colors = {"baseline": "#888888", "fec": "#1f77b4"}
|
||||
fig, axes = plt.subplots(4, 1, figsize=(14, 16), sharex=True)
|
||||
ax_lat, ax_fec, ax_loss, ax_fps = axes
|
||||
bursts = runs[-1].bursts
|
||||
|
||||
for run in runs:
|
||||
color = colors.get(run.name, None)
|
||||
xs, ys = run.latency_series()
|
||||
ax_lat.plot(xs, ys, ".", markersize=2.5, label=f"{run.name} latency", color=color, alpha=0.7)
|
||||
for x, n in run.frame_gaps():
|
||||
ax_lat.axvline(x, color=color or "red", alpha=0.5, lw=min(0.5 + n * 0.3, 3))
|
||||
shade_bursts(ax_lat, bursts)
|
||||
ax_lat.set_ylabel("capture→receive latency (ms)")
|
||||
ax_lat.set_title("Frame latency (vertical lines: frame-id gaps = lost frames, red shading: loss bursts)")
|
||||
ax_lat.legend(loc="upper right", fontsize=8)
|
||||
ax_lat.grid(alpha=0.3)
|
||||
|
||||
fec_runs = [r for r in runs if r.flexfec_state_total("received") > 0] or runs[-1:]
|
||||
for run in fec_runs:
|
||||
for state, style in [
|
||||
("received", dict(color="#1f77b4", lw=1)),
|
||||
("recovered", dict(color="#2ca02c", lw=1.8)),
|
||||
("recovery_failed", dict(color="#d62728", lw=1.2)),
|
||||
("unused", dict(color="#9467bd", lw=0.8, alpha=0.6)),
|
||||
]:
|
||||
xs, ys = run.flexfec_state_rate(state)
|
||||
ax_fec.plot(xs, ys, label=f"{run.name} {state}/s", **style)
|
||||
shade_bursts(ax_fec, bursts)
|
||||
ax_fec.set_ylabel("FEC packets/s")
|
||||
ax_fec.set_title("FlexFEC at the SFU")
|
||||
ax_fec.legend(loc="upper right", fontsize=8)
|
||||
ax_fec.grid(alpha=0.3)
|
||||
|
||||
for run in runs:
|
||||
color = colors.get(run.name, None)
|
||||
xs, ys = run.counter_rate("livekit_nack_total")
|
||||
ax_loss.plot(xs, ys, label=f"{run.name} nack/s", color=color, lw=1.2)
|
||||
xs, ys = run.counter_rate("livekit_packet_loss_total")
|
||||
ax_loss.plot(xs, ys, label=f"{run.name} packet_loss/s", color=color, lw=1.2, ls="--", alpha=0.7)
|
||||
shade_bursts(ax_loss, bursts)
|
||||
ax_loss.set_ylabel("packets/s")
|
||||
ax_loss.set_title("NACK and reported packet loss")
|
||||
ax_loss.legend(loc="upper right", fontsize=8)
|
||||
ax_loss.grid(alpha=0.3)
|
||||
|
||||
for run in runs:
|
||||
color = colors.get(run.name, None)
|
||||
xs, ys = run.fps_series()
|
||||
ax_fps.plot(xs, ys, label=f"{run.name} fps", color=color, lw=1.2)
|
||||
shade_bursts(ax_fps, bursts)
|
||||
ax_fps.set_ylabel("frames/s")
|
||||
ax_fps.set_xlabel("seconds since measurement start")
|
||||
ax_fps.set_title("Delivered frame rate at subscriber")
|
||||
ax_fps.legend(loc="lower right", fontsize=8)
|
||||
ax_fps.grid(alpha=0.3)
|
||||
|
||||
meta = runs[-1].meta
|
||||
if meta.get("DEBUG_DROP", "0") not in ("0", ""):
|
||||
profile = "uniform {}% loss injected at SFU receive".format(meta["DEBUG_DROP"])
|
||||
elif meta.get("SHAPING") == "0":
|
||||
profile = "no loss"
|
||||
else:
|
||||
profile = "base loss {} / burst {} for {}s every {}s".format(
|
||||
meta.get("BASE_LOSS", "?"),
|
||||
meta.get("BURST_LOSS", "?"),
|
||||
meta.get("BURST_LEN", "?"),
|
||||
meta.get("BURST_EVERY", "?"),
|
||||
)
|
||||
fig.suptitle(
|
||||
"FlexFEC publisher→SFU recovery — {}, codec {}".format(profile, meta.get("CODEC", "?")),
|
||||
fontsize=12,
|
||||
)
|
||||
fig.tight_layout(rect=(0, 0, 1, 0.985))
|
||||
out_path = os.path.join(out_dir, "fec_report.png")
|
||||
fig.savefig(out_path, dpi=130)
|
||||
return out_path
|
||||
|
||||
|
||||
def print_summary(runs):
|
||||
summaries = [(run.name, run.summary()) for run in runs]
|
||||
keys = [
|
||||
("frames_received", "{:.0f}"),
|
||||
("frames_expected", "{:.0f}"),
|
||||
("frames_lost", "{:.0f}"),
|
||||
("gap_events", "{:.0f}"),
|
||||
("latency_mean_ms", "{:.1f}"),
|
||||
("latency_p50_ms", "{:.1f}"),
|
||||
("latency_p95_ms", "{:.1f}"),
|
||||
("latency_p99_ms", "{:.1f}"),
|
||||
("fec_received", "{:.0f}"),
|
||||
("fec_recovery_attempts", "{:.0f}"),
|
||||
("fec_recovered", "{:.0f}"),
|
||||
("fec_recovery_failed", "{:.0f}"),
|
||||
("fec_unused", "{:.0f}"),
|
||||
("fec_invalid", "{:.0f}"),
|
||||
("nack_total", "{:.0f}"),
|
||||
("packet_loss_total", "{:.0f}"),
|
||||
]
|
||||
|
||||
name_w = 24
|
||||
header = "metric".ljust(name_w) + "".join(name.rjust(16) for name, _ in summaries)
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
for key, fmt in keys:
|
||||
row = key.ljust(name_w)
|
||||
for _, summary in summaries:
|
||||
row += fmt.format(summary[key]).rjust(16)
|
||||
print(row)
|
||||
|
||||
if len(summaries) == 2:
|
||||
base, fec = summaries[0][1], summaries[1][1]
|
||||
print()
|
||||
if fec["frames_lost"] < base["frames_lost"]:
|
||||
print(
|
||||
"frames lost reduced {} -> {} with FlexFEC".format(
|
||||
int(base["frames_lost"]), int(fec["frames_lost"])
|
||||
)
|
||||
)
|
||||
if fec["fec_recovered"] > 0:
|
||||
print("SFU recovered {} packets via FlexFEC".format(int(fec["fec_recovered"])))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run", action="append", required=True, help="run directory (repeatable)")
|
||||
parser.add_argument("--out", required=True, help="output directory for the report")
|
||||
args = parser.parse_args()
|
||||
|
||||
runs = [Run(path) for path in args.run]
|
||||
for run in runs:
|
||||
if not run.frames:
|
||||
print(f"warning: no frames recorded for {run.path}", file=sys.stderr)
|
||||
|
||||
out_path = plot(runs, args.out)
|
||||
print(f"report: {out_path}")
|
||||
print()
|
||||
print_summary(runs)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Polls the SFU's prometheus endpoint once per second and appends the FEC and
|
||||
# packet counters relevant to the FlexFEC test harness as tab-separated rows:
|
||||
# wall_us<TAB>metric{labels}<TAB>value
|
||||
#
|
||||
# Usage: ./prom_poll.sh <prometheus_port> <output_file>
|
||||
|
||||
set -u
|
||||
|
||||
PORT="${1:?prometheus port required}"
|
||||
OUT="${2:?output file required}"
|
||||
|
||||
now_us() { python3 -c 'import time; print(int(time.time() * 1e6))'; }
|
||||
|
||||
while true; do
|
||||
TS=$(now_us)
|
||||
curl -s --max-time 2 "http://127.0.0.1:${PORT}/metrics" | \
|
||||
awk -v ts="$TS" '/^livekit_(flexfec_packet|nack|packet|packet_loss|packet_out_of_order)_total/ {
|
||||
value = $NF
|
||||
metric = $0
|
||||
sub(/ [^ ]*$/, "", metric)
|
||||
print ts "\t" metric "\t" value
|
||||
}' >> "$OUT"
|
||||
sleep 1
|
||||
done
|
||||
Executable
+344
@@ -0,0 +1,344 @@
|
||||
#!/usr/bin/env bash
|
||||
# End-to-end FlexFEC test harness: publisher (rust-sdks local_video) -> SFU,
|
||||
# with traffic shaping on the publisher->SFU leg simulating a cellular uplink
|
||||
# (continuous base loss + periodic loss bursts), collecting frame metadata at
|
||||
# the subscriber and FEC/NACK/loss counters from the SFU, then plotting time
|
||||
# series and printing a summary.
|
||||
#
|
||||
# Usage:
|
||||
# ./run_fec_test.sh --mode {fec|baseline|ab} [options]
|
||||
#
|
||||
# Modes:
|
||||
# fec single run with --flex-fec on the publisher
|
||||
# baseline single run without FlexFEC (NACK/RTX recovery only)
|
||||
# ab baseline run followed by a fec run with the same loss profile,
|
||||
# producing a comparison report
|
||||
#
|
||||
# Options (defaults in brackets):
|
||||
# --duration S measurement duration per run after media starts [120]
|
||||
# --out DIR output directory [scripts/fec/out/<timestamp>]
|
||||
# --base-loss F continuous loss fraction [0.02]
|
||||
# --burst-loss F loss fraction inside bursts [0.1]
|
||||
# --burst-every S seconds between burst starts [15]
|
||||
# --burst-len S burst duration in seconds [1]
|
||||
# --fec-rate N publisher FEC protection rate percent, 0 = adaptive [30]
|
||||
# --fec-mask-type T random|bursty [bursty]
|
||||
# --codec C video codec [h264]
|
||||
# --width/--height/--fps test pattern format [1280/720/30]
|
||||
# --no-shaping skip traffic shaping (sanity run)
|
||||
# --debug-drop PCT drop PCT% of received packets inside the SFU instead of
|
||||
# OS traffic shaping (uniform loss, no sudo required;
|
||||
# implies --no-shaping)
|
||||
# --server-bin PATH use this prebuilt livekit-server instead of building
|
||||
# --skip-build skip go/cargo builds (requires --server-bin and
|
||||
# pre-built example binaries); used by sweep_fec.sh
|
||||
#
|
||||
# Requires: go, cargo, python3 with matplotlib, curl, and sudo (for shaping).
|
||||
# The rust-sdks checkout is located via RUST_SDKS_DIR [../rust-sdks].
|
||||
|
||||
set -u
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
RUST_SDKS_DIR="${RUST_SDKS_DIR:-$(cd "$REPO_ROOT/.." && pwd)/rust-sdks}"
|
||||
|
||||
MODE=""
|
||||
DURATION=120
|
||||
OUT_DIR=""
|
||||
BASE_LOSS=0.02
|
||||
BURST_LOSS=0.1
|
||||
BURST_EVERY=15
|
||||
BURST_LEN=1
|
||||
FEC_RATE=30
|
||||
FEC_MASK_TYPE="bursty"
|
||||
CODEC="h264"
|
||||
WIDTH=1280
|
||||
HEIGHT=720
|
||||
FPS=30
|
||||
SHAPING=1
|
||||
DEBUG_DROP=0
|
||||
SERVER_BIN=""
|
||||
SKIP_BUILD=0
|
||||
|
||||
SIGNAL_PORT=7880
|
||||
MEDIA_PORT=7882
|
||||
PROM_PORT=6789
|
||||
API_KEY="devkey"
|
||||
API_SECRET="fec-test-secret-fec-test-secret-00"
|
||||
ROOM_NAME="fec-test"
|
||||
|
||||
log() { echo "[fec-test] $*"; }
|
||||
die() { echo "[fec-test] ERROR: $*" >&2; exit 1; }
|
||||
now_us() { python3 -c 'import time; print(int(time.time() * 1e6))'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--mode) MODE="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--out) OUT_DIR="$2"; shift 2 ;;
|
||||
--base-loss) BASE_LOSS="$2"; shift 2 ;;
|
||||
--burst-loss) BURST_LOSS="$2"; shift 2 ;;
|
||||
--burst-every) BURST_EVERY="$2"; shift 2 ;;
|
||||
--burst-len) BURST_LEN="$2"; shift 2 ;;
|
||||
--fec-rate) FEC_RATE="$2"; shift 2 ;;
|
||||
--fec-mask-type) FEC_MASK_TYPE="$2"; shift 2 ;;
|
||||
--codec) CODEC="$2"; shift 2 ;;
|
||||
--width) WIDTH="$2"; shift 2 ;;
|
||||
--height) HEIGHT="$2"; shift 2 ;;
|
||||
--fps) FPS="$2"; shift 2 ;;
|
||||
--no-shaping) SHAPING=0; shift ;;
|
||||
--debug-drop) DEBUG_DROP="$2"; SHAPING=0; shift 2 ;;
|
||||
--server-bin) SERVER_BIN="$2"; shift 2 ;;
|
||||
--skip-build) SKIP_BUILD=1; shift ;;
|
||||
-h|--help) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$MODE" in
|
||||
fec|baseline|ab) ;;
|
||||
*) die "--mode must be fec, baseline or ab" ;;
|
||||
esac
|
||||
|
||||
case "$(uname -s)" in
|
||||
Darwin) SHAPER="$SCRIPT_DIR/shape_macos.sh"; LOOPBACK_IF="lo0" ;;
|
||||
Linux) SHAPER="$SCRIPT_DIR/shape_linux.sh"; LOOPBACK_IF="lo" ;;
|
||||
*) die "unsupported platform $(uname -s)" ;;
|
||||
esac
|
||||
|
||||
# ---------- preflight ----------
|
||||
|
||||
command -v go >/dev/null || die "go not found"
|
||||
command -v cargo >/dev/null || die "cargo not found"
|
||||
command -v curl >/dev/null || die "curl not found"
|
||||
python3 -c 'import matplotlib' 2>/dev/null || die "python3 with matplotlib required (pip3 install matplotlib)"
|
||||
[ -d "$RUST_SDKS_DIR/examples/local_video" ] || die "rust-sdks not found at $RUST_SDKS_DIR (set RUST_SDKS_DIR)"
|
||||
|
||||
if [ "$SHAPING" = "1" ]; then
|
||||
log "shaping requires sudo, validating credentials..."
|
||||
sudo -v || die "sudo required for traffic shaping (or pass --no-shaping)"
|
||||
# keep the sudo timestamp alive for long runs
|
||||
( while true; do sudo -n true 2>/dev/null; sleep 60; done ) &
|
||||
SUDO_KEEPALIVE_PID=$!
|
||||
fi
|
||||
|
||||
if [ -z "$OUT_DIR" ]; then
|
||||
OUT_DIR="$SCRIPT_DIR/out/$(date +%Y%m%d_%H%M%S)"
|
||||
fi
|
||||
mkdir -p "$OUT_DIR"
|
||||
log "output directory: $OUT_DIR"
|
||||
|
||||
# ---------- builds ----------
|
||||
|
||||
if [ "$SKIP_BUILD" = "1" ]; then
|
||||
[ -n "$SERVER_BIN" ] || die "--skip-build requires --server-bin"
|
||||
[ -x "$SERVER_BIN" ] || die "server binary not found at $SERVER_BIN"
|
||||
log "skipping builds, using prebuilt $SERVER_BIN"
|
||||
else
|
||||
SERVER_BIN="${SERVER_BIN:-$OUT_DIR/livekit-server}"
|
||||
log "building livekit-server..."
|
||||
(cd "$REPO_ROOT" && go build -o "$SERVER_BIN" ./cmd/server) || die "server build failed"
|
||||
|
||||
log "building local_video examples (release)..."
|
||||
(cd "$RUST_SDKS_DIR" && cargo build --release -p local_video -F desktop --bin publisher --bin subscriber) \
|
||||
|| die "example build failed"
|
||||
fi
|
||||
PUBLISHER_BIN="$RUST_SDKS_DIR/target/release/publisher"
|
||||
SUBSCRIBER_BIN="$RUST_SDKS_DIR/target/release/subscriber"
|
||||
[ -x "$PUBLISHER_BIN" ] || die "publisher binary not found, run once without --skip-build first"
|
||||
[ -x "$SUBSCRIBER_BIN" ] || die "subscriber binary not found, run once without --skip-build first"
|
||||
|
||||
# ---------- process management ----------
|
||||
|
||||
SERVER_PID=""
|
||||
SUBSCRIBER_PID=""
|
||||
PUBLISHER_PID=""
|
||||
PROM_POLL_PID=""
|
||||
SHAPER_PID=""
|
||||
|
||||
stop_pid() {
|
||||
local pid="$1" sig="${2:-TERM}"
|
||||
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
|
||||
kill -"$sig" "$pid" 2>/dev/null
|
||||
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
||||
kill -0 "$pid" 2>/dev/null || return 0
|
||||
sleep 0.5
|
||||
done
|
||||
kill -KILL "$pid" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
stop_shaper() {
|
||||
if [ -n "$SHAPER_PID" ] && kill -0 "$SHAPER_PID" 2>/dev/null; then
|
||||
sudo kill -TERM "$SHAPER_PID" 2>/dev/null
|
||||
sleep 2
|
||||
fi
|
||||
SHAPER_PID=""
|
||||
sudo "$SHAPER" stop >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
trap - EXIT INT TERM
|
||||
log "cleaning up processes"
|
||||
[ "$SHAPING" = "1" ] && stop_shaper
|
||||
stop_pid "$PROM_POLL_PID"
|
||||
stop_pid "$PUBLISHER_PID" INT
|
||||
stop_pid "$SUBSCRIBER_PID" INT
|
||||
stop_pid "$SERVER_PID"
|
||||
[ -n "${SUDO_KEEPALIVE_PID:-}" ] && kill "$SUDO_KEEPALIVE_PID" 2>/dev/null
|
||||
exit "${1:-1}"
|
||||
}
|
||||
trap 'cleanup 1' INT TERM
|
||||
trap 'cleanup $?' EXIT
|
||||
|
||||
# ---------- single run ----------
|
||||
|
||||
run_one() {
|
||||
local mode="$1"
|
||||
local run_dir="$OUT_DIR/$mode"
|
||||
mkdir -p "$run_dir"
|
||||
log "=== $mode run: ${DURATION}s ==="
|
||||
|
||||
# media is pinned to the loopback interface: it keeps every packet on the
|
||||
# shaped path and, on macOS, avoids the application firewall silently
|
||||
# dropping inbound UDP for unsigned freshly-built binaries
|
||||
cat > "$run_dir/server.yaml" <<EOF
|
||||
port: $SIGNAL_PORT
|
||||
bind_addresses:
|
||||
- 127.0.0.1
|
||||
rtc:
|
||||
udp_port: $MEDIA_PORT
|
||||
use_external_ip: false
|
||||
enable_loopback_candidate: true
|
||||
interfaces:
|
||||
includes:
|
||||
- $LOOPBACK_IF
|
||||
enable_flexfec: true
|
||||
prometheus:
|
||||
port: $PROM_PORT
|
||||
keys:
|
||||
$API_KEY: $API_SECRET
|
||||
room:
|
||||
auto_create: true
|
||||
logging:
|
||||
level: info
|
||||
EOF
|
||||
|
||||
log "starting livekit-server"
|
||||
LIVEKIT_DEBUG_RX_DROP_PCT="$DEBUG_DROP" \
|
||||
"$SERVER_BIN" --config "$run_dir/server.yaml" > "$run_dir/server.log" 2>&1 &
|
||||
SERVER_PID=$!
|
||||
|
||||
for i in $(seq 1 40); do
|
||||
curl -s -o /dev/null --max-time 1 "http://127.0.0.1:$SIGNAL_PORT" && break
|
||||
kill -0 "$SERVER_PID" 2>/dev/null || die "server exited early, see $run_dir/server.log"
|
||||
[ "$i" = "40" ] && die "server did not become reachable"
|
||||
sleep 0.5
|
||||
done
|
||||
log "server is up (pid $SERVER_PID)"
|
||||
|
||||
local conn_args="--url ws://127.0.0.1:$SIGNAL_PORT --api-key $API_KEY --api-secret $API_SECRET --room-name $ROOM_NAME"
|
||||
|
||||
log "starting subscriber (headless)"
|
||||
RUST_LOG=info "$SUBSCRIBER_BIN" $conn_args \
|
||||
--identity fec-sub --headless --log-frames "$run_dir/frames.csv" \
|
||||
> "$run_dir/subscriber.log" 2>&1 &
|
||||
SUBSCRIBER_PID=$!
|
||||
sleep 2
|
||||
|
||||
local fec_args=""
|
||||
if [ "$mode" = "fec" ]; then
|
||||
fec_args="--flex-fec --fec-mask-type $FEC_MASK_TYPE"
|
||||
if [ "$FEC_RATE" != "0" ]; then
|
||||
fec_args="$fec_args --fec-protection-rate $FEC_RATE"
|
||||
fi
|
||||
fi
|
||||
|
||||
log "starting publisher (test pattern ${WIDTH}x${HEIGHT}@${FPS} $CODEC${fec_args:+,$fec_args})"
|
||||
RUST_LOG=info "$PUBLISHER_BIN" $conn_args \
|
||||
--identity fec-pub --test-pattern \
|
||||
--width "$WIDTH" --height "$HEIGHT" --fps "$FPS" --codec "$CODEC" \
|
||||
--attach-timestamp --attach-frame-id $fec_args \
|
||||
> "$run_dir/publisher.log" 2>&1 &
|
||||
PUBLISHER_PID=$!
|
||||
|
||||
log "waiting for media to flow..."
|
||||
for i in $(seq 1 120); do
|
||||
if [ -f "$run_dir/frames.csv" ] && [ "$(wc -l < "$run_dir/frames.csv")" -gt 30 ]; then
|
||||
break
|
||||
fi
|
||||
kill -0 "$PUBLISHER_PID" 2>/dev/null || die "publisher exited early, see $run_dir/publisher.log"
|
||||
kill -0 "$SUBSCRIBER_PID" 2>/dev/null || die "subscriber exited early, see $run_dir/subscriber.log"
|
||||
[ "$i" = "120" ] && die "no frames received after 60s"
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
local t0
|
||||
t0=$(now_us)
|
||||
log "media flowing, starting measurement (t0=$t0)"
|
||||
cat > "$run_dir/meta.env" <<EOF
|
||||
MODE=$mode
|
||||
T0_US=$t0
|
||||
DURATION_S=$DURATION
|
||||
BASE_LOSS=$BASE_LOSS
|
||||
BURST_LOSS=$BURST_LOSS
|
||||
BURST_EVERY=$BURST_EVERY
|
||||
BURST_LEN=$BURST_LEN
|
||||
SHAPING=$SHAPING
|
||||
DEBUG_DROP=$DEBUG_DROP
|
||||
FEC_RATE=$FEC_RATE
|
||||
FEC_MASK_TYPE=$FEC_MASK_TYPE
|
||||
CODEC=$CODEC
|
||||
EOF
|
||||
|
||||
"$SCRIPT_DIR/prom_poll.sh" "$PROM_PORT" "$run_dir/prom.tsv" &
|
||||
PROM_POLL_PID=$!
|
||||
|
||||
if [ "$SHAPING" = "1" ]; then
|
||||
sudo "$SHAPER" start --port "$MEDIA_PORT" \
|
||||
--base-loss "$BASE_LOSS" --burst-loss "$BURST_LOSS" \
|
||||
--burst-every "$BURST_EVERY" --burst-len "$BURST_LEN" \
|
||||
--events "$run_dir/events.csv" \
|
||||
> "$run_dir/shaper.log" 2>&1 &
|
||||
SHAPER_PID=$!
|
||||
sleep 1
|
||||
kill -0 "$SHAPER_PID" 2>/dev/null || die "shaper failed to start, see $run_dir/shaper.log"
|
||||
fi
|
||||
|
||||
sleep "$DURATION"
|
||||
|
||||
log "measurement done, stopping"
|
||||
[ "$SHAPING" = "1" ] && stop_shaper
|
||||
echo "T1_US=$(now_us)" >> "$run_dir/meta.env"
|
||||
stop_pid "$PROM_POLL_PID"; PROM_POLL_PID=""
|
||||
stop_pid "$PUBLISHER_PID" INT; PUBLISHER_PID=""
|
||||
stop_pid "$SUBSCRIBER_PID" INT; SUBSCRIBER_PID=""
|
||||
# the final flexfec stats log line is emitted when the publisher's buffers close
|
||||
sleep 2
|
||||
grep -h "flexfec" "$run_dir/server.log" | tail -5 || true
|
||||
stop_pid "$SERVER_PID"; SERVER_PID=""
|
||||
sleep 1
|
||||
}
|
||||
|
||||
# ---------- runs + report ----------
|
||||
|
||||
case "$MODE" in
|
||||
fec) run_one fec ;;
|
||||
baseline) run_one baseline ;;
|
||||
ab)
|
||||
run_one baseline
|
||||
sleep 3
|
||||
run_one fec
|
||||
;;
|
||||
esac
|
||||
|
||||
log "generating report"
|
||||
if [ "$MODE" = "ab" ]; then
|
||||
python3 "$SCRIPT_DIR/plot_fec.py" --run "$OUT_DIR/baseline" --run "$OUT_DIR/fec" --out "$OUT_DIR" \
|
||||
| tee "$OUT_DIR/summary.txt"
|
||||
else
|
||||
python3 "$SCRIPT_DIR/plot_fec.py" --run "$OUT_DIR/$MODE" --out "$OUT_DIR" \
|
||||
| tee "$OUT_DIR/summary.txt"
|
||||
fi
|
||||
|
||||
log "done. results in $OUT_DIR"
|
||||
Executable
+112
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env bash
|
||||
# Traffic shaper for the FlexFEC test harness (Linux, tc/netem).
|
||||
#
|
||||
# Applies packet loss to UDP traffic destined to the SFU's pinned media port on
|
||||
# loopback, simulating a robot uplink over cellular: continuous low base loss
|
||||
# (Gilbert-Elliott model for realistic loss correlation) plus periodic
|
||||
# high-loss bursts. Burst on/off transitions are appended to an events file
|
||||
# (wall-clock microseconds) so plots can shade the burst windows.
|
||||
#
|
||||
# Usage:
|
||||
# sudo ./shape_linux.sh start --port 7882 --base-loss 0.02 \
|
||||
# --burst-loss 0.25 --burst-every 15 --burst-len 3 --events events.csv
|
||||
# sudo ./shape_linux.sh stop
|
||||
#
|
||||
# `start` runs in the foreground until terminated, cleaning up on exit.
|
||||
# `stop` force-cleans shaping state from a previous run.
|
||||
|
||||
set -u
|
||||
|
||||
DEV="lo"
|
||||
|
||||
PORT=7882
|
||||
BASE_LOSS=0.02
|
||||
BURST_LOSS=0.25
|
||||
BURST_EVERY=15
|
||||
BURST_LEN=3
|
||||
EVENTS_FILE=""
|
||||
|
||||
log() { echo "[shape_linux] $*" >&2; }
|
||||
|
||||
now_us() { python3 -c 'import time; print(int(time.time() * 1e6))'; }
|
||||
|
||||
record_event() {
|
||||
if [ -n "$EVENTS_FILE" ]; then
|
||||
echo "$(now_us),$1" >> "$EVENTS_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
pct() { python3 -c "print($1 * 100)"; }
|
||||
|
||||
apply_base_loss() {
|
||||
# Gilbert-Elliott: p = chance of entering the bad state, r = chance of
|
||||
# leaving it. p derived from the target average loss with r fixed at 30%
|
||||
# gives short correlated loss runs typical for radio links.
|
||||
local p
|
||||
p=$(pct "$BASE_LOSS")
|
||||
tc qdisc change dev $DEV parent 1:4 handle 40: netem loss gemodel "${p}%" 30%
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
trap - EXIT INT TERM
|
||||
log "cleaning up"
|
||||
tc qdisc del dev $DEV root 2>/dev/null
|
||||
record_event "shaper_stopped"
|
||||
log "done"
|
||||
}
|
||||
|
||||
start() {
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# 4-band prio qdisc: default TOS mapping never selects band 4, so only the
|
||||
# filtered SFU-bound UDP flow passes through the netem child
|
||||
tc qdisc add dev $DEV root handle 1: prio bands 4 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 || {
|
||||
log "failed to add root qdisc (already shaped? try '$0 stop')"
|
||||
exit 1
|
||||
}
|
||||
tc qdisc add dev $DEV parent 1:4 handle 40: netem loss gemodel "$(pct "$BASE_LOSS")%" 30%
|
||||
tc filter add dev $DEV parent 1: protocol ip prio 1 u32 \
|
||||
match ip protocol 17 0xff \
|
||||
match ip dport "$PORT" 0xffff \
|
||||
flowid 1:4
|
||||
|
||||
log "shaping active: udp dport $PORT, base loss $BASE_LOSS (gemodel), burst $BURST_LOSS for ${BURST_LEN}s every ${BURST_EVERY}s"
|
||||
record_event "shaper_started base=$BASE_LOSS burst=$BURST_LOSS"
|
||||
|
||||
# periodic burst loop
|
||||
while true; do
|
||||
sleep "$BURST_EVERY"
|
||||
tc qdisc change dev $DEV parent 1:4 handle 40: netem loss "$(pct "$BURST_LOSS")%"
|
||||
record_event "burst_on"
|
||||
log "burst on ($BURST_LOSS)"
|
||||
sleep "$BURST_LEN"
|
||||
apply_base_loss
|
||||
record_event "burst_off"
|
||||
log "burst off ($BASE_LOSS)"
|
||||
done
|
||||
}
|
||||
|
||||
CMD="${1:-}"
|
||||
shift || true
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--port) PORT="$2"; shift 2 ;;
|
||||
--base-loss) BASE_LOSS="$2"; shift 2 ;;
|
||||
--burst-loss) BURST_LOSS="$2"; shift 2 ;;
|
||||
--burst-every) BURST_EVERY="$2"; shift 2 ;;
|
||||
--burst-len) BURST_LEN="$2"; shift 2 ;;
|
||||
--events) EVENTS_FILE="$2"; shift 2 ;;
|
||||
*) log "unknown argument: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
log "ERROR: must run as root (sudo)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
case "$CMD" in
|
||||
start) start ;;
|
||||
stop) cleanup ;;
|
||||
*) echo "usage: $0 {start|stop} [--port N] [--base-loss F] [--burst-loss F] [--burst-every S] [--burst-len S] [--events FILE]" >&2; exit 1 ;;
|
||||
esac
|
||||
Executable
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# Traffic shaper for the FlexFEC test harness (macOS, dummynet via dnctl/pfctl).
|
||||
#
|
||||
# Applies packet loss to UDP traffic destined to the SFU's pinned media port on
|
||||
# loopback, simulating a robot uplink over cellular: continuous low base loss
|
||||
# plus periodic high-loss bursts. Burst on/off transitions are appended to an
|
||||
# events file (wall-clock microseconds) so plots can shade the burst windows.
|
||||
#
|
||||
# Usage:
|
||||
# sudo ./shape_macos.sh start --port 7882 --base-loss 0.02 \
|
||||
# --burst-loss 0.25 --burst-every 15 --burst-len 3 --events events.csv
|
||||
# sudo ./shape_macos.sh stop
|
||||
#
|
||||
# `start` runs in the foreground until terminated, cleaning up on exit.
|
||||
# `stop` force-cleans shaping state from a previous run.
|
||||
|
||||
set -u
|
||||
|
||||
ANCHOR="livekit_fec"
|
||||
PIPE=1
|
||||
STATE_DIR="${TMPDIR:-/tmp}/livekit_fec_shaper"
|
||||
|
||||
PORT=7882
|
||||
BASE_LOSS=0.02
|
||||
BURST_LOSS=0.25
|
||||
BURST_EVERY=15
|
||||
BURST_LEN=3
|
||||
EVENTS_FILE=""
|
||||
|
||||
log() { echo "[shape_macos] $*" >&2; }
|
||||
|
||||
now_us() { python3 -c 'import time; print(int(time.time() * 1e6))'; }
|
||||
|
||||
record_event() {
|
||||
if [ -n "$EVENTS_FILE" ]; then
|
||||
echo "$(now_us),$1" >> "$EVENTS_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
trap - EXIT INT TERM
|
||||
log "cleaning up"
|
||||
pfctl -a "$ANCHOR" -F all 2>/dev/null
|
||||
# restore the system ruleset, dropping our anchor attachment
|
||||
pfctl -f /etc/pf.conf 2>/dev/null
|
||||
dnctl -q flush 2>/dev/null
|
||||
if [ -f "$STATE_DIR/pf_token" ]; then
|
||||
pfctl -X "$(cat "$STATE_DIR/pf_token")" 2>/dev/null
|
||||
rm -f "$STATE_DIR/pf_token"
|
||||
fi
|
||||
record_event "shaper_stopped"
|
||||
log "done"
|
||||
}
|
||||
|
||||
start() {
|
||||
mkdir -p "$STATE_DIR"
|
||||
|
||||
if ! command -v dnctl >/dev/null; then
|
||||
log "ERROR: dnctl not found. dummynet is unavailable on this system,"
|
||||
log "consider Network Link Conditioner or running the test on Linux."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# configure the dummynet pipe with the base loss
|
||||
dnctl pipe $PIPE config plr "$BASE_LOSS" || { log "dnctl failed"; exit 1; }
|
||||
|
||||
# enable pf, keeping the reference token for clean disable
|
||||
local token
|
||||
token=$(pfctl -E 2>&1 | awk '/Token/ {print $NF}')
|
||||
if [ -n "$token" ]; then
|
||||
echo "$token" > "$STATE_DIR/pf_token"
|
||||
fi
|
||||
|
||||
# attach our dummynet anchor on top of the system ruleset
|
||||
pfctl -q -f - <<EOF
|
||||
include "/etc/pf.conf"
|
||||
dummynet-anchor "$ANCHOR"
|
||||
anchor "$ANCHOR"
|
||||
EOF
|
||||
|
||||
# shape UDP packets addressed to the SFU media port (publisher -> SFU leg)
|
||||
echo "dummynet in quick proto udp from any to any port $PORT pipe $PIPE" | \
|
||||
pfctl -q -a "$ANCHOR" -f -
|
||||
|
||||
log "shaping active: udp dport $PORT, base loss $BASE_LOSS, burst $BURST_LOSS for ${BURST_LEN}s every ${BURST_EVERY}s"
|
||||
record_event "shaper_started base=$BASE_LOSS burst=$BURST_LOSS"
|
||||
|
||||
# periodic burst loop
|
||||
while true; do
|
||||
sleep "$BURST_EVERY"
|
||||
dnctl pipe $PIPE config plr "$BURST_LOSS"
|
||||
record_event "burst_on"
|
||||
log "burst on ($BURST_LOSS)"
|
||||
sleep "$BURST_LEN"
|
||||
dnctl pipe $PIPE config plr "$BASE_LOSS"
|
||||
record_event "burst_off"
|
||||
log "burst off ($BASE_LOSS)"
|
||||
done
|
||||
}
|
||||
|
||||
CMD="${1:-}"
|
||||
shift || true
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--port) PORT="$2"; shift 2 ;;
|
||||
--base-loss) BASE_LOSS="$2"; shift 2 ;;
|
||||
--burst-loss) BURST_LOSS="$2"; shift 2 ;;
|
||||
--burst-every) BURST_EVERY="$2"; shift 2 ;;
|
||||
--burst-len) BURST_LEN="$2"; shift 2 ;;
|
||||
--events) EVENTS_FILE="$2"; shift 2 ;;
|
||||
*) log "unknown argument: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
log "ERROR: must run as root (sudo)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
case "$CMD" in
|
||||
start) start ;;
|
||||
stop) cleanup ;;
|
||||
*) echo "usage: $0 {start|stop} [--port N] [--base-loss F] [--burst-loss F] [--burst-every S] [--burst-len S] [--events FILE]" >&2; exit 1 ;;
|
||||
esac
|
||||
Executable
+152
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
# Parameter sweep over the FlexFEC harness: runs run_fec_test.sh across a matrix
|
||||
# of loss levels and FEC configurations, then aggregates all cells into a single
|
||||
# comparison report (sweep_report.png + sweep_summary.csv + a markdown table).
|
||||
#
|
||||
# At each loss level it runs one baseline (no FEC) plus one FEC run per
|
||||
# (rate, mask) combination, so every FEC point has a same-loss baseline to
|
||||
# compare against. The server and example binaries are built once and reused.
|
||||
#
|
||||
# Usage:
|
||||
# ./sweep_fec.sh [options]
|
||||
#
|
||||
# Options (defaults in brackets):
|
||||
# --loss-mode {debug|shaped} loss mechanism [debug]
|
||||
# debug = uniform packet drop inside the SFU, no sudo, --loss-list is %
|
||||
# shaped = OS traffic shaping bursts (needs sudo), --loss-list is the
|
||||
# burst loss fraction, base loss/cadence fixed by --base-loss etc.
|
||||
# --loss-list "L1 L2 .." loss levels to sweep [debug: "2 5 10 15"]
|
||||
# --fec-rate-list "R1 .." publisher FEC protection rates (percent) ["20 30 50"]
|
||||
# --mask-list "M1 .." FEC mask types: random and/or bursty ["bursty"]
|
||||
# --duration S measurement seconds per cell [60]
|
||||
# --codec C video codec [h264]
|
||||
# --base-loss F shaped mode: continuous base loss [0.01]
|
||||
# --burst-every S shaped mode: seconds between bursts [15]
|
||||
# --burst-len S shaped mode: burst duration [3]
|
||||
# --out DIR output directory [scripts/fec/out/sweep_<timestamp>]
|
||||
#
|
||||
# Example matrices:
|
||||
# # uniform-loss sweep (no sudo), 4 loss levels x 2 rates x 1 mask = 12 cells
|
||||
# ./sweep_fec.sh --loss-list "2 5 10 15" --fec-rate-list "20 50"
|
||||
#
|
||||
# # shaped cellular-burst sweep, vary burst intensity and compare masks
|
||||
# sudo -v && ./sweep_fec.sh --loss-mode shaped --loss-list "0.15 0.30 0.50" \
|
||||
# --fec-rate-list "30" --mask-list "random bursty"
|
||||
#
|
||||
# Runtime ~= cells * (~20s setup + duration). The example above is ~12 * 80s.
|
||||
|
||||
set -u
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
RUST_SDKS_DIR="${RUST_SDKS_DIR:-$(cd "$REPO_ROOT/.." && pwd)/rust-sdks}"
|
||||
RUNNER="$SCRIPT_DIR/run_fec_test.sh"
|
||||
|
||||
LOSS_MODE="debug"
|
||||
LOSS_LIST=""
|
||||
FEC_RATE_LIST="20 30 50"
|
||||
MASK_LIST="bursty"
|
||||
DURATION=60
|
||||
CODEC="h264"
|
||||
BASE_LOSS=0.01
|
||||
BURST_EVERY=15
|
||||
BURST_LEN=3
|
||||
OUT_DIR=""
|
||||
|
||||
log() { echo "[sweep] $*"; }
|
||||
die() { echo "[sweep] ERROR: $*" >&2; exit 1; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--loss-mode) LOSS_MODE="$2"; shift 2 ;;
|
||||
--loss-list) LOSS_LIST="$2"; shift 2 ;;
|
||||
--fec-rate-list) FEC_RATE_LIST="$2"; shift 2 ;;
|
||||
--mask-list) MASK_LIST="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--codec) CODEC="$2"; shift 2 ;;
|
||||
--base-loss) BASE_LOSS="$2"; shift 2 ;;
|
||||
--burst-every) BURST_EVERY="$2"; shift 2 ;;
|
||||
--burst-len) BURST_LEN="$2"; shift 2 ;;
|
||||
--out) OUT_DIR="$2"; shift 2 ;;
|
||||
-h|--help) sed -n '2,38p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$LOSS_MODE" in
|
||||
debug) : "${LOSS_LIST:=2 5 10 15}" ;;
|
||||
shaped) : "${LOSS_LIST:=0.15 0.30 0.50}" ;;
|
||||
*) die "--loss-mode must be debug or shaped" ;;
|
||||
esac
|
||||
|
||||
command -v go >/dev/null || die "go not found"
|
||||
command -v cargo >/dev/null || die "cargo not found"
|
||||
python3 -c 'import matplotlib' 2>/dev/null || die "python3 with matplotlib required"
|
||||
[ -d "$RUST_SDKS_DIR/examples/local_video" ] || die "rust-sdks not found at $RUST_SDKS_DIR"
|
||||
|
||||
if [ "$LOSS_MODE" = "shaped" ]; then
|
||||
sudo -v || die "shaped mode needs sudo (or use --loss-mode debug)"
|
||||
( while true; do sudo -n true 2>/dev/null; sleep 60; done ) &
|
||||
SUDO_KEEPALIVE_PID=$!
|
||||
trap '[ -n "${SUDO_KEEPALIVE_PID:-}" ] && kill "$SUDO_KEEPALIVE_PID" 2>/dev/null' EXIT
|
||||
fi
|
||||
|
||||
if [ -z "$OUT_DIR" ]; then
|
||||
OUT_DIR="$SCRIPT_DIR/out/sweep_$(date +%Y%m%d_%H%M%S)"
|
||||
fi
|
||||
mkdir -p "$OUT_DIR"
|
||||
log "output directory: $OUT_DIR"
|
||||
|
||||
# build once, reuse across all cells
|
||||
SERVER_BIN="$OUT_DIR/livekit-server"
|
||||
log "building livekit-server (once)..."
|
||||
(cd "$REPO_ROOT" && go build -o "$SERVER_BIN" ./cmd/server) || die "server build failed"
|
||||
log "building local_video examples (once)..."
|
||||
(cd "$RUST_SDKS_DIR" && cargo build --release -p local_video -F desktop --bin publisher --bin subscriber) \
|
||||
|| die "example build failed"
|
||||
|
||||
MANIFEST="$OUT_DIR/manifest.tsv"
|
||||
printf 'leaf\tmode\tloss\tfec_rate\tfec_mask\n' > "$MANIFEST"
|
||||
|
||||
run_cell() {
|
||||
# run_cell <mode> <cell_name> <extra args...>
|
||||
local mode="$1" cell="$2"; shift 2
|
||||
log "cell: $cell"
|
||||
"$RUNNER" --mode "$mode" --skip-build --server-bin "$SERVER_BIN" \
|
||||
--duration "$DURATION" --codec "$CODEC" --out "$OUT_DIR/$cell" "$@" \
|
||||
> "$OUT_DIR/$cell.log" 2>&1 || { log "WARNING: cell $cell failed, see $OUT_DIR/$cell.log"; return 1; }
|
||||
}
|
||||
|
||||
loss_args() {
|
||||
local loss="$1"
|
||||
if [ "$LOSS_MODE" = "debug" ]; then
|
||||
echo "--debug-drop $loss"
|
||||
else
|
||||
echo "--base-loss $BASE_LOSS --burst-loss $loss --burst-every $BURST_EVERY --burst-len $BURST_LEN"
|
||||
fi
|
||||
}
|
||||
|
||||
CELL_COUNT=0
|
||||
for loss in $LOSS_LIST; do
|
||||
la=$(loss_args "$loss")
|
||||
|
||||
base_cell="cell_loss${loss}_baseline"
|
||||
if run_cell baseline "$base_cell" $la; then
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "$base_cell/baseline" baseline "$loss" "-" "-" >> "$MANIFEST"
|
||||
fi
|
||||
CELL_COUNT=$((CELL_COUNT + 1))
|
||||
|
||||
for rate in $FEC_RATE_LIST; do
|
||||
for mask in $MASK_LIST; do
|
||||
fec_cell="cell_loss${loss}_rate${rate}_${mask}"
|
||||
if run_cell fec "$fec_cell" --fec-rate "$rate" --fec-mask-type "$mask" $la; then
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "$fec_cell/fec" fec "$loss" "$rate" "$mask" >> "$MANIFEST"
|
||||
fi
|
||||
CELL_COUNT=$((CELL_COUNT + 1))
|
||||
done
|
||||
done
|
||||
done
|
||||
|
||||
log "ran $CELL_COUNT cells, aggregating"
|
||||
python3 "$SCRIPT_DIR/aggregate_fec.py" --sweep "$OUT_DIR" | tee "$OUT_DIR/sweep_summary.txt"
|
||||
log "done. report in $OUT_DIR/sweep_report.png"
|
||||
Reference in New Issue
Block a user