initial fec implementation

scripts/flexfec/README.md

@@ -0,0 +1,145 @@
+# Local FlexFEC-03 test harness
+
+Scripts to validate end-to-end FlexFEC-03 between the rust-sdks `local_video`
+examples and a locally-built SFU, under simulated packet loss.
+
+FlexFEC is terminated at the SFU (it can't be passed through because the SFU
+rewrites SSRC/sequence numbers per subscriber), so the two directions are
+independent and tested separately:
+
+| Direction | What's exercised                                   | Loss to inject |
+| --------- | -------------------------------------------------- | -------------- |
+| Downlink  | SFU **generates** a repair stream per subscriber   | `down`         |
+| Uplink    | SFU **receives + decodes** a publisher repair flow | `up`           |
+
+## Prerequisites
+
+- Go toolchain (to build this SFU).
+- A `rust-sdks5` checkout with the FlexFEC changes, next to this repo
+  (`../rust-sdks5`) or pointed at via `RUST_SDKS_DIR`. Its `webrtc-sys` must be
+  built (the C++ field trials that enable FlexFEC-03 live there).
+- A camera (the publisher captures real video).
+- `sudo` for traffic shaping. macOS uses `dnctl`/`pfctl` (dummynet); Linux uses
+  `tc netem`.
+
+All processes run on one machine, so media flows over loopback. The SFU is pinned
+to UDP `7882` (see `flexfec-local.yaml`) so shaping can target it by port.
+
+## Files
+
+| File                 | Purpose                                              |
+| -------------------- | ---------------------------------------------------- |
+| `flexfec-local.yaml` | SFU config: flexfec on (pub+sub), fixed loopback port |
+| `run-sfu.sh`         | Build + run the SFU (`--dev`, devkey/secret)         |
+| `netem.sh`           | Start/stop/inspect packet-loss shaping on udp/7882   |
+| `run-publisher.sh`   | Run the publisher with `--flex-fec`                  |
+| `run-subscriber.sh`  | Run the subscriber, surface `Video FEC ...` logs     |
+
+## Workflow
+
+Use four terminals.
+
+**1 — SFU**
+
+```bash
+scripts/flexfec/run-sfu.sh
+```
+
+Connects on `ws://127.0.0.1:7880` (API key `devkey` / secret `secret`).
+
+**2 — Subscriber**
+
+```bash
+scripts/flexfec/run-subscriber.sh
+# or only show FEC lines on the console:
+FEC_ONLY=1 scripts/flexfec/run-subscriber.sh
+```
+
+**3 — Publisher** (grant camera permission on first run)
+
+```bash
+scripts/flexfec/run-publisher.sh
+```
+
+**4 — Packet loss** (needs sudo)
+
+```bash
+# Uplink test: drop 8% of publisher -> SFU packets
+sudo scripts/flexfec/netem.sh start 8 up
+
+# Downlink test: drop 8% of SFU -> subscriber packets (+ optional delay_ms, see A/B below)
+sudo scripts/flexfec/netem.sh start 8 down
+sudo scripts/flexfec/netem.sh start 8 down 150   # 8% loss + 150ms one-way delay
+
+sudo scripts/flexfec/netem.sh status
+sudo scripts/flexfec/netem.sh stop   # always clean up when done
+```
+
+## What success looks like
+
+**Downlink (`down`)** — the subscriber recovers SFU-generated repair packets.
+In the subscriber log (`logs/subscriber.log`):
+
+```
+Video FEC active: fec_ssrc=..., received_total=..., discarded_total=..., accepted_recovery_payload_total=...
+Video FEC recovery payload accepted: +N packets (accepted_total=...)
+```
+
+`accepted_recovery_payload_total` should climb while loss is applied.
+
+**Uplink (`up`)** — the SFU receives + decodes the publisher's repair stream.
+In the SFU log:
+
+```
+flexfec decoder enabled   {"fecSSRC": ..., "mediaSSRC": ...}
+flexfec recovery stats    {"fecPacketsReceived": ..., "packetsRecovered": ..., "packetsRecoveredDelta": ...}
+```
+
+`packetsRecovered` increasing confirms the SFU rebuilt lost uplink packets.
+
+To also see FlexFEC SDP negotiation (`flexfec ... from sdp`, codec registration),
+set `logging.level: debug` in `flexfec-local.yaml`.
+
+### Subscriber-side A/B (isolating FEC from RTX)
+
+On loopback RTT~=0, so NACK/RTX recovers nearly all loss before FlexFEC matters and
+its benefit is invisible. To make FEC the primary recovery path, add one-way delay so
+retransmissions arrive too late, and compare FEC on vs off under the same conditions:
+
+```bash
+# A) FEC ON (flexfec.subscriber: true). With SFU + pub + sub running:
+sudo scripts/flexfec/netem.sh start 8 down 150     # 8% loss + 150ms delay, downlink
+# ...observe subscriber for ~20s, then...
+sudo scripts/flexfec/netem.sh stop
+
+# B) FEC OFF: set flexfec.subscriber: false in flexfec-local.yaml, restart the SFU,
+#    reconnect the subscriber, then apply the SAME shaping and observe.
+```
+
+The subscriber logs a `Video quality:` line every ~3s:
+
+```
+Video quality: frames_decoded=..., frames_dropped=..., freezes=N (Xs), nack=..., pli=...,
+               packets_lost=..., rtx_recovered=..., fec_recv=..., fec_discarded=...
+```
+
+With FEC ON you expect fewer `freezes`/`frames_dropped`, lower net `packets_lost`, and a
+meaningful `fec_recv`; with FEC OFF the same loss leans entirely on `rtx_recovered` and
+(because of the delay) typically shows more freezes and dropped frames.
+
+## Notes & caveats
+
+- **Loopback shaping (macOS):** `netem.sh` loads its dummynet rules into a dedicated
+  `flexfec-loss` pf anchor and re-applies `/etc/pf.conf` non-destructively. `stop`
+  flushes the pipes and restores pf to its prior enabled/disabled state. If pf is set
+  to `skip on lo0` the script warns and shaping won't apply.
+- **Direction isolation is approximate.** Both clients share the SFU UDP port, so
+  `up`/`down` are distinguished by source vs. destination port. The publisher is the
+  dominant sender to the SFU and the subscriber the dominant receiver, so this is good
+  enough to isolate each path in practice (some RTCP is also affected).
+- **`dnctl -q flush` clears all dummynet pipes**, not just ours — fine on a dev box.
+- **Send-side BWE:** if the SFU uses pacer-based send-side BWE, subscriber FEC overhead
+  isn't accounted for by congestion control (logged as a warning by the SFU). The
+  interceptor-based send-side BWE path TWCC-tags FEC packets correctly.
+- Increase loss gradually (e.g. 3% → 10%). Beyond what a single FlexFEC repair packet
+  can cover, bursts become unrecoverable (expected).

scripts/flexfec/flexfec-linux.yaml

@@ -0,0 +1,45 @@
+# Linux FlexFEC-03 test config for the netns + netem harness (scripts/flexfec/netns.sh).
+#
+# Unlike flexfec-local.yaml (macOS, loopback + in-SFU software impairment), this runs the
+# SFU in the root netns and advertises the veth host IP 10.200.0.1 so that:
+#   - the publisher in the `robot` netns reaches the SFU over the veth pair (shaped by netem)
+#   - the subscriber in the root netns reaches the SFU at 10.200.0.1 locally (clean link)
+#
+# Run with --dev (injects devkey/secret):
+#   ~/go1.26/bin/... # build first, then:
+#   sudo scripts/flexfec/netns.sh up
+#   /tmp/flexfec-livekit-server --dev --config scripts/flexfec/flexfec-linux.yaml
+
+port: 7880
+
+# Bind signalling (HTTP/WS + TCP) to the veth host IP so the publisher in the `robot` netns
+# can reach it, plus loopback for the in-root-ns subscriber. Setting a non-loopback bind
+# address also stops --dev from pinning everything to 127.0.0.1.
+bind_addresses:
+  - 10.200.0.1
+  - 127.0.0.1
+
+rtc:
+  udp_port: 7882
+  tcp_port: 7881
+  # Advertise the veth host IP, not a public IP.
+  use_external_ip: false
+  enable_loopback_candidate: true
+  ips:
+    includes:
+      # veth subnet: 10.200.0.1 (SFU/root) <-> 10.200.0.2 (robot netns)
+      - 10.200.0.0/24
+      - 127.0.0.0/8
+
+  # Asymmetric teleop topology: robot publishes with FlexFEC (uplink recovery at the SFU);
+  # the operator/subscriber is on a clean wired link so downlink FEC is off.
+  flexfec:
+    subscriber: false
+    publisher: true
+    payload_type: 49
+    num_media_packets: 5
+    num_fec_packets: 1
+
+logging:
+  level: debug
+  json: false

scripts/flexfec/flexfec-local.yaml

@@ -0,0 +1,48 @@
+# Local FlexFEC-03 test config for the LiveKit SFU.
+#
+# Intended to be run together with `--dev` (which injects the devkey/secret API key
+# pair and binds to loopback):
+#
+#   go run ./cmd/server --dev --config scripts/flexfec/flexfec-local.yaml
+#
+# Media is pinned to a single UDP port on loopback so traffic shaping (see netem.sh)
+# can target it deterministically.
+
+port: 7880
+
+rtc:
+  # Single, fixed media port (no port_range_* so this takes effect). Shape this port
+  # with scripts/flexfec/netem.sh to simulate packet loss.
+  udp_port: 7882
+  tcp_port: 7881
+  # Keep everything on the local machine; do not try to discover a public IP.
+  use_external_ip: false
+  # Gather loopback (127.0.0.1) candidates and pin ICE to IPv4 loopback so all media flows
+  # over a single, stable path. NOTE: on an all-on-one-box macOS setup, pf/dummynet
+  # (netem.sh) cannot shape this traffic -- the kernel short-circuits same-host UDP delivery
+  # before the pf hook -- so loss/delay is injected in-SFU instead via the LK_DOWNLINK_*
+  # env vars (see pkg/sfu/downtrack_impair.go and run-sfu.sh). The media path is therefore
+  # irrelevant to shaping, so loopback (no dependency on a volatile LAN IP) is preferred.
+  enable_loopback_candidate: true
+  ips:
+    includes:
+      - 127.0.0.0/8
+
+  # FlexFEC-03 for both directions:
+  #   publisher:  receive + decode a publisher's repair stream (uplink recovery)
+  #   subscriber: generate a repair stream per subscriber (downlink recovery)
+  flexfec:
+    subscriber: false
+    publisher: true
+    payload_type: 49
+    # Low-latency teleop tuning: 5-packet block with 1 repair = 20% overhead, but the
+    # repair lands ~16ms after the block (vs ~32ms for a 10-block), well under an 80ms
+    # RTX round trip. Recovers 1 scattered loss per 5-block (ample for ~1% random loss).
+    num_media_packets: 5
+    num_fec_packets: 1
+
+logging:
+  # info surfaces the "flexfec decoder enabled" and periodic "flexfec recovery stats"
+  # logs. Bump to debug to also see SDP-level FEC negotiation ("flexfec ... from sdp").
+  level: debug
+  json: false

scripts/flexfec/netem.sh

@@ -0,0 +1,289 @@
+#!/usr/bin/env bash
+#
+# Simulate packet loss on the SFU's media UDP port so FlexFEC recovery can be
+# observed end-to-end. Works on macOS (dnctl/dummynet + pfctl) and Linux (tc netem).
+#
+# Because publisher, SFU and subscriber all run on one machine, media flows over the
+# loopback interface. Loss is matched by the SFU UDP port (default 7882):
+#   - "down" drops packets coming FROM the SFU port  -> SFU -> subscriber path
+#            (exercises subscriber-side / downlink FEC generation+recovery)
+#   - "up"   drops packets going TO the SFU port      -> publisher -> SFU path
+#            (exercises publisher-side / uplink FEC reception+recovery)
+#   - "both" drops in both directions
+#
+# Usage:
+#   sudo scripts/flexfec/netem.sh start [loss_percent] [up|down|both] [delay_ms]  # uniform loss; default: 5 down 0
+#   sudo scripts/flexfec/netem.sh burst [loss_percent] [burst_ms] [gap_ms] [up|down|both] [delay_ms]  # bursty loss
+#   sudo scripts/flexfec/netem.sh stop
+#   sudo scripts/flexfec/netem.sh status
+#
+# delay_ms adds one-way latency (so RTT ~= 2*delay_ms on loopback). Models the
+# robot's access-network round trip; RTX recovery costs ~1 RTT of buffering.
+#
+# "burst" emulates handoff/fade-style correlated loss: it toggles the loss rate
+# between loss_percent (for burst_ms) and 0 (for gap_ms) on a duty cycle, holding
+# the delay constant. Average loss ~= loss_percent * burst_ms/(burst_ms+gap_ms).
+# Run it in its own terminal; Ctrl-C stops and restores the network.
+# This is where RTX (retransmit storms, each costing an RTT) and FlexFEC (fixed
+# inline overhead) diverge most.
+#
+# Env overrides:
+#   SFU_UDP_PORT  media port to shape (default 7882)
+#   NETEM_IFACE   (Linux only) interface to shape (default lo)
+set -euo pipefail
+
+PORT="${SFU_UDP_PORT:-7882}"
+ANCHOR="flexfec-loss"
+STATE_DIR="${TMPDIR:-/tmp}/flexfec-netem"
+PF_STATE="$STATE_DIR/pf.enabled"
+OS="$(uname -s)"
+
+require_root() {
+  if [[ "$(id -u)" -ne 0 ]]; then
+    echo "error: this command needs root; re-run with sudo" >&2
+    exit 1
+  fi
+}
+
+validate_dir() {
+  case "$1" in
+    up | down | both) ;;
+    *)
+      echo "error: direction must be up|down|both (got '$1')" >&2
+      exit 1
+      ;;
+  esac
+}
+
+# ----------------------------------------------------------------------------- macOS
+
+# Install the pf anchor + dummynet rules for the requested direction. Pipes are
+# (re)configured separately so loss can be toggled live for burst mode.
+macos_setup() {
+  local dir="$1"
+
+  mkdir -p "$STATE_DIR"
+
+  # Remember whether pf was already enabled so `stop` can restore it.
+  if [[ ! -f "$PF_STATE" ]]; then
+    if pfctl -s info 2>/dev/null | grep -q 'Status: Enabled'; then
+      echo enabled >"$PF_STATE"
+    else
+      echo disabled >"$PF_STATE"
+    fi
+  fi
+
+  local rules=""
+  case "$dir" in
+    down | both) rules+="dummynet out proto udp from any port $PORT to any pipe 1"$'\n' ;;
+  esac
+  case "$dir" in
+    up | both) rules+="dummynet out proto udp from any to any port $PORT pipe 2"$'\n' ;;
+  esac
+
+  # Re-load the system ruleset plus references to our named anchor (non-destructive:
+  # existing /etc/pf.conf rules are preserved), then install our rules in the anchor.
+  {
+    cat /etc/pf.conf 2>/dev/null || true
+    echo "dummynet-anchor \"$ANCHOR\""
+    echo "anchor \"$ANCHOR\""
+  } | pfctl -f - 2>/dev/null
+  printf '%s' "$rules" | pfctl -a "$ANCHOR" -f - 2>/dev/null
+  pfctl -e 2>/dev/null || true
+
+  if pfctl -sa 2>/dev/null | grep -qi 'skip on lo'; then
+    echo "warning: pf is configured to 'skip on lo'; loopback shaping may not apply" >&2
+  fi
+}
+
+# macos_set_loss <loss_percent> <delay_ms>
+macos_set_loss() {
+  local plr
+  plr="$(awk "BEGIN { printf \"%.4f\", $1 / 100 }")"
+  dnctl pipe 1 config plr "$plr" delay "$2" >/dev/null
+  dnctl pipe 2 config plr "$plr" delay "$2" >/dev/null
+}
+
+macos_start() {
+  local loss="$1" dir="$2" delay="$3"
+  macos_setup "$dir"
+  macos_set_loss "$loss" "$delay"
+  echo "==> macOS dummynet loss enabled: ${loss}% loss, ${delay}ms delay (${dir}) on udp/${PORT}"
+}
+
+macos_burst() {
+  local loss="$1" burst_ms="$2" gap_ms="$3" dir="$4" delay="$5"
+  local burst_s gap_s avg
+  burst_s="$(awk "BEGIN { printf \"%.3f\", $burst_ms / 1000 }")"
+  gap_s="$(awk "BEGIN { printf \"%.3f\", $gap_ms / 1000 }")"
+  avg="$(awk "BEGIN { printf \"%.1f\", $loss * $burst_ms / ($burst_ms + $gap_ms) }")"
+
+  macos_setup "$dir"
+  macos_set_loss 0 "$delay"
+
+  trap 'echo; echo "==> stopping burst"; macos_stop; exit 0' INT TERM
+  echo "==> macOS bursty loss: ${loss}% for ${burst_ms}ms every $((burst_ms + gap_ms))ms (~${avg}% avg), ${delay}ms delay (${dir}) on udp/${PORT}"
+  echo "    Ctrl-C to stop and restore the network."
+  while true; do
+    macos_set_loss "$loss" "$delay"
+    printf '\r  [burst ON  %s%% loss]   ' "$loss"
+    sleep "$burst_s"
+    macos_set_loss 0 "$delay"
+    printf '\r  [burst off  0%% loss]   ' "$loss"
+    sleep "$gap_s"
+  done
+}
+
+macos_stop() {
+  dnctl -q flush 2>/dev/null || true
+  printf '' | pfctl -a "$ANCHOR" -f - 2>/dev/null || true
+  pfctl -f /etc/pf.conf 2>/dev/null || true
+  if [[ -f "$PF_STATE" && "$(cat "$PF_STATE")" == "disabled" ]]; then
+    pfctl -d 2>/dev/null || true
+  fi
+  rm -f "$PF_STATE"
+  echo "==> macOS dummynet loss disabled"
+}
+
+macos_status() {
+  echo "== dummynet pipes =="
+  dnctl list 2>/dev/null || true
+  echo "== pf anchor '$ANCHOR' =="
+  pfctl -a "$ANCHOR" -s rules 2>/dev/null || true
+  echo "== pf status =="
+  pfctl -s info 2>/dev/null | head -1 || true
+}
+
+# ----------------------------------------------------------------------------- Linux
+
+# linux_setup <dir> <delay_ms>: build prio qdisc + netem band (loss starts at 0)
+# and funnel matching UDP packets into it.
+linux_setup() {
+  local dir="$1" delay="$2"
+  local dev="${NETEM_IFACE:-lo}"
+
+  local netem_args=(loss 0%)
+  if [[ "$delay" -gt 0 ]]; then
+    netem_args+=(delay "${delay}ms")
+  fi
+
+  tc qdisc del dev "$dev" root 2>/dev/null || true
+  tc qdisc add dev "$dev" root handle 1: prio
+  tc qdisc add dev "$dev" parent 1:3 handle 30: netem "${netem_args[@]}"
+
+  if [[ "$dir" == "down" || "$dir" == "both" ]]; then
+    tc filter add dev "$dev" parent 1:0 protocol ip prio 3 u32 \
+      match ip protocol 17 0xff match ip sport "$PORT" 0xffff flowid 1:3
+  fi
+  if [[ "$dir" == "up" || "$dir" == "both" ]]; then
+    tc filter add dev "$dev" parent 1:0 protocol ip prio 3 u32 \
+      match ip protocol 17 0xff match ip dport "$PORT" 0xffff flowid 1:3
+  fi
+}
+
+# linux_set_loss <loss_percent> <delay_ms>
+linux_set_loss() {
+  local dev="${NETEM_IFACE:-lo}"
+  if [[ "$2" -gt 0 ]]; then
+    tc qdisc change dev "$dev" parent 1:3 handle 30: netem loss "${1}%" delay "${2}ms"
+  else
+    tc qdisc change dev "$dev" parent 1:3 handle 30: netem loss "${1}%"
+  fi
+}
+
+linux_start() {
+  local loss="$1" dir="$2" delay="$3"
+  local dev="${NETEM_IFACE:-lo}"
+  linux_setup "$dir" "$delay"
+  linux_set_loss "$loss" "$delay"
+  echo "==> Linux netem enabled: ${loss}% loss, ${delay}ms delay (${dir}) on udp/${PORT} dev ${dev}"
+}
+
+linux_burst() {
+  local loss="$1" burst_ms="$2" gap_ms="$3" dir="$4" delay="$5"
+  local dev="${NETEM_IFACE:-lo}"
+  local burst_s gap_s avg
+  burst_s="$(awk "BEGIN { printf \"%.3f\", $burst_ms / 1000 }")"
+  gap_s="$(awk "BEGIN { printf \"%.3f\", $gap_ms / 1000 }")"
+  avg="$(awk "BEGIN { printf \"%.1f\", $loss * $burst_ms / ($burst_ms + $gap_ms) }")"
+
+  linux_setup "$dir" "$delay"
+
+  trap 'echo; echo "==> stopping burst"; linux_stop; exit 0' INT TERM
+  echo "==> Linux bursty loss: ${loss}% for ${burst_ms}ms every $((burst_ms + gap_ms))ms (~${avg}% avg), ${delay}ms delay (${dir}) on udp/${PORT} dev ${dev}"
+  echo "    Ctrl-C to stop and restore the network."
+  while true; do
+    linux_set_loss "$loss" "$delay"
+    printf '\r  [burst ON  %s%% loss]   ' "$loss"
+    sleep "$burst_s"
+    linux_set_loss 0 "$delay"
+    printf '\r  [burst off  0%% loss]   ' "$loss"
+    sleep "$gap_s"
+  done
+}
+
+linux_stop() {
+  local dev="${NETEM_IFACE:-lo}"
+  tc qdisc del dev "$dev" root 2>/dev/null || true
+  echo "==> Linux netem loss disabled (dev ${dev})"
+}
+
+linux_status() {
+  local dev="${NETEM_IFACE:-lo}"
+  echo "== qdisc (dev ${dev}) =="
+  tc qdisc show dev "$dev" || true
+  echo "== filters (dev ${dev}) =="
+  tc filter show dev "$dev" || true
+}
+
+# ----------------------------------------------------------------------------- dispatch
+
+cmd="${1:-}"
+
+case "$cmd" in
+  start)
+    require_root
+    loss="${2:-5}"
+    dir="${3:-down}"
+    delay="${4:-0}"
+    validate_dir "$dir"
+    case "$OS" in
+      Darwin) macos_start "$loss" "$dir" "$delay" ;;
+      Linux) linux_start "$loss" "$dir" "$delay" ;;
+      *) echo "error: unsupported OS '$OS'" >&2; exit 1 ;;
+    esac
+    ;;
+  burst)
+    require_root
+    loss="${2:-40}"
+    burst_ms="${3:-200}"
+    gap_ms="${4:-2000}"
+    dir="${5:-down}"
+    delay="${6:-20}"
+    validate_dir "$dir"
+    case "$OS" in
+      Darwin) macos_burst "$loss" "$burst_ms" "$gap_ms" "$dir" "$delay" ;;
+      Linux) linux_burst "$loss" "$burst_ms" "$gap_ms" "$dir" "$delay" ;;
+      *) echo "error: unsupported OS '$OS'" >&2; exit 1 ;;
+    esac
+    ;;
+  stop)
+    require_root
+    case "$OS" in
+      Darwin) macos_stop ;;
+      Linux) linux_stop ;;
+    esac
+    ;;
+  status)
+    case "$OS" in
+      Darwin) macos_status ;;
+      Linux) linux_status ;;
+    esac
+    ;;
+  *)
+    echo "usage: sudo $0 {start [loss%] [up|down|both] [delay_ms]" >&2
+    echo "               | burst [loss%] [burst_ms] [gap_ms] [up|down|both] [delay_ms]" >&2
+    echo "               | stop | status}" >&2
+    exit 1
+    ;;
+esac

scripts/flexfec/netns.sh

@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+#
+# Linux network-namespace + netem harness for the asymmetric teleop FlexFEC test.
+#
+# Topology (models a robot on a lossy 5G uplink talking to a wired operator):
+#
+#     robot netns (publisher)            root netns (SFU + subscriber/operator)
+#     10.200.0.2  veth-robot  <=========>  veth-host 10.200.0.1   [SFU :7880/:7882]
+#                              netem here                          [subscriber -> 10.200.0.1]
+#
+# Only the robot<->SFU leg traverses the veth pair, so netem on the veth shapes the
+# robot uplink/downlink ONLY. The subscriber (operator) sits in the root netns and reaches
+# the SFU at 10.200.0.1 locally, so its link stays clean -- exactly the teleop case where
+# the robot is on 5G and the operator is on a stable hardline.
+#
+# Loss is applied to the robot->SFU (uplink) direction; delay is applied symmetrically so a
+# NACK->RTX recovery costs a full RTT (the latency publisher-side FlexFEC avoids by letting
+# the SFU repair the stream inline).
+#
+# Usage:
+#   sudo scripts/flexfec/netns.sh up                 # create ns + veth (no impairment)
+#   sudo scripts/flexfec/netns.sh shape <loss%> <owd_ms>   # apply/replace netem
+#   sudo scripts/flexfec/netns.sh clear              # remove netem (clean link)
+#   sudo scripts/flexfec/netns.sh status             # show qdiscs + addrs
+#   sudo scripts/flexfec/netns.sh down               # tear everything down
+#   scripts/flexfec/netns.sh exec <cmd...>           # run cmd inside robot netns (as $SUDO_USER)
+#
+# Example:
+#   sudo scripts/flexfec/netns.sh up
+#   sudo scripts/flexfec/netns.sh shape 3 20         # 3% uplink loss, 20ms each way (40ms RTT)
+set -euo pipefail
+
+NS=robot
+VETH_HOST=veth-host
+VETH_ROBOT=veth-robot
+HOST_IP=10.200.0.1
+ROBOT_IP=10.200.0.2
+PREFIX=24
+
+# Privileged commands are run via passwordless sudo for ip/tc only (see the sudoers drop-in
+# in the harness docs). The script itself does NOT need to run as root.
+if [[ "$(id -u)" == "0" ]]; then
+  SUDO=""
+else
+  SUDO="sudo -n"
+fi
+
+cmd_up() {
+  # Idempotent: tear down any prior instance first.
+  $SUDO ip netns del "$NS" 2>/dev/null || true
+  $SUDO ip link del "$VETH_HOST" 2>/dev/null || true
+
+  $SUDO ip netns add "$NS"
+  $SUDO ip link add "$VETH_HOST" type veth peer name "$VETH_ROBOT"
+  $SUDO ip link set "$VETH_ROBOT" netns "$NS"
+
+  $SUDO ip addr add "$HOST_IP/$PREFIX" dev "$VETH_HOST"
+  $SUDO ip link set "$VETH_HOST" up
+
+  $SUDO ip netns exec "$NS" ip addr add "$ROBOT_IP/$PREFIX" dev "$VETH_ROBOT"
+  $SUDO ip netns exec "$NS" ip link set "$VETH_ROBOT" up
+  $SUDO ip netns exec "$NS" ip link set lo up
+
+  echo "up: $NS netns ready"
+  echo "  root: $VETH_HOST $HOST_IP/$PREFIX   robot: $VETH_ROBOT $ROBOT_IP/$PREFIX"
+  echo "  SFU should advertise $HOST_IP (use flexfec-linux.yaml); publisher connects to ws://$HOST_IP:7880"
+}
+
+# shape <loss_percent> <one_way_delay_ms> [jitter_ms] [corr_pct]
+# When jitter_ms/corr_pct are given they feed netem's loss correlation + delay jitter to
+# approximate bursty (e.g. 5G handoff) conditions rather than uniform random loss.
+cmd_shape() {
+  local loss="${1:?usage: shape <loss%> <owd_ms> [jitter_ms] [loss_corr%]}"
+  local owd="${2:?usage: shape <loss%> <owd_ms> [jitter_ms] [loss_corr%]}"
+  local jitter="${3:-0}"
+  local corr="${4:-0}"
+
+  local delayspec="delay ${owd}ms"
+  [[ "$jitter" != "0" ]] && delayspec="delay ${owd}ms ${jitter}ms distribution normal"
+  local lossspec="loss ${loss}%"
+  [[ "$corr" != "0" ]] && lossspec="loss ${loss}% ${corr}%"
+
+  # Uplink (robot -> SFU): loss + delay. Applied on robot-side egress.
+  $SUDO ip netns exec "$NS" tc qdisc replace dev "$VETH_ROBOT" root netem $delayspec $lossspec
+  # Downlink (SFU -> robot, carries NACK/RTCP/PLI): delay only, no loss (operator link is
+  # clean, but the return path still costs propagation delay so RTX pays a full RTT).
+  $SUDO tc qdisc replace dev "$VETH_HOST" root netem delay "${owd}ms"
+
+  echo "shape: uplink ${lossspec} ${delayspec} ; downlink delay=${owd}ms (RTT≈$((owd*2))ms)"
+}
+
+cmd_clear() {
+  $SUDO ip netns exec "$NS" tc qdisc del dev "$VETH_ROBOT" root 2>/dev/null || true
+  $SUDO tc qdisc del dev "$VETH_HOST" root 2>/dev/null || true
+  echo "clear: netem removed (clean link)"
+}
+
+cmd_status() {
+  echo "=== root ns: $VETH_HOST ==="
+  $SUDO ip addr show "$VETH_HOST" 2>/dev/null | sed -n '2,4p' || echo "  (absent)"
+  $SUDO tc qdisc show dev "$VETH_HOST" 2>/dev/null || true
+  echo "=== robot ns: $VETH_ROBOT ==="
+  $SUDO ip netns exec "$NS" ip addr show "$VETH_ROBOT" 2>/dev/null | sed -n '2,4p' || echo "  (absent)"
+  $SUDO ip netns exec "$NS" tc qdisc show dev "$VETH_ROBOT" 2>/dev/null || true
+}
+
+cmd_down() {
+  $SUDO ip netns del "$NS" 2>/dev/null || true
+  $SUDO ip link del "$VETH_HOST" 2>/dev/null || true
+  echo "down: torn down"
+}
+
+# exec <cmd...> : run inside robot netns, dropping back to the invoking user. The outer
+# `sudo ip netns exec` runs as root; the inner `sudo -u $user` drops privileges (root -> user
+# needs no password), so the publisher runs as the normal user with its env intact.
+cmd_exec() {
+  local user="${SUDO_USER:-$USER}"
+  exec $SUDO ip netns exec "$NS" sudo -u "$user" --preserve-env=LIVEKIT_URL,LIVEKIT_API_KEY,LIVEKIT_API_SECRET,RUST_LOG,ROOM,PUB_IDENTITY,PATH,HOME,CARGO_HOME,RUSTUP_HOME "$@"
+}
+
+action="${1:-}"
+shift || true
+case "$action" in
+  up)     cmd_up "$@" ;;
+  shape)  cmd_shape "$@" ;;
+  clear)  cmd_clear "$@" ;;
+  status) cmd_status "$@" ;;
+  down)   cmd_down "$@" ;;
+  exec)   cmd_exec "$@" ;;
+  *) echo "usage: $0 {up|shape <loss%> <owd_ms>|clear|status|down|exec <cmd...>}" >&2; exit 2 ;;
+esac

scripts/flexfec/run-leg-linux.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+#
+# Run one FlexFEC test "leg" on the Linux box: start the SFU, a publisher (robot), and a
+# subscriber (operator), let them run, then print a summary of SFU FEC recovery + the
+# subscriber's frame-latency / video-quality stats.
+#
+# Two modes:
+#   - loopback (default): everything on 127.0.0.1, no impairment. Smoke test.
+#   - netns (USE_NETNS=1): publisher runs inside the `robot` netns (set up + shaped via
+#     netns.sh), SFU + subscriber run in the root netns reaching the SFU at 10.200.0.1.
+#     This models a lossy robot uplink with a clean operator downlink.
+#
+# Env:
+#   LABEL          leg label for the banner
+#   DURATION       seconds to run (default 35)
+#   USE_NETNS      1 to run publisher in the robot netns (default 0 = loopback)
+#   SFU_BIN        path to SFU binary (default /tmp/flexfec-livekit-server)
+#   RUST_DIR       rust-sdks5 dir (default ~/workspace/rust-sdks5)
+#   LK_DIR         livekit-flexfec dir (default ~/workspace/livekit-flexfec)
+#   PUB_EXTRA      extra publisher args (e.g. "--max-playout-delay 30")
+set -uo pipefail
+
+LABEL="${LABEL:-leg}"
+DURATION="${DURATION:-35}"
+USE_NETNS="${USE_NETNS:-0}"
+SFU_BIN="${SFU_BIN:-/tmp/flexfec-livekit-server}"
+RUST_DIR="${RUST_DIR:-$HOME/workspace/rust-sdks5}"
+LK_DIR="${LK_DIR:-$HOME/workspace/livekit-flexfec}"
+PUB_EXTRA="${PUB_EXTRA:-}"
+
+if [[ "$USE_NETNS" == "1" ]]; then
+  SFU_HOST="10.200.0.1"
+else
+  SFU_HOST="127.0.0.1"
+fi
+
+# FEC=on (default) uses flexfec-linux.yaml (publisher: true). FEC=off writes a sibling
+# config with publisher: false so the SFU does not negotiate uplink FlexFEC and the robot
+# falls back to RTX-only -- the A/B baseline.
+BASE_CONFIG="$LK_DIR/scripts/flexfec/flexfec-linux.yaml"
+if [[ "${FEC:-on}" == "off" ]]; then
+  CONFIG="/tmp/flexfec-linux-fecoff.yaml"
+  sed -E 's/^( *publisher:) *true/\1 false/' "$BASE_CONFIG" > "$CONFIG"
+else
+  CONFIG="$BASE_CONFIG"
+fi
+
+LOGDIR=/tmp/flexfec-logs
+mkdir -p "$LOGDIR"
+SFU_LOG="$LOGDIR/sfu.log"
+PUB_LOG="$LOGDIR/pub.log"
+SUB_LOG="$LOGDIR/sub.log"
+
+cleanup() {
+  pkill -f flexfec-livekit-server 2>/dev/null
+  pkill -f "target/debug/publisher" 2>/dev/null
+  pkill -f "target/debug/subscriber" 2>/dev/null
+}
+cleanup
+sleep 1
+
+echo "==================== LEG: $LABEL (netns=$USE_NETNS, ${DURATION}s) ===================="
+
+# In netns mode, apply netem shaping to the robot uplink. NETEM_LOSS/% NETEM_OWD ms, plus
+# optional NETEM_JITTER ms and NETEM_CORR % for bursty (5G-like) loss/jitter.
+if [[ "$USE_NETNS" == "1" ]]; then
+  NETEM_LOSS="${NETEM_LOSS:-0}"
+  NETEM_OWD="${NETEM_OWD:-0}"
+  if [[ "$NETEM_LOSS" != "0" || "$NETEM_OWD" != "0" ]]; then
+    "$LK_DIR/scripts/flexfec/netns.sh" shape "$NETEM_LOSS" "$NETEM_OWD" "${NETEM_JITTER:-0}" "${NETEM_CORR:-0}"
+  else
+    "$LK_DIR/scripts/flexfec/netns.sh" clear
+  fi
+fi
+
+# --- SFU (root netns) ---
+# In-SFU software impairment (used when netem/root is unavailable). All no-ops unless set.
+#   PUB_LOSS / PUB_DELAY_MS        robot uplink (publisher->SFU) loss + one-way delay
+#   DOWNLINK_LOSS / DOWNLINK_DELAY_MS / UPLINK_DELAY_MS   operator downlink + NACK delay
+( cd "$LK_DIR" && \
+  LK_PUB_LOSS="${PUB_LOSS:-0}" LK_PUB_DELAY_MS="${PUB_DELAY_MS:-0}" \
+  LK_DOWNLINK_LOSS="${DOWNLINK_LOSS:-0}" LK_DOWNLINK_DELAY_MS="${DOWNLINK_DELAY_MS:-0}" \
+  LK_UPLINK_DELAY_MS="${UPLINK_DELAY_MS:-0}" \
+  "$SFU_BIN" --dev --config "$CONFIG" > "$SFU_LOG" 2>&1 ) &
+sleep 3
+
+export LIVEKIT_URL="ws://$SFU_HOST:7880"
+export LIVEKIT_API_KEY=devkey
+export LIVEKIT_API_SECRET=secret
+export RUST_LOG="${RUST_LOG:-info}"
+ROOM=flexfec-test
+
+PUB="$RUST_DIR/target/debug/publisher"
+SUB="$RUST_DIR/target/debug/subscriber"
+
+# --- Publisher (robot) ---
+PUB_ARGS=(--flex-fec --room-name "$ROOM" --identity flexfec-pub --test-pattern --animate-test-pattern --attach-timestamp --attach-frame-id)
+# shellcheck disable=SC2206
+[[ -n "$PUB_EXTRA" ]] && PUB_ARGS+=($PUB_EXTRA)
+
+if [[ "$USE_NETNS" == "1" ]]; then
+  # netns.sh sudoes ip/tc internally; do NOT wrap it in sudo (the script isn't allowlisted).
+  "$LK_DIR/scripts/flexfec/netns.sh" exec env \
+    LIVEKIT_URL="$LIVEKIT_URL" LIVEKIT_API_KEY=devkey LIVEKIT_API_SECRET=secret RUST_LOG="$RUST_LOG" \
+    "$PUB" "${PUB_ARGS[@]}" > "$PUB_LOG" 2>&1 &
+else
+  "$PUB" "${PUB_ARGS[@]}" > "$PUB_LOG" 2>&1 &
+fi
+sleep 4
+
+# --- Subscriber (operator, root netns) ---
+# The subscriber opens an eframe/wgpu window (frame-latency stats are produced by its render
+# loop), so it needs a display. The box has a real X server on :1; use it unless DISPLAY is
+# already set. xvfb-run is used as a fallback if available.
+SUB_DISPLAY="${DISPLAY:-:1}"
+SUB_XAUTH="${XAUTHORITY:-/run/user/$(id -u)/gdm/Xauthority}"
+DISPLAY="$SUB_DISPLAY" XAUTHORITY="$SUB_XAUTH" "$SUB" --room-name "$ROOM" --identity flexfec-sub > "$SUB_LOG" 2>&1 &
+
+sleep "$DURATION"
+cleanup
+sleep 1
+
+echo "-- SFU FlexFEC recovery (publisher uplink), last 3 --"
+grep -iE 'flexfec recovery stats' "$SFU_LOG" 2>/dev/null | tail -3 | sed -E 's/.*"fecPacketsReceived"/fecRecv/; s/, "fecReportedReceived.*//'
+echo "-- SFU flexfec decoder enabled? --"
+grep -iE 'flexfec decoder enabled' "$SFU_LOG" 2>/dev/null | tail -1
+echo "-- frame latency, last 6 windows --"
+grep -iE 'frame latency' "$SUB_LOG" 2>/dev/null | tail -6
+echo "-- video quality (last) --"
+grep -iE 'Video quality' "$SUB_LOG" 2>/dev/null | tail -1
+echo "-- jitter buffer (last) --"
+grep -iE 'jitter buffer' "$SUB_LOG" 2>/dev/null | tail -1
+echo "============================================================================"

scripts/flexfec/run-publisher.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+#
+# Run the rust-sdks local_video publisher with FlexFEC publishing enabled.
+#
+# Connects to the local dev SFU (run-sfu.sh) and publishes camera video with a
+# FlexFEC-03 repair stream (--flex-fec). Output is teed to a log file.
+#
+# Usage:
+#   scripts/flexfec/run-publisher.sh [extra publisher args...]
+#
+# Env overrides:
+#   RUST_SDKS_DIR   path to the rust-sdks5 checkout (default: ../rust-sdks5)
+#   LIVEKIT_URL     default ws://127.0.0.1:7880
+#   LIVEKIT_API_KEY / LIVEKIT_API_SECRET   default devkey / secret
+#   ROOM            room name (default flexfec-test)
+#   PUB_IDENTITY    participant identity (default flexfec-pub)
+#   RUST_LOG        log filter (default info)
+#   LOG_DIR         where to write logs (default scripts/flexfec/logs)
+#   TEST_PATTERN    1 = SMPTE test pattern (no camera); 0 = use camera (default 1)
+#   ATTACH_META     1 = attach per-frame timestamp + frame id so the subscriber can log
+#                   end-to-end frame latency; 0 = off (default 1)
+#   MIN_PLAYOUT_DELAY / MAX_PLAYOUT_DELAY   ms; when set, recreates the room with a
+#                   subscriber playout-delay cap (passed through to the publisher)
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+RUST_SDKS_DIR="${RUST_SDKS_DIR:-$REPO_ROOT/../rust-sdks5}"
+LOG_DIR="${LOG_DIR:-$SCRIPT_DIR/logs}"
+
+if [[ ! -d "$RUST_SDKS_DIR" ]]; then
+  echo "error: rust-sdks checkout not found at '$RUST_SDKS_DIR' (set RUST_SDKS_DIR)" >&2
+  exit 1
+fi
+
+export LIVEKIT_URL="${LIVEKIT_URL:-ws://127.0.0.1:7880}"
+export LIVEKIT_API_KEY="${LIVEKIT_API_KEY:-devkey}"
+export LIVEKIT_API_SECRET="${LIVEKIT_API_SECRET:-secret}"
+export RUST_LOG="${RUST_LOG:-info}"
+ROOM="${ROOM:-flexfec-test}"
+PUB_IDENTITY="${PUB_IDENTITY:-flexfec-pub}"
+
+PUB_ARGS=(--flex-fec --room-name "$ROOM" --identity "$PUB_IDENTITY")
+if [[ "${TEST_PATTERN:-1}" == "1" ]]; then
+  PUB_ARGS+=(--test-pattern)
+fi
+if [[ "${ATTACH_META:-1}" == "1" ]]; then
+  PUB_ARGS+=(--attach-timestamp --attach-frame-id)
+fi
+if [[ -n "${MIN_PLAYOUT_DELAY:-}" ]]; then
+  PUB_ARGS+=(--min-playout-delay "$MIN_PLAYOUT_DELAY")
+fi
+if [[ -n "${MAX_PLAYOUT_DELAY:-}" ]]; then
+  PUB_ARGS+=(--max-playout-delay "$MAX_PLAYOUT_DELAY")
+fi
+
+mkdir -p "$LOG_DIR"
+echo "==> Publisher -> $LIVEKIT_URL room=$ROOM identity=$PUB_IDENTITY (FlexFEC on)"
+echo "    args: ${PUB_ARGS[*]}"
+echo "    log:  $LOG_DIR/publisher.log"
+
+cd "$RUST_SDKS_DIR"
+set -o pipefail
+cargo run -p local_video --features desktop --bin publisher -- \
+  "${PUB_ARGS[@]}" \
+  "$@" 2>&1 | tee "$LOG_DIR/publisher.log"

scripts/flexfec/run-sfu.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+#
+# Build and run the LiveKit SFU locally with FlexFEC-03 enabled.
+#
+# Uses `--dev` (devkey/secret API keys, loopback bind) together with the local
+# FlexFEC config. Media is pinned to UDP 7882 so netem.sh can shape it.
+#
+# Usage:
+#   scripts/flexfec/run-sfu.sh [extra go-run/server args...]
+#
+# Env overrides:
+#   CONFIG          path to the SFU config (default: scripts/flexfec/flexfec-local.yaml)
+#   Operator/subscriber side (SFU<->operator):
+#     DOWNLINK_LOSS      fractional SFU->subscriber loss, e.g. 0.01 for 1% (unset = none)
+#     DOWNLINK_DELAY_MS  one-way SFU->subscriber delay in ms (unset = none)
+#     UPLINK_DELAY_MS    one-way subscriber->SFU feedback (NACK/RTCP) delay in ms
+#   Robot/publisher side (robot<->SFU), e.g. a lossy 5G first hop:
+#     PUB_LOSS           fractional publisher->SFU loss, e.g. 0.03 for 3% (unset = none)
+#     PUB_DELAY_MS       one-way robot<->SFU delay in ms (inbound media + outbound NACK)
+#
+# These inject impairment in-SFU (pkg/sfu/downtrack_impair.go for the subscriber egress,
+# pkg/sfu/buffer/buffer_impair.go for the publisher ingress), replacing netem.sh for the
+# all-on-one-box setup where macOS pf/dummynet can't shape same-host UDP. Delays apply to
+# retransmissions too; set the two delays equal for a symmetric link so a NACK->RTX
+# recovery costs a full RTT -- the latency FlexFEC avoids by repairing inline.
+#
+# Teleop topology example (lossy 5G robot uplink, clean wired operator):
+#   PUB_LOSS=0.03 PUB_DELAY_MS=40 scripts/flexfec/run-sfu.sh   # + flexfec.publisher: true
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+CONFIG="${CONFIG:-$SCRIPT_DIR/flexfec-local.yaml}"
+
+cd "$REPO_ROOT"
+
+if [[ -n "${DOWNLINK_LOSS:-}" ]]; then
+  export LK_DOWNLINK_LOSS="$DOWNLINK_LOSS"
+fi
+if [[ -n "${DOWNLINK_DELAY_MS:-}" ]]; then
+  export LK_DOWNLINK_DELAY_MS="$DOWNLINK_DELAY_MS"
+fi
+if [[ -n "${UPLINK_DELAY_MS:-}" ]]; then
+  export LK_UPLINK_DELAY_MS="$UPLINK_DELAY_MS"
+fi
+if [[ -n "${PUB_LOSS:-}" ]]; then
+  export LK_PUB_LOSS="$PUB_LOSS"
+fi
+if [[ -n "${PUB_DELAY_MS:-}" ]]; then
+  export LK_PUB_DELAY_MS="$PUB_DELAY_MS"
+fi
+
+echo "==> Building & running LiveKit SFU with FlexFEC"
+echo "    repo:   $REPO_ROOT"
+echo "    config: $CONFIG"
+echo "    url:    ws://127.0.0.1:7880   (devkey / secret)"
+echo "    media:  udp/7882 on loopback"
+echo "    operator downlink: loss=${LK_DOWNLINK_LOSS:-none} delay_ms=${LK_DOWNLINK_DELAY_MS:-none} nack_delay_ms=${LK_UPLINK_DELAY_MS:-none}"
+echo "    robot uplink:      loss=${LK_PUB_LOSS:-none} delay_ms=${LK_PUB_DELAY_MS:-none}"
+echo
+
+exec go run ./cmd/server --dev --config "$CONFIG" "$@"

scripts/flexfec/run-subscriber.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+#
+# Run the rust-sdks local_video subscriber against the local dev SFU and surface
+# FlexFEC stats. The subscriber logs "Video FEC ..." lines as repair packets arrive
+# and recovery payload is accepted.
+#
+# Usage:
+#   scripts/flexfec/run-subscriber.sh [extra subscriber args...]
+#
+# Env overrides:
+#   RUST_SDKS_DIR   path to the rust-sdks5 checkout (default: ../rust-sdks5)
+#   LIVEKIT_URL     default ws://127.0.0.1:7880
+#   LIVEKIT_API_KEY / LIVEKIT_API_SECRET   default devkey / secret
+#   ROOM            room name (default flexfec-test)
+#   SUB_IDENTITY    participant identity (default flexfec-sub)
+#   RUST_LOG        log filter (default info)
+#   LOG_DIR         where to write logs (default scripts/flexfec/logs)
+#   FEC_ONLY        if set to 1, only print FEC + frame-latency lines to the console
+#
+# Frame latency: when the publisher runs with --attach-timestamp/--attach-frame-id, the
+# subscriber emits "Subscriber frame latency: ..." lines (capture->receive->decode). The
+# receive_to_decode value is the jitter-buffer wait, which is where FEC-ON and RTX-only
+# diverge; capture_to_decode is the end-to-end glass-to-decode latency.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+RUST_SDKS_DIR="${RUST_SDKS_DIR:-$REPO_ROOT/../rust-sdks5}"
+LOG_DIR="${LOG_DIR:-$SCRIPT_DIR/logs}"
+
+if [[ ! -d "$RUST_SDKS_DIR" ]]; then
+  echo "error: rust-sdks checkout not found at '$RUST_SDKS_DIR' (set RUST_SDKS_DIR)" >&2
+  exit 1
+fi
+
+export LIVEKIT_URL="${LIVEKIT_URL:-ws://127.0.0.1:7880}"
+export LIVEKIT_API_KEY="${LIVEKIT_API_KEY:-devkey}"
+export LIVEKIT_API_SECRET="${LIVEKIT_API_SECRET:-secret}"
+export RUST_LOG="${RUST_LOG:-info}"
+ROOM="${ROOM:-flexfec-test}"
+SUB_IDENTITY="${SUB_IDENTITY:-flexfec-sub}"
+
+mkdir -p "$LOG_DIR"
+echo "==> Subscriber -> $LIVEKIT_URL room=$ROOM identity=$SUB_IDENTITY"
+echo "    log: $LOG_DIR/subscriber.log   (grep 'Video FEC' for FEC, 'frame latency' for latency)"
+
+cd "$RUST_SDKS_DIR"
+set -o pipefail
+if [[ "${FEC_ONLY:-0}" == "1" ]]; then
+  cargo run -p local_video --features desktop --bin subscriber -- \
+    --room-name "$ROOM" \
+    --identity "$SUB_IDENTITY" \
+    "$@" 2>&1 | tee "$LOG_DIR/subscriber.log" | grep --line-buffered -iE 'fec|recover|frame latency' || true
+else
+  cargo run -p local_video --features desktop --bin subscriber -- \
+    --room-name "$ROOM" \
+    --identity "$SUB_IDENTITY" \
+    "$@" 2>&1 | tee "$LOG_DIR/subscriber.log"
+fi