simpler mock protocol (#4641)

enabling us to build a comprehensive set of API tests for our clients
This commit is contained in:
David Zhao
2026-07-03 13:17:59 +02:00
committed by GitHub
parent 8f6a9cb8b7
commit 00348c1299
5 changed files with 256 additions and 95 deletions
+45 -29
View File
@@ -7,23 +7,27 @@ image and booted by each SDK's CI.
## Why it looks the way it does
- **Stateless.** All behavior is selected by per-request `X-Lk-Mock-*` headers,
so the server holds no mutable state and tests run in parallel.
- **Stateless.** All behavior is selected by a single per-request `X-Lk-Mock`
header (a JSON object), so the server holds no mutable state and tests run in
parallel.
- **Multi-port = multi-region.** The process binds one listener per simulated
region (`--ports`). A port's position in the list is its **region index**;
index `0` is the primary the SDK is initially pointed at. `GET
/settings/regions` advertises all of them in order.
- **One header drives every attempt.** The SDK sends the same control header on
the initial request *and* every failover retry. Each listener decides what to
do from its **own** index, so a single `X-Lk-Mock-Fail-Regions: 0` makes the
primary fail while the first fallback succeeds — no coordination needed.
do from its **own** index, so a single `X-Lk-Mock: {"failRegions":[0]}` makes
the primary fail while the first fallback succeeds — no coordination needed.
- **Realistic latency.** Methods that block in the real server block here too:
`CreateSIPParticipant` with `wait_until_answered` and `TransferSIPParticipant`
take ~11s before responding, so SDKs can exercise their timeouts.
- **The whole API is mocked with populated responses.** Every RoomService,
Egress, Ingress, SIP, and Connector method returns a type-correct, populated
response: scalar fields that share a name with the request are echoed (e.g.
`name`, `metadata`, `identity`, timeouts), `id`/`sid` fields get placeholder
values, and list endpoints return one element. Both protobuf and JSON Twirp
clients are supported. A client can override the response entirely with the
`X-Lk-Mock-Response` header (see below). Unregistered/future methods fall back
`response` field (see below). Unregistered/future methods fall back
to an empty (all-default) message, which still decodes cleanly.
## Running
@@ -46,20 +50,31 @@ docker run -p 9999-10002:9999-10002 livekit/test-server
## Control protocol
Request headers (sent by the SDK on API calls; the SDK must forward
client-configured custom headers onto the `/settings/regions` fetch and every
failover retry):
All behavior is driven by a single `X-Lk-Mock` request header whose value is a
JSON object. The SDK sends the same header on API calls, on the
`/settings/regions` fetch, and on every failover retry (it must forward
client-configured custom headers onto all of them). Omit the header — or any
field — for normal behavior. Every field is optional:
| Header | Default | Effect |
| Field | Default | Effect |
|---|---|---|
| `X-Lk-Mock-Fail-Regions` | — | comma list of region indices that fail this request, e.g. `0` or `0,1`. Each listener fails only if its own index is listed. |
| `X-Lk-Mock-Fail-Mode` | `status` | how a failing region fails: `status`, `drop` (close connection → transport error), `delay`. |
| `X-Lk-Mock-Fail-Status` | `503` | HTTP status when failing with `status`/`delay`. |
| `X-Lk-Mock-Fail-Twirp-Code` | derived from status | Twirp error code string in the failure body. |
| `X-Lk-Mock-Delay-Ms` | `30000` | delay before a `delay`-mode region responds (for timeout tests). |
| `X-Lk-Mock-Regions-Status` | `200` | override the status of `GET /settings/regions`. |
| `X-Lk-Mock-Response` | — | protojson of the response message for the called method; replaces the populated default, giving full control over the returned payload. |
| `X-Lk-Mock-Skip-Auth` | | `true` disables permission enforcement for the request (use for tests that aren't about authz, e.g. failover tests with a placeholder token). |
| `failRegions` | — | array of region indices that fail this request, e.g. `[0]` or `[0,1]`. Each listener fails only if its own index is listed. |
| `failMode` | `status` | how a failing region fails: `status` (write a Twirp error), or `drop` (close the connection → transport error). |
| `failStatus` | `503` | HTTP status for a `status`-mode failure. |
| `failTwirpCode` | derived from status | Twirp error code string in the failure body. |
| `delayMs` | | delay (ms) before responding, on success or failure. Overrides a method's natural latency — use it for timeout tests, or set it to skip a SIP method's built-in ~11s wait. |
| `regionsStatus` | `200` | override the status of `GET /settings/regions`. |
| `response` | — | the response message for the called method (a JSON object, protojson-shaped); replaces the populated default, giving full control over the returned payload. |
| `skipAuth` | `false` | `true` disables permission enforcement for the request (use for tests that aren't about authz, e.g. failover tests with a placeholder token). |
Example: `X-Lk-Mock: {"skipAuth":true,"failRegions":[0],"failStatus":400}`
> **Deprecated:** the older per-setting headers — `X-Lk-Mock-Fail-Regions`,
> `X-Lk-Mock-Fail-Mode` (incl. the `delay` mode), `X-Lk-Mock-Fail-Status`,
> `X-Lk-Mock-Fail-Twirp-Code`, `X-Lk-Mock-Delay-Ms`, `X-Lk-Mock-Regions-Status`,
> `X-Lk-Mock-Response`, `X-Lk-Mock-Skip-Auth` — are still honored for existing
> clients and will be removed later. When `X-Lk-Mock` is also present, its fields
> take precedence per-field. New clients should use `X-Lk-Mock` only.
Response headers:
@@ -96,23 +111,24 @@ is configured with (`secret` by default).
| `sip.admin` | SIP trunk & dispatch-rule CRUD |
| `sip.call` | `CreateSIPParticipant`; `TransferSIPParticipant` (also needs `roomAdmin`) |
Send `X-Lk-Mock-Skip-Auth: true` to bypass enforcement for tests that aren't
Send `X-Lk-Mock: {"skipAuth":true}` to bypass enforcement for tests that aren't
about permissions.
## Common recipes
| Goal | Headers |
| Goal | `X-Lk-Mock` value |
|---|---|
| Happy path | valid token with the method's grant → 200 from region `0` |
| Bypass auth (failover tests) | `X-Lk-Mock-Skip-Auth: true` |
| Missing-permission error | token without the required grant → 403 |
| Failover succeeds on region 1 | `X-Lk-Mock-Fail-Regions: 0` |
| Exhaust to region 2 | `X-Lk-Mock-Fail-Regions: 0,1` |
| All regions down | `X-Lk-Mock-Fail-Regions: 0,1,2,3` |
| 4xx, no retry | `X-Lk-Mock-Fail-Regions: 0` + `X-Lk-Mock-Fail-Status: 400` |
| Transport-error failover | `X-Lk-Mock-Fail-Regions: 0` + `X-Lk-Mock-Fail-Mode: drop` |
| Region discovery unreachable | `X-Lk-Mock-Regions-Status: 500` |
| Custom response payload | `X-Lk-Mock-Response: {"sid":"RM_x","name":"my-room"}` |
| Happy path | (no header) — valid token with the method's grant → 200 from region `0` |
| Bypass auth (failover tests) | `{"skipAuth":true}` |
| Missing-permission error | (no header) — token without the required grant → 403 |
| Failover succeeds on region 1 | `{"failRegions":[0]}` |
| Exhaust to region 2 | `{"failRegions":[0,1]}` |
| All regions down | `{"failRegions":[0,1,2,3]}` |
| 4xx, no retry | `{"failRegions":[0],"failStatus":400}` |
| Transport-error failover | `{"failRegions":[0],"failMode":"drop"}` |
| Timeout test | `{"delayMs":30000}` |
| Region discovery unreachable | `{"regionsStatus":500}` |
| Custom response payload | `{"response":{"sid":"RM_x","name":"my-room"}}` |
Note: SDK region failover normally only engages for `*.livekit.cloud` hosts.
Since tests point at `127.0.0.1`, set the SDK's failover-enable option to its
+16 -4
View File
@@ -31,8 +31,8 @@ import (
// uses), against the mock's configured API secret (default "secret", matching
// `livekit-server --dev`); set --api-secret / LK_TEST_SERVER_API_SECRET to change it.
//
// Set X-Lk-Mock-Skip-Auth: true to bypass enforcement for tests that aren't
// about permissions (e.g. region-failover tests using a placeholder token).
// Set `"skipAuth": true` in the X-Lk-Mock header to bypass enforcement for tests
// that aren't about permissions (e.g. region-failover tests with a placeholder token).
// perm describes the grants a method requires. roomAdmin additionally requires
// the token's room to match the request's room; destRoom further requires the
@@ -119,8 +119,8 @@ var methodPerms = map[string]perm{
// authorize enforces the permissions a method requires. It returns the HTTP
// status and Twirp error code to send (0, "" means authorized / not enforced).
func (h *mockHandler) authorize(key string, r *http.Request, req proto.Message) (int, string) {
if strings.EqualFold(r.Header.Get(headerSkipAuth), "true") {
func (h *mockHandler) authorize(key string, r *http.Request, cfg *mockConfig, req proto.Message) (int, string) {
if cfg.SkipAuth {
return 0, ""
}
p, known := methodPerms[key]
@@ -210,3 +210,15 @@ func requestString(req proto.Message, field string) string {
}
return m.Get(fd).String()
}
func requestBool(req proto.Message, field string) bool {
if req == nil {
return false
}
m := req.ProtoReflect()
fd := m.Descriptor().Fields().ByName(protoreflect.Name(field))
if fd == nil || fd.Kind() != protoreflect.BoolKind || fd.IsList() {
return false
}
return m.Get(fd).Bool()
}
+132
View File
@@ -0,0 +1,132 @@
// Copyright 2026 LiveKit, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"encoding/json"
"net/http"
"strconv"
"strings"
)
const (
// headerMock carries the whole mock control config as a JSON object; see
// mockConfig and the README. Sent by the SDK on API calls (and forwarded onto
// the /settings/regions fetch and every failover retry).
headerMock = "X-Lk-Mock"
// headerRegion is set on responses to the index of the region that served it.
headerRegion = "X-Lk-Mock-Region"
)
// Deprecated: the individual X-Lk-Mock-* control headers predate the unified
// X-Lk-Mock JSON header. They are still honored for existing clients; new
// clients should send X-Lk-Mock instead. When X-Lk-Mock is present its fields
// take precedence over any legacy header.
const (
legacyHeaderFailRegions = "X-Lk-Mock-Fail-Regions"
legacyHeaderFailMode = "X-Lk-Mock-Fail-Mode"
legacyHeaderFailStatus = "X-Lk-Mock-Fail-Status"
legacyHeaderFailTwirpCode = "X-Lk-Mock-Fail-Twirp-Code"
legacyHeaderDelayMs = "X-Lk-Mock-Delay-Ms"
legacyHeaderRegionsStatus = "X-Lk-Mock-Regions-Status"
legacyHeaderResponse = "X-Lk-Mock-Response"
legacyHeaderSkipAuth = "X-Lk-Mock-Skip-Auth"
)
// legacyDefaultDelayMs is the sleep used by the deprecated "delay" fail mode when
// no X-Lk-Mock-Delay-Ms is given (long enough to trip client timeouts).
const legacyDefaultDelayMs = 30_000
// mockConfig is the JSON value of the X-Lk-Mock request header. Every field is
// optional; the zero value means "behave normally". A single object keeps the
// control protocol simple — the SDK serializes one struct instead of juggling a
// header per knob.
type mockConfig struct {
// FailRegions lists region indices that should fail this request. A listener
// fails only if its own region index appears here, so one config can make the
// primary fail while a fallback succeeds.
FailRegions []int `json:"failRegions,omitempty"`
// FailMode selects how a failing region fails: "status" (default) writes a
// Twirp error; "drop" closes the connection to force a transport error.
// ("delay" is a deprecated legacy mode; new clients use DelayMs instead.)
FailMode string `json:"failMode,omitempty"`
// FailStatus is the HTTP status for a "status"-mode failure (default 503).
FailStatus int `json:"failStatus,omitempty"`
// FailTwirpCode overrides the Twirp error code string in the failure body
// (default derived from FailStatus).
FailTwirpCode string `json:"failTwirpCode,omitempty"`
// DelayMs delays the response by this many milliseconds before returning,
// whether the region succeeds or fails. It overrides a method's natural
// latency (see methodLatency): set it high for timeout tests, or to 0 to skip
// a SIP method's built-in wait. Nil means "use the natural latency".
DelayMs *int `json:"delayMs,omitempty"`
// RegionsStatus overrides the HTTP status of GET /settings/regions (default 200).
RegionsStatus int `json:"regionsStatus,omitempty"`
// Response is the protojson of the response message for the called method; it
// replaces the populated default, giving full control over the payload.
Response json.RawMessage `json:"response,omitempty"`
// SkipAuth disables permission enforcement for this request (for tests that
// aren't about authz, e.g. failover tests with a placeholder token).
SkipAuth bool `json:"skipAuth,omitempty"`
// legacyDelayMs is the sleep used by the deprecated "delay" fail mode. It is
// populated only from the legacy X-Lk-Mock-Delay-Ms header, never from JSON.
legacyDelayMs int
}
// parseMockConfig builds the request's config. Deprecated individual X-Lk-Mock-*
// headers form the base; the unified X-Lk-Mock JSON header (if present) is
// overlaid on top, so its fields win per-field while absent fields keep the
// legacy value.
func parseMockConfig(r *http.Request) mockConfig {
cfg := parseLegacyConfig(r)
if v := r.Header.Get(headerMock); v != "" {
// Unmarshal overwrites only the fields present in the JSON; the unexported
// legacyDelayMs is untouched.
_ = json.Unmarshal([]byte(v), &cfg)
}
return cfg
}
// parseLegacyConfig reads the deprecated per-setting headers into a config.
func parseLegacyConfig(r *http.Request) mockConfig {
var cfg mockConfig
for _, part := range strings.Split(r.Header.Get(legacyHeaderFailRegions), ",") {
if idx, err := strconv.Atoi(strings.TrimSpace(part)); err == nil {
cfg.FailRegions = append(cfg.FailRegions, idx)
}
}
cfg.FailMode = r.Header.Get(legacyHeaderFailMode)
cfg.FailStatus = parseStatus(r.Header.Get(legacyHeaderFailStatus))
cfg.FailTwirpCode = r.Header.Get(legacyHeaderFailTwirpCode)
cfg.RegionsStatus = parseStatus(r.Header.Get(legacyHeaderRegionsStatus))
if resp := r.Header.Get(legacyHeaderResponse); resp != "" {
cfg.Response = json.RawMessage(resp)
}
cfg.SkipAuth = strings.EqualFold(r.Header.Get(legacyHeaderSkipAuth), "true")
cfg.legacyDelayMs = legacyDefaultDelayMs
if ms, err := strconv.Atoi(r.Header.Get(legacyHeaderDelayMs)); err == nil && ms >= 0 {
cfg.legacyDelayMs = ms
}
return cfg
}
// parseStatus returns a valid HTTP status from s, or 0 if absent/invalid.
func parseStatus(s string) int {
if v, err := strconv.Atoi(strings.TrimSpace(s)); err == nil && v >= 100 && v <= 599 {
return v
}
return 0
}
+44 -10
View File
@@ -19,6 +19,7 @@ import (
"net/http"
"strconv"
"strings"
"time"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/reflect/protoreflect"
@@ -135,25 +136,58 @@ func (h *mockHandler) serveAPI(w http.ResponseWriter, r *http.Request) {
}
}
cfg := parseMockConfig(r)
// Permission enforcement comes first, mirroring the real server.
if status, code := h.authorize(key, r, req); status != 0 {
if status, code := h.authorize(key, r, &cfg, req); status != 0 {
writeTwirpErrorCode(w, status, code, "mock: "+code)
return
}
if h.shouldFail(r) {
h.fail(w, r)
// Delay before responding (success or failure). An explicit delayMs overrides
// the method's natural latency — e.g. CreateSIPParticipant blocking until the
// callee answers.
delay := methodLatency(key, req)
if cfg.DelayMs != nil {
delay = time.Duration(*cfg.DelayMs) * time.Millisecond
}
if delay > 0 {
time.Sleep(delay)
}
if h.shouldFail(&cfg) {
h.fail(w, &cfg)
return
}
h.writeAPIResponse(w, r, json, known, req, spec)
h.writeAPIResponse(w, json, known, req, spec, &cfg)
}
// methodLatency returns the realistic time a method blocks before responding, so
// the mock approximates the real server's behavior. CreateSIPParticipant blocks
// until the callee answers when wait_until_answered is set; TransferSIPParticipant
// always blocks until the transfer (REFER) completes.
func methodLatency(key string, req proto.Message) time.Duration {
switch key {
case "livekit.SIP/CreateSIPParticipant":
if requestBool(req, "wait_until_answered") {
return sipAnswerLatency
}
case "livekit.SIP/TransferSIPParticipant":
return sipAnswerLatency
}
return 0
}
// sipAnswerLatency is how long a SIP call takes to be answered/transferred in the
// mock — long enough to exercise client-side timeouts around these calls.
const sipAnswerLatency = 11 * time.Second
// writeAPIResponse serves a populated, type-correct response for a known API
// method. The response is the reflection-populated default unless the request
// carries an X-Lk-Mock-Response header (protojson), which overrides it
// entirely. Content type (protobuf vs JSON) mirrors the request.
func (h *mockHandler) writeAPIResponse(w http.ResponseWriter, r *http.Request, json, known bool, req proto.Message, spec apiSpec) {
// method. The response is the reflection-populated default unless the mock
// config carries a `response` (protojson), which overrides it entirely. Content
// type (protobuf vs JSON) mirrors the request.
func (h *mockHandler) writeAPIResponse(w http.ResponseWriter, json, known bool, req proto.Message, spec apiSpec, cfg *mockConfig) {
w.Header().Set(headerRegion, strconv.Itoa(h.regionIndex))
if !known {
@@ -164,8 +198,8 @@ func (h *mockHandler) writeAPIResponse(w http.ResponseWriter, r *http.Request, j
}
resp := spec.newResp()
if override := r.Header.Get(headerResponse); override != "" {
if err := protojson.Unmarshal([]byte(override), resp); err != nil {
if len(cfg.Response) > 0 {
if err := protojson.Unmarshal(cfg.Response, resp); err != nil {
// Malformed override: fall back to the populated default.
resp = spec.newResp()
populateMessage(resp.ProtoReflect(), req.ProtoReflect(), 1)
+19 -52
View File
@@ -22,6 +22,7 @@ import (
"net/http"
"os"
"os/signal"
"slices"
"strconv"
"strings"
"syscall"
@@ -31,23 +32,6 @@ import (
"github.com/livekit/protocol/utils/protojson"
)
// X-Lk-Mock-* request headers control the mock's behavior; see the README.
const (
headerFailRegions = "X-Lk-Mock-Fail-Regions"
headerFailMode = "X-Lk-Mock-Fail-Mode"
headerFailStatus = "X-Lk-Mock-Fail-Status"
headerFailTwirpCode = "X-Lk-Mock-Fail-Twirp-Code"
headerDelayMs = "X-Lk-Mock-Delay-Ms"
headerRegionsStatus = "X-Lk-Mock-Regions-Status"
headerResponse = "X-Lk-Mock-Response"
// headerSkipAuth disables permission enforcement for a request.
headerSkipAuth = "X-Lk-Mock-Skip-Auth"
// headerRegion is set on responses to the index of the region that served it.
headerRegion = "X-Lk-Mock-Region"
)
const defaultDelayMs = 30_000
func main() {
portsFlag := flagValue("--ports", "LK_TEST_SERVER_PORTS", "9999,10000,10001,10002")
advertiseHost := flagValue("--advertise-host", "LK_TEST_SERVER_ADVERTISE_HOST", "http://127.0.0.1")
@@ -116,8 +100,9 @@ func (h *mockHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
func (h *mockHandler) handleRegions(w http.ResponseWriter, r *http.Request) {
if status := parseStatus(r.Header.Get(headerRegionsStatus), 0); status != 0 && status != http.StatusOK {
w.WriteHeader(status)
cfg := parseMockConfig(r)
if cfg.RegionsStatus != 0 && cfg.RegionsStatus != http.StatusOK {
w.WriteHeader(cfg.RegionsStatus)
return
}
body, err := protojson.Marshal(h.regions)
@@ -135,21 +120,12 @@ func (h *mockHandler) handleTwirp(w http.ResponseWriter, r *http.Request) {
h.serveAPI(w, r)
}
func (h *mockHandler) shouldFail(r *http.Request) bool {
for _, part := range strings.Split(r.Header.Get(headerFailRegions), ",") {
part = strings.TrimSpace(part)
if part == "" {
continue
}
if idx, err := strconv.Atoi(part); err == nil && idx == h.regionIndex {
return true
}
}
return false
func (h *mockHandler) shouldFail(cfg *mockConfig) bool {
return slices.Contains(cfg.FailRegions, h.regionIndex)
}
func (h *mockHandler) fail(w http.ResponseWriter, r *http.Request) {
switch strings.ToLower(r.Header.Get(headerFailMode)) {
func (h *mockHandler) fail(w http.ResponseWriter, cfg *mockConfig) {
switch strings.ToLower(cfg.FailMode) {
case "drop":
if hj, ok := w.(http.Hijacker); ok {
if conn, _, err := hj.Hijack(); err == nil {
@@ -158,20 +134,21 @@ func (h *mockHandler) fail(w http.ResponseWriter, r *http.Request) {
}
}
w.WriteHeader(http.StatusServiceUnavailable)
return
case "delay":
delay := defaultDelayMs
if ms, err := strconv.Atoi(r.Header.Get(headerDelayMs)); err == nil && ms >= 0 {
delay = ms
}
time.Sleep(time.Duration(delay) * time.Millisecond)
writeTwirpError(w, r, parseStatus(r.Header.Get(headerFailStatus), http.StatusServiceUnavailable))
default:
writeTwirpError(w, r, parseStatus(r.Header.Get(headerFailStatus), http.StatusServiceUnavailable))
// Deprecated legacy mode: sleep, then status-fail. New clients should set
// DelayMs (which delays every response) instead.
time.Sleep(time.Duration(cfg.legacyDelayMs) * time.Millisecond)
}
status := cfg.FailStatus
if status < 100 || status > 599 {
status = http.StatusServiceUnavailable
}
writeTwirpError(w, cfg, status)
}
func writeTwirpError(w http.ResponseWriter, r *http.Request, status int) {
code := r.Header.Get(headerFailTwirpCode)
func writeTwirpError(w http.ResponseWriter, cfg *mockConfig, status int) {
code := cfg.FailTwirpCode
if code == "" {
code = twirpCodeForStatus(status)
}
@@ -205,16 +182,6 @@ func twirpCodeForStatus(status int) string {
}
}
func parseStatus(s string, def int) int {
if s == "" {
return def
}
if v, err := strconv.Atoi(strings.TrimSpace(s)); err == nil && v >= 100 && v <= 599 {
return v
}
return def
}
func parsePorts(s string) ([]int, error) {
var ports []int
for _, part := range strings.Split(s, ",") {