From 7f8f71e7730b61b71ea72b3f8fbc9f6b02dfc7f4 Mon Sep 17 00:00:00 2001 From: efiten Date: Sun, 13 Sep 2026 19:59:18 +0200 Subject: [PATCH] fix(live): reconnect when the websocket goes silent (#1074) (#2020) ## Problem #1074 reports that after a proxy dropped the WebSocket, live updates only came back 8 to 10 minutes later. The client only reconnects from `onclose` (`public/app.js:791` on master). A half-open connection (a proxy or NAT dropping state without a FIN reaching the browser, a laptop that slept) can keep a WebSocket OPEN for minutes without `onclose`, and nothing retries in the meantime. The server does ping every 30s (`cmd/server/websocket.go:252` on master), but ping frames are answered by the browser below the page and JS cannot observe them. The only app-level frames are packet broadcasts (`websocket.go:337, 343, 366`), which stop on a quiet mesh. So the client had no signal to tell a quiet mesh from a dead socket. ## Change Server (`cmd/server/websocket.go`): - On the existing ping tick, `writePump` also writes the text frame `{"type":"heartbeat"}` (`:283`, bytes at `:80`). One 20-byte frame per client per 30s, from the goroutine that already writes the ping, no hub lock. - The interval moves to `Hub.pingInterval` (default 30s, `:118`) so a test can shorten it. Client (`public/app.js`): - Every frame refreshes `wsLastMessageAt` (`:841`). One timer (`checkWSLiveness`, `:806`) fires at last message + `WS_STALE_MS` (75s, one late or lost heartbeat of slack) and replaces the socket if it is still silent. It is armed at socket creation, so a stuck handshake is covered too. - `dropWS` (`:796`) detaches the old socket's handlers before `close()`, so a late close event cannot schedule a second connection. - `connectWS` (`:818`) cancels a pending reconnect and drops the previous socket, so the watchdog, resume checks, `onclose` and pull-to-reconnect cannot stack sockets. Before this, `pullReconnect` on a non-open socket left a third socket 3s later. The 3s `WS_RECONNECT_MS` delay after `onclose` is unchanged (`:837`). - `visibilitychange` (to visible) and `online` run the check immediately (`:861`), because a hidden or sleeping tab's timers can run late. - Heartbeat frames are matched by exact bytes (`:842`) and are not pulsed or dispatched to `onWS` listeners. Compatibility: tabs loaded before the deploy dispatch heartbeats to their listeners until reloaded. Every current listener filters on `msg.type`, so the visible effect is a logo pulse and a `/stats` cache refresh every 30s. Perf: one `Date.now()` and one string compare per WS message on the client; one extra 20-byte write per client per 30s on the server. ## Tests - `test-ws-stale-watchdog-1074.js`: real `app.js` in a vm with a fake clock, timers and WebSocket. 12 tests: silence past the threshold replaces the socket exactly once; a handshake that never opens is replaced; heartbeats and packet traffic keep the socket; heartbeats are not dispatched; resume and `online` after silence reconnect immediately, with recent traffic they do not, and hiding does not trigger a check; repeated resume events open one socket; after `onclose` only the reconnect timer is pending; pull-to-reconnect leaves one socket. 9 of 12 fail on master. 12 of 12 source mutations (threshold, reconnect path, detaching, timer cleanup, resume wiring, heartbeat filter) are caught. Registered in `test-all.sh` and the deploy.yml unit step. - `TestWritePumpSendsAppHeartbeat`: fails with a read timeout without the heartbeat, even with pings every 20ms. `TestHubDefaultPingInterval` pins the 30s interval that `WS_STALE_MS` assumes. - `go test ./...` in `cmd/server` passes; gofmt and go vet are clean. ## Browser validation On a staging instance (build `139e484e`, together with #1979's branch), in Chrome, no console errors: - A `{"type":"heartbeat"}` frame arrived on the open socket within the observation window. - Silent socket: after `ws.onmessage = null`, the page replaced the socket after 76.2s (threshold 75s plus a 250ms poll); the old socket ended in CLOSED, the new one OPEN. - Normal close: `ws.close()` led to a new OPEN socket after 4.1s, and exactly one new `WebSocket` was constructed. ## Not verified - The reporter's proxy setup was not reproduced; that their delay was a half-open socket is a hypothesis consistent with the symptom. Hence `Refs`, not `Fixes`. - Laptop sleep and the `visibilitychange` / `online` resume path were only covered by the unit test, not in a browser. - Behaviour under Chrome's intensive background-timer throttling and mobile tab freezing was not measured; a frozen but healthy tab may do one unnecessary reconnect on resume. - Go tests were run without `-race`. Refs #1074 ## Review follow-up (commit `72e5e906`) An independent review found no blocking bug: all data writes stay on the write goroutine, pong-based dead-client detection still works, and no ordering of onclose, watchdog, resume checks and pull ends with two live sockets or none. It reproduced the silent-socket case in headless Chromium through a blackholing TCP proxy (replacement 75.0 s after the last frame). Changed: 1. **Startup wiring tested.** The first resume test now boots through the page's real `DOMContentLoaded` listeners, so removing `setupWSResumeCheck()` from startup makes it fail. 2. **Wall clock stepping back.** If the clock steps back after the last message, the watchdog no longer re-arms for the size of the step (a 1 h step used to delay detection by about an hour). A negative silence reading is treated as stale, so the socket is replaced within `WS_STALE_MS` of the step (`public/app.js:810-814`). A step in either direction costs at most one extra reconnect on a healthy socket. `Date.now()` stays the clock so a tab resumed after sleep is still checked against real elapsed time. 3. **Pull-to-reconnect at once.** On an OPEN socket, pull-to-reconnect now replaces it through `connectWS()` instead of closing it and waiting for onclose, which took 63 s on a half-open connection in the review's measurement (`public/app.js:927-934`). This was slow on master too; it is safe now that `connectWS()` detaches the old socket. Tests: 12 to 16 in `test-ws-stale-watchdog-1074.js`; `test-pull-to-reconnect.js`, `test-pull-to-reconnect-1091.js` and `test-live.js` pass. Correction to the compatibility note: tabs opened before the deploy treat the heartbeat like any other message. Besides the logo pulse, `app.js` runs `updateNavStats` on every message and invalidates the cached `/stats` and `/nodes` responses 5 s later; `packets.js` also pushes every message into `pauseBuffer` unfiltered (~1310-1313), so an old tab with Packets paused sees its counter rise by 2 per minute. Cosmetic: heartbeats are filtered out on replay, and a reload ends it. Not verified: real hidden-tab or mobile freeze behaviour, Firefox and Safari, and the reporter's proxy setup. --------- Co-authored-by: Claude Opus 5 (1M context) --- cmd/server/websocket.go | 21 +- cmd/server/websocket_test.go | 41 ++++ public/app.js | 89 ++++++-- test-all.sh | 1 + test-pull-to-reconnect.js | 4 +- test-ws-stale-watchdog-1074.js | 386 +++++++++++++++++++++++++++++++++ 6 files changed, 520 insertions(+), 22 deletions(-) create mode 100644 test-ws-stale-watchdog-1074.js diff --git a/cmd/server/websocket.go b/cmd/server/websocket.go index 923c04c2..14befa64 100644 --- a/cmd/server/websocket.go +++ b/cmd/server/websocket.go @@ -19,6 +19,7 @@ type Hub struct { upgrader websocket.Upgrader allowedOrigins []string // exact-match allowlist for /ws CheckOrigin (see SetAllowedOrigins) limits *wsLimiter // #1794: per-IP caps and deny list; nil allows everything + pingInterval time.Duration } // SetAllowedOrigins configures the exact-match origin allowlist consulted by @@ -74,6 +75,10 @@ func (h *Hub) checkOrigin(r *http.Request) bool { return false } +// wsHeartbeat is written to every client on each ping tick. public/app.js +// compares incoming frames against these exact bytes, so keep them in step. +var wsHeartbeat = []byte(`{"type":"heartbeat"}`) + // Client is a single WebSocket connection. type Client struct { conn *websocket.Conn @@ -109,7 +114,8 @@ func (h *Hub) ConfigureLimits(maxConnsPerIP, upgradesPerMin int, trustedProxies, func NewHub() *Hub { h := &Hub{ - clients: make(map[*Client]bool), + clients: make(map[*Client]bool), + pingInterval: 30 * time.Second, } h.upgrader = websocket.Upgrader{ ReadBufferSize: 1024, @@ -214,7 +220,7 @@ func (h *Hub) ServeWS(w http.ResponseWriter, r *http.Request) { } h.Register(client) - go client.writePump() + go client.writePump(h.pingInterval) go client.readPump(h) } @@ -248,8 +254,8 @@ func (c *Client) readPump(hub *Hub) { } } -func (c *Client) writePump() { - ticker := time.NewTicker(30 * time.Second) +func (c *Client) writePump(pingInterval time.Duration) { + ticker := time.NewTicker(pingInterval) defer func() { ticker.Stop() c.conn.Close() @@ -270,6 +276,13 @@ func (c *Client) writePump() { if err := c.conn.WriteMessage(websocket.PingMessage, nil); err != nil { return } + // #1074: the ping above keeps proxies and this server's read + // deadline happy, but browser JS never sees ping frames. Without a + // frame the page receives, a quiet mesh and a dead socket look the + // same to the client, so it could not detect the latter. + if err := c.conn.WriteMessage(websocket.TextMessage, wsHeartbeat); err != nil { + return + } } } } diff --git a/cmd/server/websocket_test.go b/cmd/server/websocket_test.go index 5d5691b8..fbdd6210 100644 --- a/cmd/server/websocket_test.go +++ b/cmd/server/websocket_test.go @@ -413,3 +413,44 @@ func TestHubRegisterUnregister(t *testing.T) { t.Errorf("expected 0 clients, got %d", hub.ClientCount()) } } + +// #1074: browsers cannot see protocol-level pings, so a page cannot tell a +// quiet mesh from a dead socket unless the server also sends something the +// page receives. The heartbeat rides the same ticker as the ping. +func TestWritePumpSendsAppHeartbeat(t *testing.T) { + hub := NewHub() + hub.pingInterval = 20 * time.Millisecond + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hub.ServeWS(w, r) + })) + defer srv.Close() + + conn, _, err := websocket.DefaultDialer.Dial("ws"+srv.URL[4:], nil) + if err != nil { + t.Fatalf("dial error: %v", err) + } + defer conn.Close() + + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + kind, msg, err := conn.ReadMessage() + if err != nil { + t.Fatalf("expected a heartbeat with no broadcast traffic, got read error: %v", err) + } + if kind != websocket.TextMessage { + t.Fatalf("heartbeat must be a text frame so page JS receives it, got frame type %d", kind) + } + // public/app.js matches these exact bytes to keep heartbeats away from + // page listeners; changing them breaks that match. + if string(msg) != `{"type":"heartbeat"}` { + t.Fatalf("heartbeat bytes changed: %s", msg) + } +} + +// The client's stale threshold (WS_STALE_MS in public/app.js, 75s) assumes +// a heartbeat at least every 30s. +func TestHubDefaultPingInterval(t *testing.T) { + if got := NewHub().pingInterval; got != 30*time.Second { + t.Fatalf("pingInterval = %v, want 30s (public/app.js WS_STALE_MS depends on it)", got) + } +} diff --git a/public/app.js b/public/app.js index 9aad3d31..5969dbed 100644 --- a/public/app.js +++ b/public/app.js @@ -635,6 +635,18 @@ function buildHexLegend(ranges) { let ws = null; let wsListeners = []; +// #1074: a half-open connection (a proxy idle timeout, NAT state dropped, a +// laptop that slept) can keep a WebSocket OPEN for minutes without onclose +// ever firing, and the page just stops updating. The server writes +// WS_HEARTBEAT on every 30s ping tick (cmd/server/websocket.go), so a socket +// that has received nothing for WS_STALE_MS is replaced. 75s tolerates one +// late or lost heartbeat. +const WS_STALE_MS = 75000; +const WS_HEARTBEAT = '{"type":"heartbeat"}'; +let wsLastMessageAt = 0; +let wsWatchdogTimer = null; +let wsReconnectTimer = null; + // --- Brand-logo packet-driven pulse (#1173) --- // Replaces the legacy live-dot indicator. Class-toggle only (CSS animations); colors come from // --logo-accent / --logo-accent-hi tokens. Test seam at window.__corescopeLogo. @@ -778,20 +790,60 @@ const Logo = (function () { return api; })(); +// Detach before closing: a half-open socket may not fire onclose until the +// browser gives up on the closing handshake, and a late onclose from a socket +// already replaced would schedule a second connection. +function dropWS() { + clearTimeout(wsWatchdogTimer); + wsWatchdogTimer = null; + if (!ws) return; + const old = ws; + ws = null; + old.onopen = old.onclose = old.onerror = old.onmessage = null; + try { old.close(); } catch (_) {} +} + +function checkWSLiveness() { + if (!ws) return; + clearTimeout(wsWatchdogTimer); + const silentMs = Date.now() - wsLastMessageAt; + // A negative reading means the wall clock stepped back since the last + // message, so the silence can no longer be measured: replace the socket + // like a stale one rather than re-arm for the size of the step. Date.now() + // stays the clock because performance.now() may not count time asleep. + if (silentMs >= 0 && silentMs < WS_STALE_MS) { + wsWatchdogTimer = setTimeout(checkWSLiveness, WS_STALE_MS - silentMs); + return; + } + Logo.setConnected(false); + connectWS(); +} + function connectWS() { + clearTimeout(wsReconnectTimer); + wsReconnectTimer = null; + dropWS(); const proto = location.protocol === 'https:' ? 'wss:' : 'ws:'; - ws = new WebSocket(`${proto}//${location.host}`); - ws.onopen = () => Logo.setConnected(true); - ws.onclose = () => { + const sock = new WebSocket(`${proto}//${location.host}`); + ws = sock; + // Measured from creation, so a handshake that never completes is caught too. + wsLastMessageAt = Date.now(); + wsWatchdogTimer = setTimeout(checkWSLiveness, WS_STALE_MS); + sock.onopen = () => Logo.setConnected(true); + sock.onclose = () => { + clearTimeout(wsWatchdogTimer); + wsWatchdogTimer = null; Logo.setConnected(false); // WS_RECONNECT_MS comes from roles.js and is settable as cacheTTL's // sibling `wsReconnectMs`. It used to apply only to the live map's own // socket; now that every view shares this one, the operator's setting // applies here or nowhere. - setTimeout(connectWS, window.WS_RECONNECT_MS || 3000); + wsReconnectTimer = setTimeout(connectWS, window.WS_RECONNECT_MS || 3000); }; - ws.onerror = () => ws.close(); - ws.onmessage = (e) => { + sock.onerror = () => sock.close(); + sock.onmessage = (e) => { + wsLastMessageAt = Date.now(); + if (e.data === WS_HEARTBEAT) return; Logo.pulse(e); try { const msg = JSON.parse(e.data); @@ -808,6 +860,15 @@ function connectWS() { }; } +// Timers in a hidden or sleeping tab can run late, so check as soon as the +// page is back instead of waiting out a watchdog that may be minutes behind. +function setupWSResumeCheck() { + document.addEventListener('visibilitychange', () => { + if (!document.hidden) checkWSLiveness(); + }); + window.addEventListener('online', checkWSLiveness); +} + function onWS(fn) { wsListeners.push(fn); } function offWS(fn) { wsListeners = wsListeners.filter(f => f !== fn); } @@ -866,16 +927,11 @@ function pullReconnect() { // If WS is connected (readyState OPEN), give a brief "Connected" // confirmation but still cycle so the user sees fresh data. const wasOpen = ws && ws.readyState === 1; - if (wasOpen) { - _showPullToast('Connected', true); - // Fast cycle: close and let onclose reconnect immediately - try { ws.close(); } catch (e) {} - } else { - _showPullToast('Reconnecting…', true); - try { if (ws) ws.close(); } catch (e) {} - // onclose handler schedules reconnect; force one now in case ws was null - try { connectWS(); } catch (e) {} - } + _showPullToast(wasOpen ? 'Connected' : 'Reconnecting…', true); + // Replace the socket now in both cases: an OPEN socket may be half-open, + // and its onclose can take about a minute to fire after close(). + // connectWS() detaches and closes the old socket itself. + try { connectWS(); } catch (e) {} } function _isTouchDevice() { @@ -1201,6 +1257,7 @@ window.addEventListener('timestamp-mode-changed', () => { }); window.addEventListener('DOMContentLoaded', () => { connectWS(); + setupWSResumeCheck(); setupPullToReconnect(); // --- Dark Mode --- diff --git a/test-all.sh b/test-all.sh index e0063d97..349586c3 100755 --- a/test-all.sh +++ b/test-all.sh @@ -172,6 +172,7 @@ node test-top-routes-overlay.js node test-traces.js node test-url-state.js node test-warmup-banner.js +node test-ws-stale-watchdog-1074.js node test-xss-escape-sinks.js echo "All standalone frontend suites passed" diff --git a/test-pull-to-reconnect.js b/test-pull-to-reconnect.js index 5f2ccdc3..0a847dfe 100644 --- a/test-pull-to-reconnect.js +++ b/test-pull-to-reconnect.js @@ -1,8 +1,8 @@ /* test-pull-to-reconnect.js — behavioral tests for pull-to-reconnect (#1063) * Loads app.js in a vm sandbox, stubs WebSocket + DOM, asserts that: * - pullReconnect() exists as a global helper - * - calling it closes the existing WS (which triggers the existing - * auto-reconnect path) + * - calling it replaces the existing WS through connectWS(), which + * detaches and closes the old socket (#1074) * - setupPullToReconnect() exists and wires touchstart/touchmove/touchend * listeners on the document * - a pull-down gesture at scrollTop=0 over the threshold triggers diff --git a/test-ws-stale-watchdog-1074.js b/test-ws-stale-watchdog-1074.js new file mode 100644 index 00000000..3bba3f6b --- /dev/null +++ b/test-ws-stale-watchdog-1074.js @@ -0,0 +1,386 @@ +/* test-ws-stale-watchdog-1074.js: the shared WebSocket in app.js must notice + * a socket that has gone silent and replace it (#1074). + * + * A half-open TCP connection (a proxy idle timeout, NAT state dropped, a + * laptop that slept) can leave the browser's WebSocket in OPEN state for + * minutes without ever firing onclose, so the page silently stops updating. + * The server sends an app-level heartbeat every 30s; the client treats a + * socket that has received nothing for WS_STALE_MS as dead. + * + * Loads the real public/app.js in a vm sandbox with a fake clock, fake timers + * and a fake WebSocket. + */ +'use strict'; + +const vm = require('vm'); +const fs = require('fs'); +const assert = require('assert'); + +console.log('--- test-ws-stale-watchdog-1074.js ---'); + +// Mirrors WS_STALE_MS in public/app.js and wsHeartbeat in cmd/server/websocket.go. +const STALE_MS = 75000; +const HEARTBEAT = '{"type":"heartbeat"}'; + +let passed = 0, failed = 0; +function test(name, fn) { + try { fn(); passed++; console.log(` ✅ ${name}`); } + catch (e) { failed++; console.log(` ❌ ${name}: ${e.message}\n ${e.stack.split('\n').slice(1, 3).join('\n ')}`); } +} + +function inertElement() { + return new Proxy(function () {}, { + get(_, key) { + if (key === Symbol.toPrimitive) return () => ''; + if (key === Symbol.iterator) return function* () {}; + return inertElement(); + }, + set() { return true; }, + apply() { return inertElement(); }, + }); +} + +function makeSandbox() { + // now drives timers and performance.now(); wallOffset is added to Date.now() + // only, so a test can step the wall clock the way NTP or a user can. + const clock = { now: 1700000000000, wallOffset: 0 }; + let nextId = 1; + const timers = new Map(); // id -> { fn, at } + + function setTimeoutFake(fn, ms) { + const id = nextId++; + timers.set(id, { fn, at: clock.now + Math.max(0, ms || 0) }); + return id; + } + function clearTimeoutFake(id) { timers.delete(id); } + // Runs every timer due within the next `ms`, in due order, moving the clock. + function advance(ms) { + const end = clock.now + ms; + for (let fired = 0; ; fired++) { + if (fired > 10000) throw new Error('timer storm: over 10000 timers fired within ' + ms + 'ms'); + let dueId = null, due = null; + for (const [id, t] of timers) { + if (t.at <= end && (due === null || t.at < due.at)) { dueId = id; due = t; } + } + if (dueId === null) break; + timers.delete(dueId); + clock.now = due.at; + due.fn(); + } + clock.now = end; + } + + class FakeDate extends Date {} + FakeDate.now = () => clock.now + clock.wallOffset; + + const sockets = []; + function FakeWS(url) { + this.url = url; + this.readyState = 0; // CONNECTING + this.closed = false; + this.onopen = null; this.onclose = null; this.onerror = null; this.onmessage = null; + sockets.push(this); + } + FakeWS.prototype.open = function () { this.readyState = 1; if (this.onopen) this.onopen({}); }; + FakeWS.prototype.message = function (data) { if (this.onmessage) this.onmessage({ data }); }; + FakeWS.prototype.close = function () { + if (this.closed) return; + this.closed = true; + this.readyState = 3; + if (this.onclose) this.onclose({}); + }; + FakeWS.prototype.send = function () {}; + + const docListeners = {}; + const winListeners = {}; + const document = { + readyState: 'complete', + hidden: false, + documentElement: { scrollTop: 0, style: { setProperty() {} }, setAttribute() {}, getAttribute() { return null; } }, + body: { appendChild() {}, contains() { return true; } }, + head: { appendChild() {} }, + createElement() { return { style: {}, classList: { add() {}, remove() {} }, appendChild() {}, setAttribute() {} }; }, + getElementById() { return null; }, + querySelector() { return null; }, + querySelectorAll() { return []; }, + addEventListener(ev, fn) { (docListeners[ev] = docListeners[ev] || []).push(fn); }, + removeEventListener() {}, + }; + const window = { + addEventListener(ev, fn) { (winListeners[ev] = winListeners[ev] || []).push(fn); }, + removeEventListener() {}, + dispatchEvent() {}, + matchMedia() { return { matches: false, addEventListener() {} }; }, + WS_RECONNECT_MS: 3000, + }; + const ctx = { + console, + setTimeout: setTimeoutFake, clearTimeout: clearTimeoutFake, + setInterval() { return 0; }, clearInterval() {}, + Date: FakeDate, Math, JSON, Object, Array, String, Number, Boolean, + Error, RegExp, Map, Set, Symbol, Promise, + requestAnimationFrame() { return 0; }, + performance: { now: () => clock.now }, + location: { protocol: 'http:', host: 'localhost', hash: '' }, + navigator: { userAgent: 'test' }, + WebSocket: FakeWS, + fetch() { return Promise.resolve({ ok: true, json() { return Promise.resolve({}); } }); }, + localStorage: { getItem() { return null; }, setItem() {}, removeItem() {} }, + document, window, + CustomEvent: function (type, init) { this.type = type; this.detail = (init || {}).detail; }, + }; + window.location = ctx.location; + window.document = document; + ctx.self = window; + ctx.globalThis = ctx; + vm.createContext(ctx); + vm.runInContext(fs.readFileSync('public/app.js', 'utf8'), ctx); + + return { + ctx, clock, timers, advance, sockets, document, + fireDoc(ev) { (docListeners[ev] || []).forEach((fn) => fn({ type: ev })); }, + fireWin(ev) { (winListeners[ev] || []).forEach((fn) => fn({ type: ev })); }, + // Runs app.js's real startup listeners, as the browser does. The startup + // code wires the whole page shell, so every element lookup gets an inert + // stand-in that accepts any property or call. + boot() { + document.getElementById = document.querySelector = () => inertElement(); + ctx.getComputedStyle = () => inertElement(); + (winListeners.DOMContentLoaded || []).forEach((fn) => fn({ type: 'DOMContentLoaded' })); + }, + }; +} + +console.log('\n=== silence past the stale threshold replaces the socket ==='); + +test('an open socket that receives nothing for WS_STALE_MS is closed and replaced exactly once', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + + box.advance(STALE_MS - 1); + assert.strictEqual(box.sockets.length, 1, 'must not reconnect before the threshold'); + assert.strictEqual(first.closed, false, 'must not close before the threshold'); + + box.advance(1); + assert.strictEqual(first.closed, true, 'the silent socket must be closed'); + assert.strictEqual(box.sockets.length, 2, 'exactly one replacement socket must be opened'); + + // Nothing else is pending that would open a third socket: the replacement + // is still connecting, and the dead socket's handlers are detached. + box.advance(STALE_MS - 1); + assert.strictEqual(box.sockets.length, 2, 'no second replacement'); + assert.strictEqual(first.onclose, null, 'the dropped socket must not be able to schedule a reconnect'); + assert.strictEqual(first.onmessage, null, 'the dropped socket must not dispatch messages'); +}); + +test('a socket that never opens is also replaced after WS_STALE_MS', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + box.advance(STALE_MS); + assert.strictEqual(box.sockets[0].closed, true); + assert.strictEqual(box.sockets.length, 2); +}); + +console.log('\n=== regular traffic keeps the socket ==='); + +test('heartbeats every 30s keep one socket alive for ten minutes', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + for (let i = 0; i < 20; i++) { + box.advance(30000); + first.message(HEARTBEAT); + } + assert.strictEqual(box.sockets.length, 1, 'no reconnect while heartbeats arrive'); + assert.strictEqual(first.closed, false); +}); + +test('packet traffic without heartbeats also counts as liveness', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + for (let i = 0; i < 20; i++) { + box.advance(STALE_MS - 1000); + first.message(JSON.stringify({ type: 'packet', data: { id: i } })); + } + assert.strictEqual(box.sockets.length, 1); + assert.strictEqual(first.closed, false); +}); + +test('a heartbeat is not dispatched to page listeners and does not pulse the logo', () => { + const box = makeSandbox(); + const seen = []; + box.ctx.onWS((m) => seen.push(m)); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + const logo = box.ctx.window.__corescopeLogo; + const before = logo.stats.triggered + logo.stats.dropped; + + first.message(HEARTBEAT); + assert.strictEqual(seen.length, 0, 'heartbeat must not reach onWS listeners'); + assert.strictEqual(logo.stats.triggered + logo.stats.dropped, before, 'heartbeat must not pulse the logo'); + + first.message(JSON.stringify({ type: 'packet', data: { id: 1 } })); + assert.strictEqual(seen.length, 1, 'a packet must still reach listeners'); +}); + +console.log('\n=== resuming a hidden tab or coming back online checks immediately ==='); + +function throttledJump(box, ms) { + // A hidden tab's timers can be held back; move the clock without running them. + box.clock.now += ms; +} + +test('visibilitychange to visible after a long silence reconnects without waiting for the timer', () => { + const box = makeSandbox(); + // The real startup path, so a page that never calls setupWSResumeCheck fails here. + box.boot(); + assert.strictEqual(box.sockets.length, 1, 'startup opens one socket'); + box.sockets[0].open(); + + box.document.hidden = true; + box.fireDoc('visibilitychange'); + throttledJump(box, 5 * 60 * 1000); + box.document.hidden = false; + box.fireDoc('visibilitychange'); + + assert.strictEqual(box.sockets[0].closed, true, 'the silent socket must be dropped on resume'); + assert.strictEqual(box.sockets.length, 2, 'a replacement must be opened on resume'); +}); + +test('visibilitychange to visible with recent traffic keeps the socket', () => { + const box = makeSandbox(); + box.ctx.setupWSResumeCheck(); + box.ctx.connectWS(); + box.sockets[0].open(); + throttledJump(box, 20000); + box.sockets[0].message(HEARTBEAT); + throttledJump(box, 20000); + box.fireDoc('visibilitychange'); + assert.strictEqual(box.sockets.length, 1); + assert.strictEqual(box.sockets[0].closed, false); + assert.strictEqual(box.timers.size, 1, 'the check must replace the watchdog, not add a second one'); +}); + +test('becoming hidden does not trigger a check', () => { + const box = makeSandbox(); + box.ctx.setupWSResumeCheck(); + box.ctx.connectWS(); + box.sockets[0].open(); + throttledJump(box, 5 * 60 * 1000); + box.document.hidden = true; + box.fireDoc('visibilitychange'); + assert.strictEqual(box.sockets.length, 1); +}); + +test('the online event after a long silence reconnects immediately', () => { + const box = makeSandbox(); + box.ctx.setupWSResumeCheck(); + box.ctx.connectWS(); + box.sockets[0].open(); + throttledJump(box, STALE_MS + 1); + box.fireWin('online'); + assert.strictEqual(box.sockets[0].closed, true); + assert.strictEqual(box.sockets.length, 2); +}); + +test('repeated resume events open one replacement, not one each', () => { + const box = makeSandbox(); + box.ctx.setupWSResumeCheck(); + box.ctx.connectWS(); + box.sockets[0].open(); + throttledJump(box, STALE_MS + 1); + box.fireWin('online'); + box.fireDoc('visibilitychange'); + box.fireWin('online'); + assert.strictEqual(box.sockets.length, 2, 'the replacement is fresh, so later checks leave it alone'); +}); + +console.log('\n=== a wall-clock step does not disable the watchdog ==='); + +test('the clock stepping back an hour as the socket goes silent still replaces it within WS_STALE_MS', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + for (let i = 0; i < 3; i++) { + box.advance(30000); + first.message(HEARTBEAT); + } + box.clock.wallOffset -= 60 * 60 * 1000; // step back, then nothing more arrives + box.advance(STALE_MS); + assert.strictEqual(first.closed, true, 'the silent socket must be dropped'); + assert.strictEqual(box.sockets.length, 2, 'one replacement within WS_STALE_MS of the step'); +}); + +for (const [label, stepMs] of [['forward', 60 * 60 * 1000], ['back', -60 * 60 * 1000]]) { + test(`a clock step ${label} on a healthy socket costs at most one extra reconnect`, () => { + const box = makeSandbox(); + box.ctx.connectWS(); + box.sockets[0].open(); + box.advance(40000); + box.sockets[0].message(HEARTBEAT); + box.advance(10000); + box.clock.wallOffset += stepMs; + for (let i = 0; i < 20; i++) { + box.advance(30000); + const cur = box.sockets[box.sockets.length - 1]; + if (cur.readyState === 0) cur.open(); + cur.message(HEARTBEAT); + } + assert.ok(box.sockets.length <= 2, 'got ' + box.sockets.length + ' sockets over ten minutes'); + }); +} + +console.log('\n=== timers are cleaned up on close ==='); + +test('after onclose only the reconnect timer is pending, and it opens one socket', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + first.close(); // the browser saw the close + + assert.strictEqual(box.timers.size, 1, 'the watchdog must be cleared; only the reconnect remains'); + box.advance(3000); + assert.strictEqual(box.sockets.length, 2, 'the reconnect opens one socket'); + box.sockets[1].open(); + for (let i = 0; i < 6; i++) { + box.advance(30000); + box.sockets[1].message(HEARTBEAT); + } + assert.strictEqual(box.sockets.length, 2, 'the closed socket\'s watchdog must not fire later'); +}); + +test('pullReconnect on a socket that is not open leaves one socket and no stray reconnect', () => { + const box = makeSandbox(); + box.ctx.connectWS(); // still CONNECTING + box.ctx.window.pullReconnect(); + assert.strictEqual(box.sockets.length, 2, 'pull opens a replacement at once'); + assert.strictEqual(box.sockets[0].closed, true, 'the previous socket is closed'); + box.advance(3000); + assert.strictEqual(box.sockets.length, 2, 'the old socket\'s close must not schedule a third socket'); +}); + +test('pullReconnect on an open socket replaces it at once instead of waiting for onclose', () => { + const box = makeSandbox(); + box.ctx.connectWS(); + const first = box.sockets[0]; + first.open(); + // A half-open socket may not fire onclose for about a minute after close(). + first.close = function () { this.closed = true; this.readyState = 2; }; + box.ctx.window.pullReconnect(); + assert.strictEqual(box.sockets.length, 2, 'the replacement must exist right after the pull'); + assert.strictEqual(first.closed, true, 'the previous socket is closed'); + assert.strictEqual(first.onclose, null, 'the previous socket is detached, so a late onclose cannot reconnect again'); + box.advance(STALE_MS - 1); + assert.strictEqual(box.sockets.length, 2, 'exactly one socket results from the pull'); +}); + +console.log('\n=== Results: ' + passed + ' passed, ' + failed + ' failed ===\n'); +process.exit(failed > 0 ? 1 : 0);