Files
meshcore-analyzer/cmd/server/websocket.go
T
efitenandClaude Opus 5 7f8f71e773 fix(live): reconnect when the websocket goes silent (#1074) (#2020)
## Problem

#1074 reports that after a proxy dropped the WebSocket, live updates
only came back 8 to 10 minutes later.

The client only reconnects from `onclose` (`public/app.js:791` on
master). A half-open connection (a proxy or NAT dropping state without a
FIN reaching the browser, a laptop that slept) can keep a WebSocket OPEN
for minutes without `onclose`, and nothing retries in the meantime.

The server does ping every 30s (`cmd/server/websocket.go:252` on
master), but ping frames are answered by the browser below the page and
JS cannot observe them. The only app-level frames are packet broadcasts
(`websocket.go:337, 343, 366`), which stop on a quiet mesh. So the
client had no signal to tell a quiet mesh from a dead socket.

## Change

Server (`cmd/server/websocket.go`):
- On the existing ping tick, `writePump` also writes the text frame
`{"type":"heartbeat"}` (`:283`, bytes at `:80`). One 20-byte frame per
client per 30s, from the goroutine that already writes the ping, no hub
lock.
- The interval moves to `Hub.pingInterval` (default 30s, `:118`) so a
test can shorten it.

Client (`public/app.js`):
- Every frame refreshes `wsLastMessageAt` (`:841`). One timer
(`checkWSLiveness`, `:806`) fires at last message + `WS_STALE_MS` (75s,
one late or lost heartbeat of slack) and replaces the socket if it is
still silent. It is armed at socket creation, so a stuck handshake is
covered too.
- `dropWS` (`:796`) detaches the old socket's handlers before `close()`,
so a late close event cannot schedule a second connection.
- `connectWS` (`:818`) cancels a pending reconnect and drops the
previous socket, so the watchdog, resume checks, `onclose` and
pull-to-reconnect cannot stack sockets. Before this, `pullReconnect` on
a non-open socket left a third socket 3s later. The 3s `WS_RECONNECT_MS`
delay after `onclose` is unchanged (`:837`).
- `visibilitychange` (to visible) and `online` run the check immediately
(`:861`), because a hidden or sleeping tab's timers can run late.
- Heartbeat frames are matched by exact bytes (`:842`) and are not
pulsed or dispatched to `onWS` listeners.

Compatibility: tabs loaded before the deploy dispatch heartbeats to
their listeners until reloaded. Every current listener filters on
`msg.type`, so the visible effect is a logo pulse and a `/stats` cache
refresh every 30s.

Perf: one `Date.now()` and one string compare per WS message on the
client; one extra 20-byte write per client per 30s on the server.

## Tests

- `test-ws-stale-watchdog-1074.js`: real `app.js` in a vm with a fake
clock, timers and WebSocket. 12 tests: silence past the threshold
replaces the socket exactly once; a handshake that never opens is
replaced; heartbeats and packet traffic keep the socket; heartbeats are
not dispatched; resume and `online` after silence reconnect immediately,
with recent traffic they do not, and hiding does not trigger a check;
repeated resume events open one socket; after `onclose` only the
reconnect timer is pending; pull-to-reconnect leaves one socket. 9 of 12
fail on master. 12 of 12 source mutations (threshold, reconnect path,
detaching, timer cleanup, resume wiring, heartbeat filter) are caught.
Registered in `test-all.sh` and the deploy.yml unit step.
- `TestWritePumpSendsAppHeartbeat`: fails with a read timeout without
the heartbeat, even with pings every 20ms. `TestHubDefaultPingInterval`
pins the 30s interval that `WS_STALE_MS` assumes.
- `go test ./...` in `cmd/server` passes; gofmt and go vet are clean.

## Browser validation

On a staging instance (build `139e484e`, together with #1979's branch),
in Chrome, no console errors:

- A `{"type":"heartbeat"}` frame arrived on the open socket within the
observation window.
- Silent socket: after `ws.onmessage = null`, the page replaced the
socket after 76.2s (threshold 75s plus a 250ms poll); the old socket
ended in CLOSED, the new one OPEN.
- Normal close: `ws.close()` led to a new OPEN socket after 4.1s, and
exactly one new `WebSocket` was constructed.

## Not verified

- The reporter's proxy setup was not reproduced; that their delay was a
half-open socket is a hypothesis consistent with the symptom. Hence
`Refs`, not `Fixes`.
- Laptop sleep and the `visibilitychange` / `online` resume path were
only covered by the unit test, not in a browser.
- Behaviour under Chrome's intensive background-timer throttling and
mobile tab freezing was not measured; a frozen but healthy tab may do
one unnecessary reconnect on resume.
- Go tests were run without `-race`.

Refs #1074



## Review follow-up (commit `72e5e906`)

An independent review found no blocking bug: all data writes stay on the
write goroutine, pong-based dead-client detection still works, and no
ordering of onclose, watchdog, resume checks and pull ends with two live
sockets or none. It reproduced the silent-socket case in headless
Chromium through a blackholing TCP proxy (replacement 75.0 s after the
last frame). Changed:

1. **Startup wiring tested.** The first resume test now boots through
the page's real `DOMContentLoaded` listeners, so removing
`setupWSResumeCheck()` from startup makes it fail.
2. **Wall clock stepping back.** If the clock steps back after the last
message, the watchdog no longer re-arms for the size of the step (a 1 h
step used to delay detection by about an hour). A negative silence
reading is treated as stale, so the socket is replaced within
`WS_STALE_MS` of the step (`public/app.js:810-814`). A step in either
direction costs at most one extra reconnect on a healthy socket.
`Date.now()` stays the clock so a tab resumed after sleep is still
checked against real elapsed time.
3. **Pull-to-reconnect at once.** On an OPEN socket, pull-to-reconnect
now replaces it through `connectWS()` instead of closing it and waiting
for onclose, which took 63 s on a half-open connection in the review's
measurement (`public/app.js:927-934`). This was slow on master too; it
is safe now that `connectWS()` detaches the old socket.

Tests: 12 to 16 in `test-ws-stale-watchdog-1074.js`;
`test-pull-to-reconnect.js`, `test-pull-to-reconnect-1091.js` and
`test-live.js` pass.

Correction to the compatibility note: tabs opened before the deploy
treat the heartbeat like any other message. Besides the logo pulse,
`app.js` runs `updateNavStats` on every message and invalidates the
cached `/stats` and `/nodes` responses 5 s later; `packets.js` also
pushes every message into `pauseBuffer` unfiltered (~1310-1313), so an
old tab with Packets paused sees its counter rise by 2 per minute.
Cosmetic: heartbeats are filtered out on replay, and a reload ends it.

Not verified: real hidden-tab or mobile freeze behaviour, Firefox and
Safari, and the reporter's proxy setup.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 19:59:18 +02:00

394 lines
11 KiB
Go

package main
import (
"encoding/json"
"log"
"net/http"
"net/url"
"strings"
"sync"
"time"
"github.com/gorilla/websocket"
)
// Hub manages WebSocket clients and broadcasts.
type Hub struct {
mu sync.RWMutex
clients map[*Client]bool
upgrader websocket.Upgrader
allowedOrigins []string // exact-match allowlist for /ws CheckOrigin (see SetAllowedOrigins)
limits *wsLimiter // #1794: per-IP caps and deny list; nil allows everything
pingInterval time.Duration
}
// SetAllowedOrigins configures the exact-match origin allowlist consulted by
// the WebSocket upgrader's CheckOrigin. The "*" wildcard is deliberately NOT
// honored here (it IS honored by the HTTP CORS middleware): OWASP's
// WebSocket Security Cheat Sheet recommends an explicit allowlist for CSWSH
// defense. If "*" appears in the slice, it is ignored and a startup WARN is
// logged once per call.
//
// See: https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html
func (h *Hub) SetAllowedOrigins(origins []string) {
h.mu.Lock()
defer h.mu.Unlock()
h.allowedOrigins = append(h.allowedOrigins[:0], origins...)
for _, o := range origins {
if o == "*" {
log.Println(`[ws] WARNING: CORSAllowedOrigins contains "*" — CORS allows any origin for XHR, but /ws upgrade enforces explicit allowlist only (OWASP CSWSH guidance). Add specific origins to allow cross-origin WebSocket clients.`)
break
}
}
}
// checkOrigin is the gorilla/websocket Upgrader.CheckOrigin hook. Rules:
// - empty Origin header → allow (non-browser client; rate-limit / IP gate
// is handled separately, see #1794).
// - Origin host == request Host (same-origin) → allow.
// - Origin in allowedOrigins by exact case-insensitive match → allow.
// - "*" in allowedOrigins is ignored (see SetAllowedOrigins).
// - anything else → reject (gorilla returns 403).
func (h *Hub) checkOrigin(r *http.Request) bool {
origin := r.Header.Get("Origin")
if origin == "" {
return true
}
u, err := url.Parse(origin)
if err != nil {
return false
}
if strings.EqualFold(u.Host, r.Host) {
return true
}
h.mu.RLock()
allowed := h.allowedOrigins
h.mu.RUnlock()
for _, o := range allowed {
if o == "*" {
continue // deliberately not honored — see SetAllowedOrigins
}
if strings.EqualFold(o, origin) {
return true
}
}
return false
}
// wsHeartbeat is written to every client on each ping tick. public/app.js
// compares incoming frames against these exact bytes, so keep them in step.
var wsHeartbeat = []byte(`{"type":"heartbeat"}`)
// Client is a single WebSocket connection.
type Client struct {
conn *websocket.Conn
send chan []byte
closeOnce sync.Once
// release returns this connection's slot to the per-IP concurrent cap
// (#1794). Always non-nil so Unregister needs no nil check; it is a no-op
// when limits are disabled.
release func()
}
// ConfigureLimits installs the #1794 transport limits. Called once at
// startup, before the listener binds, so no upgrade can race a half-built
// limiter. Passing zero values for both caps leaves only the deny list
// active, which is a legitimate configuration.
func (h *Hub) ConfigureLimits(maxConnsPerIP, upgradesPerMin int, trustedProxies, deny []string) {
l := newWSLimiter()
l.maxConnsPerIP = maxConnsPerIP
l.upgradesPerMin = upgradesPerMin
l.trustedProxies = parseCIDRList(trustedProxies, "webSocket.trustedProxies")
l.denyNets = parseCIDRList(deny, "webSocket.deny")
h.mu.Lock()
h.limits = l
h.mu.Unlock()
if len(l.denyNets) > 0 {
log.Printf("[ws] deny list active: %d network(s)", len(l.denyNets))
}
if len(l.trustedProxies) == 0 && (maxConnsPerIP > 0 || upgradesPerMin > 0) {
log.Printf("[ws] per-IP limits configured with no trustedProxies: they apply only to " +
"directly-connected clients. Behind a reverse proxy, set webSocket.trustedProxies.")
}
}
func NewHub() *Hub {
h := &Hub{
clients: make(map[*Client]bool),
pingInterval: 30 * time.Second,
}
h.upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 4096,
CheckOrigin: h.checkOrigin,
}
return h
}
func (h *Hub) ClientCount() int {
h.mu.RLock()
defer h.mu.RUnlock()
return len(h.clients)
}
func (h *Hub) Register(c *Client) {
h.mu.Lock()
h.clients[c] = true
h.mu.Unlock()
log.Printf("[ws] client connected (%d total)", h.ClientCount())
}
func (h *Hub) Unregister(c *Client) {
h.mu.Lock()
if _, ok := h.clients[c]; ok {
delete(h.clients, c)
c.closeOnce.Do(func() { close(c.send) })
}
h.mu.Unlock()
// #1794: outside the lock — release takes the limiter's own mutex, and
// holding two locks in one order here and the other order elsewhere is how
// deadlocks are made. Idempotent, so a double Unregister cannot over-credit.
if c.release != nil {
c.release()
}
log.Printf("[ws] client disconnected (%d total)", h.ClientCount())
}
// Close gracefully disconnects all WebSocket clients.
func (h *Hub) Close() {
h.mu.Lock()
for c := range h.clients {
c.conn.WriteControl(
websocket.CloseMessage,
websocket.FormatCloseMessage(websocket.CloseGoingAway, "server shutting down"),
time.Now().Add(3*time.Second),
)
c.closeOnce.Do(func() { close(c.send) })
delete(h.clients, c)
}
h.mu.Unlock()
log.Println("[ws] all clients disconnected")
}
// Broadcast sends a message to all connected clients.
func (h *Hub) Broadcast(msg interface{}) {
data, err := json.Marshal(msg)
if err != nil {
log.Printf("[ws] marshal error: %v", err)
return
}
h.mu.RLock()
defer h.mu.RUnlock()
for c := range h.clients {
select {
case c.send <- data:
default:
// Client buffer full — drop
}
}
}
// ServeWS handles the WebSocket upgrade and runs the client.
func (h *Hub) ServeWS(w http.ResponseWriter, r *http.Request) {
// #1794: decide BEFORE the upgrade. Rejecting afterwards would already
// have allocated the connection and completed the handshake, which is the
// resource this is meant to protect.
ok, reason, key, release := h.limits.allow(r)
if !ok {
status := http.StatusForbidden
if reason == wsRejectRate {
// 429 tells a well-behaved client to back off; 403 would read as
// "never come back" for what is a temporary refusal.
status = http.StatusTooManyRequests
}
log.Printf("[ws] reject ip=%s reason=%s", key, reason)
http.Error(w, "websocket upgrade refused", status)
return
}
conn, err := h.upgrader.Upgrade(w, r, nil)
if err != nil {
log.Printf("[ws] upgrade error: %v", err)
release() // the slot was charged above and this connection never happened
return
}
client := &Client{
conn: conn,
send: make(chan []byte, 256),
release: release,
}
h.Register(client)
go client.writePump(h.pingInterval)
go client.readPump(h)
}
// wsOrStatic upgrades WebSocket requests at any path, serves static files otherwise.
func wsOrStatic(hub *Hub, static http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.EqualFold(r.Header.Get("Upgrade"), "websocket") {
hub.ServeWS(w, r)
return
}
static.ServeHTTP(w, r)
})
}
func (c *Client) readPump(hub *Hub) {
defer func() {
hub.Unregister(c)
c.conn.Close()
}()
c.conn.SetReadLimit(512)
c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
c.conn.SetPongHandler(func(string) error {
c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
return nil
})
for {
_, _, err := c.conn.ReadMessage()
if err != nil {
break
}
}
}
func (c *Client) writePump(pingInterval time.Duration) {
ticker := time.NewTicker(pingInterval)
defer func() {
ticker.Stop()
c.conn.Close()
}()
for {
select {
case message, ok := <-c.send:
c.conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
if !ok {
c.conn.WriteMessage(websocket.CloseMessage, []byte{})
return
}
if err := c.conn.WriteMessage(websocket.TextMessage, message); err != nil {
return
}
case <-ticker.C:
c.conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
if err := c.conn.WriteMessage(websocket.PingMessage, nil); err != nil {
return
}
// #1074: the ping above keeps proxies and this server's read
// deadline happy, but browser JS never sees ping frames. Without a
// frame the page receives, a quiet mesh and a dead socket look the
// same to the client, so it could not detect the latter.
if err := c.conn.WriteMessage(websocket.TextMessage, wsHeartbeat); err != nil {
return
}
}
}
}
// Poller watches for new transmissions in SQLite and broadcasts them.
type Poller struct {
db *DB
hub *Hub
store *PacketStore // optional: if set, new transmissions are ingested into memory
interval time.Duration
stop chan struct{}
}
func NewPoller(db *DB, hub *Hub, interval time.Duration) *Poller {
return &Poller{db: db, hub: hub, interval: interval, stop: make(chan struct{})}
}
func (p *Poller) Start() {
lastID := p.db.GetMaxTransmissionID()
lastObsID := p.db.GetMaxObservationID()
// If the store already loaded data, use its max IDs as a floor.
// This prevents replaying the entire DB when the DB query fails
// (e.g., corrupted DB returns 0 from COALESCE).
if p.store != nil {
if storeMax := p.store.MaxTransmissionID(); storeMax > lastID {
lastID = storeMax
}
if storeMaxObs := p.store.MaxObservationID(); storeMaxObs > lastObsID {
lastObsID = storeMaxObs
}
}
log.Printf("[poller] starting from transmission ID %d, obs ID %d, interval %v", lastID, lastObsID, p.interval)
ticker := time.NewTicker(p.interval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
if p.store != nil {
// Ingest new transmissions into in-memory store and broadcast
newTxs, newMax := p.store.IngestNewFromDB(lastID, 100)
if newMax > lastID {
lastID = newMax
}
// Ingest new observations for existing transmissions (fixes #174)
nextObsID := lastObsID
if err := p.db.conn.QueryRow(`
SELECT COALESCE(MAX(id), ?) FROM (
SELECT id FROM observations
WHERE id > ?
ORDER BY id ASC
LIMIT 500
)`, lastObsID, lastObsID).Scan(&nextObsID); err != nil {
nextObsID = lastObsID
}
newObs := p.store.IngestNewObservations(lastObsID, 500)
if nextObsID > lastObsID {
lastObsID = nextObsID
}
if len(newTxs) > 0 {
log.Printf("[broadcast] sending %d packets to %d clients (lastID now %d)", len(newTxs), p.hub.ClientCount(), lastID)
}
for _, tx := range newTxs {
p.hub.Broadcast(WSMessage{
Type: "packet",
Data: tx,
})
}
for _, obs := range newObs {
p.hub.Broadcast(WSMessage{
Type: "packet",
Data: obs,
})
}
} else {
// Fallback: direct DB query (used when store is nil, e.g. tests)
newTxs, err := p.db.GetNewTransmissionsSince(lastID, 100)
if err != nil {
log.Printf("[poller] error: %v", err)
continue
}
for _, tx := range newTxs {
id, _ := tx["id"].(int)
if id > lastID {
lastID = id
}
// Copy packet fields for the nested packet (avoids circular ref)
pkt := make(map[string]interface{}, len(tx))
for k, v := range tx {
pkt[k] = v
}
tx["packet"] = pkt
p.hub.Broadcast(WSMessage{
Type: "packet",
Data: tx,
})
}
}
case <-p.stop:
return
}
}
}
func (p *Poller) Stop() {
close(p.stop)
}