mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 04:57:07 +00:00
## Summary Two pre-existing ways a running server **permanently** missed data. The database was correct; the server's in-memory store was not, and stayed wrong until the next restart. Found while comparing several server replicas side by side in #2396: two replicas that restarted mid-ingest each lacked one transmission and its 12 observations. Split out of #2396 at review's request, because it affects every SQLite install. ## The bugs 1. **Startup window.** The poller took its starting cursors from a fresh `MAX(id)` *after* the startup load finished. A row committed between the load's last chunk snapshot and that query ended up in neither the store nor any later poll. Its later observations were then dropped as orphans. 2. **Failed poll.** A failed observation read still advanced the cursor to its upper bound, skipping up to 500 observations for good. ## The fix - `NewPacketStore` records the transmission and observation high-water marks **before** the load, as the advert-evidence and rewrite feeds already did. - The loaded max transmission id stays as a floor. - The loaded max observation id no longer is one: a chunk-1 transmission can gain an observation whose id is *below* one that chunk 2 loaded. - Rows read twice are deduplicated and broadcast at most once. - A failed read keeps the cursor and retries on the next tick, logging once per failure streak. - Comments and a test pin why an observation of a transmission whose chunk has not merged yet cannot be skipped: the poller's evidence refresh waits on that chunk's lock. ## Performance - **Startup:** two extra indexed `MAX(id)` queries, which are primary-key lookups. - **Poll loop:** unchanged complexity. The only new per-tick work is a dedupe check on rows read twice, a map lookup by id (O(1)). Re-reads happen only for rows in the startup window, which is bounded by what was committed during the load, and on a retry after a failed read. ## Testing `cmd/server/startup_poll_gap_test.go` seeds the ingestor's own DDL plus `dbschema.Apply`: | Test | What it pins | |------|--------------| | `TestStartupLoadPollerGap_SQLite` | A transmission and observations committed between the load's snapshot and the poller's start reach the store | | `TestStartupLoadPollerGapMultiChunk_SQLite` | Same, across 3 chunks, including an observation id below the max loaded one | | `TestStartupObsDuringChunkMerge_SQLite` | An observation of a transmission whose chunk has not merged yet is not skipped | | `TestPollerRetriesFailedRead_SQLite` | A failed observation read does not advance the cursor | **Red/green:** against master's `store.go`, `websocket.go`, `chunked_load.go` and `db.go`, the first, second and fourth tests fail with the rows missing, e.g.: ``` tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever) obs 3, committed while the observation read failed, never reached the store: the cursor passed it ``` The third depends on test hooks this change adds, so it cannot run against the old code. `go vet` and `go test` pass on all modules, with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Recommended **priority** (not a dependency): 1. **#2410** — musl 8 MiB thread stacks. A public, unauthenticated GET can SIGSEGV the whole server on the shipped image, so merge this first. 2. **#2408** — never lose rows during startup / a failed poll. 3. **#2407** — explicit `observers.idx` primary key. 4. **#2409** — content-hash migration moved into the ingestor. After these land, #2396 (Postgres) will be rebased onto master. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
447 lines
14 KiB
Go
447 lines
14 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"log"
|
|
"net/http"
|
|
"net/url"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/gorilla/websocket"
|
|
)
|
|
|
|
// Hub manages WebSocket clients and broadcasts.
|
|
type Hub struct {
|
|
mu sync.RWMutex
|
|
clients map[*Client]bool
|
|
upgrader websocket.Upgrader
|
|
allowedOrigins []string // allowlist for /ws CheckOrigin (see SetAllowedOrigins)
|
|
limits *wsLimiter // #1794: per-IP caps and deny list; nil allows everything
|
|
pingInterval time.Duration
|
|
}
|
|
|
|
// SetAllowedOrigins configures the origin allowlist consulted by the
|
|
// WebSocket upgrader's CheckOrigin: exact origins, or one wildcard label
|
|
// scoped to a project (originMatches). The "*" wildcard is deliberately NOT
|
|
// honored here (it IS honored by the HTTP CORS middleware): OWASP's
|
|
// WebSocket Security Cheat Sheet recommends an explicit allowlist for CSWSH
|
|
// defense. If "*" appears in the slice, it is ignored and a startup WARN is
|
|
// logged once per call.
|
|
//
|
|
// See: https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html
|
|
func (h *Hub) SetAllowedOrigins(origins []string) {
|
|
h.mu.Lock()
|
|
defer h.mu.Unlock()
|
|
h.allowedOrigins = append(h.allowedOrigins[:0], origins...)
|
|
for _, o := range origins {
|
|
if o == "*" {
|
|
log.Println(`[ws] WARNING: CORSAllowedOrigins contains "*" — CORS allows any origin for XHR, but /ws upgrade enforces explicit allowlist only (OWASP CSWSH guidance). Add specific origins to allow cross-origin WebSocket clients.`)
|
|
break
|
|
}
|
|
}
|
|
}
|
|
|
|
// checkOrigin is the gorilla/websocket Upgrader.CheckOrigin hook. Rules:
|
|
// - empty Origin header → allow (non-browser client; rate-limit / IP gate
|
|
// is handled separately, see #1794).
|
|
// - Origin host == request Host (same-origin) → allow.
|
|
// - Origin matches an allowedOrigins entry → allow (originMatches: exact
|
|
// and case-insensitive, or one wildcard label).
|
|
// - "*" in allowedOrigins is ignored (see SetAllowedOrigins).
|
|
// - anything else → reject (gorilla returns 403).
|
|
func (h *Hub) checkOrigin(r *http.Request) bool {
|
|
origin := r.Header.Get("Origin")
|
|
if origin == "" {
|
|
return true
|
|
}
|
|
u, err := url.Parse(origin)
|
|
if err != nil {
|
|
return false
|
|
}
|
|
if strings.EqualFold(u.Host, r.Host) {
|
|
return true
|
|
}
|
|
h.mu.RLock()
|
|
allowed := h.allowedOrigins
|
|
h.mu.RUnlock()
|
|
for _, o := range allowed {
|
|
if o == "*" {
|
|
continue // deliberately not honored — see SetAllowedOrigins
|
|
}
|
|
if originMatches(o, origin) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// wsHeartbeat is written to every client on each ping tick. public/app.js
|
|
// compares incoming frames against these exact bytes, so keep them in step.
|
|
var wsHeartbeat = []byte(`{"type":"heartbeat"}`)
|
|
|
|
// Client is a single WebSocket connection.
|
|
type Client struct {
|
|
conn *websocket.Conn
|
|
send chan []byte
|
|
closeOnce sync.Once
|
|
// release returns this connection's slot to the per-IP concurrent cap
|
|
// (#1794). Always non-nil so Unregister needs no nil check; it is a no-op
|
|
// when limits are disabled.
|
|
release func()
|
|
}
|
|
|
|
// ConfigureLimits installs the #1794 transport limits. Called once at
|
|
// startup, before the listener binds, so no upgrade can race a half-built
|
|
// limiter. Passing zero values for both caps leaves only the deny list
|
|
// active, which is a legitimate configuration.
|
|
func (h *Hub) ConfigureLimits(maxConnsPerIP, upgradesPerMin int, trustedProxies, deny []string) {
|
|
l := newWSLimiter()
|
|
l.maxConnsPerIP = maxConnsPerIP
|
|
l.upgradesPerMin = upgradesPerMin
|
|
l.trustedProxies = parseCIDRList(trustedProxies, "webSocket.trustedProxies")
|
|
l.denyNets = parseCIDRList(deny, "webSocket.deny")
|
|
h.mu.Lock()
|
|
h.limits = l
|
|
h.mu.Unlock()
|
|
if len(l.denyNets) > 0 {
|
|
log.Printf("[ws] deny list active: %d network(s)", len(l.denyNets))
|
|
}
|
|
if len(l.trustedProxies) == 0 && (maxConnsPerIP > 0 || upgradesPerMin > 0) {
|
|
log.Printf("[ws] per-IP limits configured with no trustedProxies: they apply only to " +
|
|
"directly-connected clients. Behind a reverse proxy, set webSocket.trustedProxies.")
|
|
}
|
|
}
|
|
|
|
func NewHub() *Hub {
|
|
h := &Hub{
|
|
clients: make(map[*Client]bool),
|
|
pingInterval: 30 * time.Second,
|
|
}
|
|
h.upgrader = websocket.Upgrader{
|
|
ReadBufferSize: 1024,
|
|
WriteBufferSize: 4096,
|
|
CheckOrigin: h.checkOrigin,
|
|
}
|
|
return h
|
|
}
|
|
|
|
func (h *Hub) ClientCount() int {
|
|
h.mu.RLock()
|
|
defer h.mu.RUnlock()
|
|
return len(h.clients)
|
|
}
|
|
|
|
func (h *Hub) Register(c *Client) {
|
|
h.mu.Lock()
|
|
h.clients[c] = true
|
|
h.mu.Unlock()
|
|
log.Printf("[ws] client connected (%d total)", h.ClientCount())
|
|
}
|
|
|
|
func (h *Hub) Unregister(c *Client) {
|
|
h.mu.Lock()
|
|
if _, ok := h.clients[c]; ok {
|
|
delete(h.clients, c)
|
|
c.closeOnce.Do(func() { close(c.send) })
|
|
}
|
|
h.mu.Unlock()
|
|
// #1794: outside the lock — release takes the limiter's own mutex, and
|
|
// holding two locks in one order here and the other order elsewhere is how
|
|
// deadlocks are made. Idempotent, so a double Unregister cannot over-credit.
|
|
if c.release != nil {
|
|
c.release()
|
|
}
|
|
log.Printf("[ws] client disconnected (%d total)", h.ClientCount())
|
|
}
|
|
|
|
// Close gracefully disconnects all WebSocket clients.
|
|
func (h *Hub) Close() {
|
|
h.mu.Lock()
|
|
for c := range h.clients {
|
|
c.conn.WriteControl(
|
|
websocket.CloseMessage,
|
|
websocket.FormatCloseMessage(websocket.CloseGoingAway, "server shutting down"),
|
|
time.Now().Add(3*time.Second),
|
|
)
|
|
c.closeOnce.Do(func() { close(c.send) })
|
|
delete(h.clients, c)
|
|
}
|
|
h.mu.Unlock()
|
|
log.Println("[ws] all clients disconnected")
|
|
}
|
|
|
|
// Broadcast sends a message to all connected clients.
|
|
func (h *Hub) Broadcast(msg interface{}) {
|
|
data, err := json.Marshal(msg)
|
|
if err != nil {
|
|
log.Printf("[ws] marshal error: %v", err)
|
|
return
|
|
}
|
|
h.mu.RLock()
|
|
defer h.mu.RUnlock()
|
|
for c := range h.clients {
|
|
select {
|
|
case c.send <- data:
|
|
default:
|
|
// Client buffer full — drop
|
|
}
|
|
}
|
|
}
|
|
|
|
// ServeWS handles the WebSocket upgrade and runs the client.
|
|
func (h *Hub) ServeWS(w http.ResponseWriter, r *http.Request) {
|
|
// #1794: decide BEFORE the upgrade. Rejecting afterwards would already
|
|
// have allocated the connection and completed the handshake, which is the
|
|
// resource this is meant to protect.
|
|
ok, reason, key, release := h.limits.allow(r)
|
|
if !ok {
|
|
status := http.StatusForbidden
|
|
if reason == wsRejectRate {
|
|
// 429 tells a well-behaved client to back off; 403 would read as
|
|
// "never come back" for what is a temporary refusal.
|
|
status = http.StatusTooManyRequests
|
|
}
|
|
log.Printf("[ws] reject ip=%s reason=%s", key, reason)
|
|
http.Error(w, "websocket upgrade refused", status)
|
|
return
|
|
}
|
|
|
|
conn, err := h.upgrader.Upgrade(w, r, nil)
|
|
if err != nil {
|
|
log.Printf("[ws] upgrade error: %v", err)
|
|
release() // the slot was charged above and this connection never happened
|
|
return
|
|
}
|
|
|
|
client := &Client{
|
|
conn: conn,
|
|
send: make(chan []byte, 256),
|
|
release: release,
|
|
}
|
|
h.Register(client)
|
|
|
|
go client.writePump(h.pingInterval)
|
|
go client.readPump(h)
|
|
}
|
|
|
|
// wsOrStatic upgrades WebSocket requests at any path, serves static files otherwise.
|
|
func wsOrStatic(hub *Hub, static http.Handler) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
if strings.EqualFold(r.Header.Get("Upgrade"), "websocket") {
|
|
hub.ServeWS(w, r)
|
|
return
|
|
}
|
|
static.ServeHTTP(w, r)
|
|
})
|
|
}
|
|
|
|
func (c *Client) readPump(hub *Hub) {
|
|
defer func() {
|
|
hub.Unregister(c)
|
|
c.conn.Close()
|
|
}()
|
|
c.conn.SetReadLimit(512)
|
|
c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
|
|
c.conn.SetPongHandler(func(string) error {
|
|
c.conn.SetReadDeadline(time.Now().Add(60 * time.Second))
|
|
return nil
|
|
})
|
|
for {
|
|
_, _, err := c.conn.ReadMessage()
|
|
if err != nil {
|
|
break
|
|
}
|
|
}
|
|
}
|
|
|
|
func (c *Client) writePump(pingInterval time.Duration) {
|
|
ticker := time.NewTicker(pingInterval)
|
|
defer func() {
|
|
ticker.Stop()
|
|
c.conn.Close()
|
|
}()
|
|
for {
|
|
select {
|
|
case message, ok := <-c.send:
|
|
c.conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
|
|
if !ok {
|
|
c.conn.WriteMessage(websocket.CloseMessage, []byte{})
|
|
return
|
|
}
|
|
if err := c.conn.WriteMessage(websocket.TextMessage, message); err != nil {
|
|
return
|
|
}
|
|
case <-ticker.C:
|
|
c.conn.SetWriteDeadline(time.Now().Add(10 * time.Second))
|
|
if err := c.conn.WriteMessage(websocket.PingMessage, nil); err != nil {
|
|
return
|
|
}
|
|
// #1074: the ping above keeps proxies and this server's read
|
|
// deadline happy, but browser JS never sees ping frames. Without a
|
|
// frame the page receives, a quiet mesh and a dead socket look the
|
|
// same to the client, so it could not detect the latter.
|
|
if err := c.conn.WriteMessage(websocket.TextMessage, wsHeartbeat); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Poller watches for new transmissions in SQLite and broadcasts them.
|
|
type Poller struct {
|
|
db *DB
|
|
hub *Hub
|
|
store *PacketStore // optional: if set, new transmissions are ingested into memory
|
|
interval time.Duration
|
|
stop chan struct{}
|
|
}
|
|
|
|
func NewPoller(db *DB, hub *Hub, interval time.Duration) *Poller {
|
|
return &Poller{db: db, hub: hub, interval: interval, stop: make(chan struct{})}
|
|
}
|
|
|
|
// pollFailure logs a failing poll once per streak of failures, and once when
|
|
// it recovers. Owned by the Start goroutine.
|
|
type pollFailure struct {
|
|
what string
|
|
failing bool
|
|
}
|
|
|
|
func (f *pollFailure) note(err error, cursor int) {
|
|
switch {
|
|
case err != nil && !f.failing:
|
|
f.failing = true
|
|
log.Printf("[poller] %s poll failed, retrying from id %d each tick: %v", f.what, cursor, err)
|
|
case err == nil && f.failing:
|
|
f.failing = false
|
|
log.Printf("[poller] %s poll recovered at id %d", f.what, cursor)
|
|
}
|
|
}
|
|
|
|
func (p *Poller) Start() {
|
|
lastID, lastObsID := p.startCursors()
|
|
log.Printf("[poller] starting from transmission ID %d, obs ID %d, interval %v", lastID, lastObsID, p.interval)
|
|
|
|
txFail := pollFailure{what: "transmission"}
|
|
obsFail := pollFailure{what: "observation"}
|
|
ticker := time.NewTicker(p.interval)
|
|
defer ticker.Stop()
|
|
|
|
for {
|
|
select {
|
|
case <-ticker.C:
|
|
if p.store != nil {
|
|
// Radio history first (#1869): a retune or the radio id of a
|
|
// new observation must be known before it is broadcast.
|
|
refreshRadioHistory(p.db.conn, false)
|
|
// Ingest new transmissions into in-memory store and broadcast
|
|
// On error newMax is lastID: the same rows are read next tick.
|
|
newTxs, newMax, err := p.store.ingestNewFromDB(lastID, 100)
|
|
txFail.note(err, lastID)
|
|
if newMax > lastID {
|
|
lastID = newMax
|
|
}
|
|
// Ingest new observations for existing transmissions (fixes #174).
|
|
// The bound below is read BEFORE ingestNewObservations polls the
|
|
// advert feed, which waits on advertEvidenceMu while a startup
|
|
// chunk sits between its query and its merge. So an observation
|
|
// of a transmission that chunk has not merged yet is either read
|
|
// after the merge or lies above the bound, never skipped below it.
|
|
nextObsID := lastObsID
|
|
if err := p.db.conn.QueryRow(`
|
|
SELECT COALESCE(MAX(id), ?) FROM (
|
|
SELECT id FROM observations
|
|
WHERE id > ?
|
|
ORDER BY id ASC
|
|
LIMIT 500
|
|
)`, lastObsID, lastObsID).Scan(&nextObsID); err != nil {
|
|
nextObsID = lastObsID
|
|
}
|
|
newObs, err := p.store.ingestNewObservations(lastObsID, 500)
|
|
obsFail.note(err, lastObsID)
|
|
// A failed read saw none of these rows: keep the cursor.
|
|
if err == nil && nextObsID > lastObsID {
|
|
lastObsID = nextObsID
|
|
}
|
|
if len(newTxs) > 0 {
|
|
log.Printf("[broadcast] sending %d packets to %d clients (lastID now %d)", len(newTxs), p.hub.ClientCount(), lastID)
|
|
}
|
|
for _, tx := range newTxs {
|
|
p.hub.Broadcast(WSMessage{
|
|
Type: "packet",
|
|
Data: tx,
|
|
})
|
|
}
|
|
for _, obs := range newObs {
|
|
p.hub.Broadcast(WSMessage{
|
|
Type: "packet",
|
|
Data: obs,
|
|
})
|
|
}
|
|
} else {
|
|
// Fallback: direct DB query (used when store is nil, e.g. tests)
|
|
newTxs, err := p.db.GetNewTransmissionsSince(lastID, 100)
|
|
if err != nil {
|
|
log.Printf("[poller] error: %v", err)
|
|
continue
|
|
}
|
|
for _, tx := range newTxs {
|
|
id, _ := tx["id"].(int)
|
|
if id > lastID {
|
|
lastID = id
|
|
}
|
|
// Copy packet fields for the nested packet (avoids circular ref)
|
|
pkt := make(map[string]interface{}, len(tx))
|
|
for k, v := range tx {
|
|
pkt[k] = v
|
|
}
|
|
tx["packet"] = pkt
|
|
p.hub.Broadcast(WSMessage{
|
|
Type: "packet",
|
|
Data: tx,
|
|
})
|
|
}
|
|
}
|
|
case <-p.stop:
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
// startCursors picks the ids polling starts above. A fresh MAX(id) here would
|
|
// skip every row committed between the load's last snapshot and now: it is in
|
|
// neither the store nor any later poll. So with a store, polling starts at the
|
|
// high-water marks taken before the load. Rows between a mark and the end of
|
|
// the load are read again and dedupe: transmissions by hash, observations by
|
|
// byObsID or observer+path, and only rows new to the store are broadcast.
|
|
func (p *Poller) startCursors() (lastID, lastObsID int) {
|
|
if p.store == nil {
|
|
return p.db.GetMaxTransmissionID(), p.db.GetMaxObservationID()
|
|
}
|
|
txMark, obsMark, ok := p.store.StartupMarks()
|
|
if ok {
|
|
lastID, lastObsID = txMark, obsMark
|
|
} else {
|
|
// No marks: keep the old fresh-MAX start, floored by the store so a
|
|
// failed query (COALESCE 0) cannot replay the whole DB.
|
|
lastID, lastObsID = p.db.GetMaxTransmissionID(), p.db.GetMaxObservationID()
|
|
if storeMaxObs := p.store.MaxObservationID(); storeMaxObs > lastObsID {
|
|
lastObsID = storeMaxObs
|
|
}
|
|
}
|
|
// The loaded max transmission id is a safe floor: one writer commits ids
|
|
// in order, so every lower id was visible to the chunk that loaded it.
|
|
// The loaded max observation id is not: a chunk-1 transmission can gain
|
|
// an observation after chunk 1 whose id is below one loaded in chunk 2.
|
|
if storeMax := p.store.MaxTransmissionID(); storeMax > lastID {
|
|
lastID = storeMax
|
|
}
|
|
return lastID, lastObsID
|
|
}
|
|
|
|
func (p *Poller) Stop() {
|
|
close(p.stop)
|
|
}
|