mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-06-05 17:01:30 +00:00
Kpa-clawbot
f87eb3601c
## Summary Fixes #450 — staging deployment flaky due to container not shutting down cleanly. ## Root Causes 1. **Server never closed DB on shutdown** — SQLite WAL lock held indefinitely, blocking new container startup 2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills connections instead of draining them 3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM then immediately SIGKILL (default 10s is often not enough for WAL checkpoint) 4. **Supervisor didn't forward SIGTERM** — missing `stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of graceful shutdown 5. **Deploy scripts used default `docker stop` timeout** — only 10s grace period ## Changes ### Go Server (`cmd/server/`) - **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s context timeout — drains in-flight requests before closing - **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway` frames to all connected clients - **DB close on shutdown**: Explicitly closes DB after HTTP server stops (was never closed before) - **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close — flushes WAL to main DB file and removes WAL/SHM lock files ### Go Ingestor (`cmd/ingestor/`) - **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method, called before `Close()` - **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight messages to drain ### Docker Compose (all 4 variants) - Added `stop_grace_period: 30s` and `stop_signal: SIGTERM` ### Supervisor Configs (both variants) - Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor programs ### Deploy Scripts - `deploy-staging.sh`: `docker stop -t 30` with explicit grace period - `deploy-live.sh`: `docker stop -t 30` with explicit grace period ## Shutdown Sequence (after fix) 1. Docker sends SIGTERM to supervisord (PID 1) 2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s each) 3. Server: stops poller → drains HTTP (15s) → closes WS clients → checkpoints WAL → closes DB 4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL → closes DB 5. Docker waits up to 30s total before SIGKILL ## Tests All existing tests pass: - `cd cmd/server && go test ./...` ✅ - `cd cmd/ingestor && go test ./...` ✅ --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
54 lines
1.2 KiB
Plaintext
54 lines
1.2 KiB
Plaintext
[supervisord]
|
|
nodaemon=true
|
|
user=root
|
|
logfile=/dev/stdout
|
|
logfile_maxbytes=0
|
|
pidfile=/var/run/supervisord.pid
|
|
|
|
[program:mosquitto]
|
|
command=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
|
|
autostart=true
|
|
autorestart=true
|
|
stdout_logfile=/dev/stdout
|
|
stdout_logfile_maxbytes=0
|
|
stderr_logfile=/dev/stderr
|
|
stderr_logfile_maxbytes=0
|
|
|
|
[program:corescope-ingestor]
|
|
command=/app/corescope-ingestor -config /app/config.json
|
|
directory=/app
|
|
autostart=true
|
|
autorestart=true
|
|
startretries=10
|
|
startsecs=2
|
|
stopsignal=TERM
|
|
stopwaitsecs=20
|
|
stdout_logfile=/dev/stdout
|
|
stdout_logfile_maxbytes=0
|
|
stderr_logfile=/dev/stderr
|
|
stderr_logfile_maxbytes=0
|
|
|
|
[program:corescope-server]
|
|
command=/app/corescope-server -config-dir /app -db /app/data/meshcore.db -public /app/public -port 3000
|
|
directory=/app
|
|
autostart=true
|
|
autorestart=true
|
|
startretries=10
|
|
startsecs=2
|
|
stopsignal=TERM
|
|
stopwaitsecs=20
|
|
stdout_logfile=/dev/stdout
|
|
stdout_logfile_maxbytes=0
|
|
stderr_logfile=/dev/stderr
|
|
stderr_logfile_maxbytes=0
|
|
|
|
[program:caddy]
|
|
command=/usr/sbin/caddy run --config /etc/caddy/Caddyfile
|
|
environment=XDG_DATA_HOME="/data"
|
|
autostart=true
|
|
autorestart=true
|
|
stdout_logfile=/dev/stdout
|
|
stdout_logfile_maxbytes=0
|
|
stderr_logfile=/dev/stderr
|
|
stderr_logfile_maxbytes=0
|