fix: staging deploy failures — OOM + config.json directory mount

Root causes from CI logs:
1. 'read /app/config.json: is a directory' — Docker creates a directory
   when bind-mounting a non-existent file. The entrypoint now detects
   and removes directory config.json before falling back to example.
2. 'unable to open database file: out of memory (14)' — old container
   (3GB) not fully exited when new one starts. Deploy now uses
   'docker compose down' with timeout and waits for memory reclaim.
3. Supervisor gave up after 3 fast retries (FATAL in ~6s). Increased
   startretries to 10 and startsecs to 2 for server and ingestor.

Additional:
- Deploy step ensures staging config.json exists before starting
- Healthcheck: added start_period=60s, increased timeout and retries
- No longer uses manage.sh (CI working dir != repo checkout dir)
This commit is contained in:
you
2026-03-29 23:16:46 +00:00
parent 5bf2cdd812
commit 3dd68d4418
4 changed files with 37 additions and 3 deletions
+23 -1
View File
@@ -262,7 +262,29 @@ jobs:
uses: actions/checkout@v5
- name: Deploy staging
run: ./manage.sh restart staging
run: |
# Use docker compose down (not just stop/rm) to properly clean up
# the old container, network, and release memory before starting new one
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging down --timeout 30 2>/dev/null || true
# Wait for container to be fully gone and OS to reclaim memory (3GB limit)
for i in $(seq 1 15); do
if ! docker ps -a --format '{{.Names}}' | grep -q 'corescope-staging-go'; then
break
fi
sleep 1
done
sleep 5 # extra pause for OS memory reclaim
# Ensure staging config exists (docker creates a directory if bind mount source missing)
STAGING_DATA="${STAGING_DATA_DIR:-$HOME/meshcore-staging-data}"
if [ ! -f "$STAGING_DATA/config.json" ]; then
echo "Staging config missing — copying from repo config.example.json"
mkdir -p "$STAGING_DATA"
cp config.example.json "$STAGING_DATA/config.json"
fi
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging up -d staging-go
- name: Healthcheck staging container
run: |
+3 -2
View File
@@ -34,8 +34,9 @@ services:
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
timeout: 5s
retries: 3
timeout: 10s
retries: 5
start_period: 60s
volumes:
# Named volume for Caddy TLS certificates (not user data — managed by Caddy internally)
+7
View File
@@ -1,5 +1,12 @@
#!/bin/sh
# Fix: Docker creates a directory when bind-mounting a non-existent file.
# If config.json is a directory (from a failed mount), remove it and use the example.
if [ -d /app/config.json ]; then
echo "[entrypoint] WARNING: config.json is a directory (broken bind mount) — removing and using example"
rm -rf /app/config.json
fi
# Copy example config if no config.json exists (not bind-mounted)
if [ ! -f /app/config.json ]; then
echo "[entrypoint] No config.json found, copying from config.example.json"
+4
View File
@@ -19,6 +19,8 @@ command=/app/corescope-ingestor -config /app/config.json
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
@@ -29,6 +31,8 @@ command=/app/corescope-server -config-dir /app -db /app/data/meshcore.db -public
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr