Commit Graph

19 Commits

Author SHA1 Message Date
Kpa-clawbot
26c47df814 fix: entrypoint .env support + deployment docs for bare docker run (#704)
## Summary

Fixes #702 — `.env` file `DISABLE_MOSQUITTO`/`DISABLE_CADDY` ignored
when using `docker run`.

## Changes

### Entrypoint sources `/app/data/.env`
The entrypoint now sources `/app/data/.env` (if present) before the
`DISABLE_*` checks. This works regardless of how the container is
started — `docker run`, compose, or `manage.sh`.

```bash
if [ -f /app/data/.env ]; then
  set -a
  . /app/data/.env
  set +a
fi
```

### `DISABLE_CADDY` added to compose files
Both `docker-compose.yml` and `docker-compose.staging.yml` now forward
`DISABLE_CADDY` to the container environment (was missing — only
`DISABLE_MOSQUITTO` was wired).

### Deployment docs updated
- `docs/deployment.md`: bare `docker run` is now the primary/recommended
approach with a full parameter reference table
- Documents the `/app/data/.env` convenience feature
- Compose and `manage.sh` marked as legacy alternatives
- `DISABLE_CADDY` added to the environment variable reference

### README quick start updated
Shows the full `docker run` command with `--restart`, ports, and
volumes. Includes HTTPS variant. Documents `-e` flags and `.env` file.

### v3.5.0 release notes
Updated the env var documentation to mention the `.env` file support.

## Testing
- All Go server tests pass
- All Go ingestor tests pass
- No logic changes to Go code — entrypoint shell script + docs only

---------

Co-authored-by: you <you@example.com>
2026-04-11 20:43:16 -07:00
Kpa-clawbot
cae14da05e fix: implement DISABLE_CADDY env var in Docker entrypoint (#629) (#637)
## Summary

Implements the `DISABLE_CADDY` environment variable in the Docker
entrypoint, fixing #629.

## Problem

The `DISABLE_CADDY` env var was documented but had no effect — the
entrypoint only handled `DISABLE_MOSQUITTO`.

## Changes

### New supervisord configs
- **`supervisord-go-no-caddy.conf`** — mosquitto + ingestor + server (no
Caddy)
- **`supervisord-go-no-mosquitto-no-caddy.conf`** — ingestor + server
only

### Updated entrypoint (`docker/entrypoint-go.sh`)
Handles all 4 combinations:
| DISABLE_MOSQUITTO | DISABLE_CADDY | Config used |
|---|---|---|
| false | false | `supervisord.conf` (default) |
| true | false | `supervisord-no-mosquitto.conf` |
| false | true | `supervisord-no-caddy.conf` |
| true | true | `supervisord-no-mosquitto-no-caddy.conf` |

### Dockerfiles
Added COPY lines for the new configs in both `Dockerfile` and
`Dockerfile.go`.

## Testing

```bash
# Verify correct config selection
docker run -e DISABLE_CADDY=true corescope
# Should log: [config] Caddy reverse proxy disabled (DISABLE_CADDY=true)

docker run -e DISABLE_CADDY=true -e DISABLE_MOSQUITTO=true corescope
# Should log both disabled messages
```

Fixes #629

Co-authored-by: you <you@example.com>
2026-04-05 15:26:40 -07:00
Kpa-clawbot
f87eb3601c fix: graceful container shutdown for reliable deployments (#453)
## Summary

Fixes #450 — staging deployment flaky due to container not shutting down
cleanly.

## Root Causes

1. **Server never closed DB on shutdown** — SQLite WAL lock held
indefinitely, blocking new container startup
2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills
connections instead of draining them
3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM
then immediately SIGKILL (default 10s is often not enough for WAL
checkpoint)
4. **Supervisor didn't forward SIGTERM** — missing
`stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of
graceful shutdown
5. **Deploy scripts used default `docker stop` timeout** — only 10s
grace period

## Changes

### Go Server (`cmd/server/`)
- **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s
context timeout — drains in-flight requests before closing
- **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway`
frames to all connected clients
- **DB close on shutdown**: Explicitly closes DB after HTTP server stops
(was never closed before)
- **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close
— flushes WAL to main DB file and removes WAL/SHM lock files

### Go Ingestor (`cmd/ingestor/`)
- **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method,
called before `Close()`
- **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight
messages to drain

### Docker Compose (all 4 variants)
- Added `stop_grace_period: 30s` and `stop_signal: SIGTERM`

### Supervisor Configs (both variants)
- Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor
programs

### Deploy Scripts
- `deploy-staging.sh`: `docker stop -t 30` with explicit grace period
- `deploy-live.sh`: `docker stop -t 30` with explicit grace period

## Shutdown Sequence (after fix)

1. Docker sends SIGTERM to supervisord (PID 1)
2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s
each)
3. Server: stops poller → drains HTTP (15s) → closes WS clients →
checkpoints WAL → closes DB
4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL →
closes DB
5. Docker waits up to 30s total before SIGKILL

## Tests

All existing tests pass:
- `cd cmd/server && go test ./...` 
- `cd cmd/ingestor && go test ./...` 

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
2026-04-01 12:19:20 -07:00
Kpa-clawbot
5aa4fbb600 chore: normalize all files to LF line endings 2026-03-30 22:52:46 -07:00
Kpa-clawbot
6922d63b1c Add DISABLE_MOSQUITTO support for external brokers (#309)
## Summary
- add `DISABLE_MOSQUITTO` support in container startup by switching
supervisord config when disabled
- add a no-mosquitto supervisord config
(`docker/supervisord-go-no-mosquitto.conf`)
- fix Compose port mapping regression so host ports map to fixed
internal listener ports (`80`, `443`, `1883`)
- add compose variants without MQTT port publishing
(`docker-compose.no-mosquitto.yml`,
`docker-compose.staging.no-mosquitto.yml`)
- update `manage.sh` setup flow to ask `Use built-in MQTT broker?
[Y/n]`, skip MQTT port prompt when disabled, persist
`DISABLE_MOSQUITTO`, and use no-mosquitto compose files when
starting/stopping/restarting
- align `.env.example` staging keys with compose
(`STAGING_GO_HTTP_PORT`, `STAGING_GO_MQTT_PORT`)
- fix staging Caddyfile generation to use `STAGING_GO_HTTP_PORT`
- fix `.env.example` staging default comments to match actual values
(82/1885)

## Validation performed
-  `bash -n manage.sh` passes.
-  With `DISABLE_MOSQUITTO=true`, no-mosquitto compose overrides are
selected, Mosquitto is not started, and MQTT port is not published.
-  With `DISABLE_MOSQUITTO=false`, standard compose files are used,
Mosquitto starts, and MQTT port mapping is present.
- ℹ️ Runtime Docker validation requires a running Docker host.

Fixes #267

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 21:36:29 -07:00
Kpa-clawbot
16a99159cc fix: config.json lives in data dir, not bind-mounted as file (#282)
Removes the separate config.json file bind mount from both compose
files. The data directory mount already covers it, and the Go server
searches /app/data/config.json via LoadConfig.

- Entrypoint symlinks /app/data/config.json for ingestor compatibility
- manage.sh setup creates config in data dir, prompts admin if missing
- manage.sh start checks config exists before starting, offers to create
- deploy.yml simplified — no more sudo rm or directory cleanup
- Backup/restore updated to use data dir path

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 18:55:44 +00:00
Kpa-clawbot
61ff72fc80 Revert "fix: config.json lives in data dir, not bind-mounted as file"
This reverts commit 57ebd76070.
2026-03-30 10:00:17 -07:00
Kpa-clawbot
57ebd76070 fix: config.json lives in data dir, not bind-mounted as file
Removes the separate config.json file bind mount from both compose
files. The data directory mount already covers it, and the Go server
searches /app/data/config.json via LoadConfig.

- Entrypoint symlinks /app/data/config.json for ingestor compatibility
- manage.sh setup creates config in data dir, prompts admin if missing
- manage.sh start checks config exists before starting, offers to create
- deploy.yml simplified — no more sudo rm or directory cleanup
- Backup/restore updated to use data dir path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 09:58:22 -07:00
you
3dd68d4418 fix: staging deploy failures — OOM + config.json directory mount
Root causes from CI logs:
1. 'read /app/config.json: is a directory' — Docker creates a directory
   when bind-mounting a non-existent file. The entrypoint now detects
   and removes directory config.json before falling back to example.
2. 'unable to open database file: out of memory (14)' — old container
   (3GB) not fully exited when new one starts. Deploy now uses
   'docker compose down' with timeout and waits for memory reclaim.
3. Supervisor gave up after 3 fast retries (FATAL in ~6s). Increased
   startretries to 10 and startsecs to 2 for server and ingestor.

Additional:
- Deploy step ensures staging config.json exists before starting
- Healthcheck: added start_period=60s, increased timeout and retries
- No longer uses manage.sh (CI working dir != repo checkout dir)
2026-03-29 23:16:46 +00:00
Kpa-clawbot
f438411a27 chore: remove deprecated Node.js backend (-11,291 lines) (#265)
## Summary

Removes all deprecated Node.js backend server code. The Go server
(`cmd/server/`) has been the production backend — the Node.js server was
kept "just in case" but is no longer needed.

### Removed (19 files, -11,291 lines)

**Backend server (6 files):**
`server.js`, `db.js`, `decoder.js`, `server-helpers.js`,
`packet-store.js`, `iata-coords.js`

**Backend tests (9 files):**
`test-decoder.js`, `test-decoder-spec.js`, `test-server-helpers.js`,
`test-server-routes.js`, `test-packet-store.js`, `test-db.js`,
`test-db-migration.js`, `test-regional-filter.js`,
`test-regional-integration.js`

**Backend tooling (4 files):**
`tools/e2e-test.js`, `tools/frontend-test.js`, `benchmark.js`,
`benchmark-ab.sh`

### Updated
- `AGENTS.md` — Rewritten architecture section for Go, explicit
deprecation warnings
- `test-all.sh` — Only runs frontend tests
- `package.json` — Updated test:unit
- `scripts/validate.sh` — Removed Node.js server syntax check
- `docker/supervisord.conf` — Points to Go binary

### NOT touched
- `public/` (active frontend) 
- `test-e2e-playwright.js` (frontend E2E tests) 
- Frontend test files (`test-packet-filter.js`, `test-aging.js`,
`test-frontend-helpers.js`) 
- `package.json` / Playwright deps 

### Follow-up
- Server-only npm deps (express, better-sqlite3, mqtt, ws, supertest)
can be cleaned from package.json separately
- `Dockerfile.node` can be removed separately

---------

Co-authored-by: you <you@example.com>
2026-03-29 15:53:51 -07:00
Kpa-clawbot
cdcaa476f2 rename: MeshCore Analyzer → CoreScope (Phase 1 — backend + infra)
Rename product branding, binary names, Docker images, container names,
Go modules, proto go_package, CI, manage.sh, and documentation.

Preserved (backward compat):
- meshcore.db database filename
- meshcore-data / meshcore-staging-data directory paths
- MQTT topics (meshcore/#, meshcore/+/+/packets, etc.)
- proto package namespace (meshcore.v1)
- localStorage keys

Changes by category:
- Go modules: github.com/corescope/{server,ingestor}
- Binaries: corescope-server, corescope-ingestor
- Docker images: corescope:latest, corescope-go:latest
- Containers: corescope-prod, corescope-staging, corescope-staging-go
- Supervisord programs: corescope, corescope-server, corescope-ingestor
- Branding: siteName, heroTitle, startup logs, fallback HTML
- Proto go_package: github.com/corescope/proto/v1
- CI: container refs, deploy path
- Docs: 8 markdown files updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-28 14:08:15 -07:00
Kpa-clawbot
7f45e807d9 fix: convert Go Docker scripts to LF line endings
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 01:19:46 -07:00
Kpa-clawbot
9737374c40 Add Go staging deployment: Dockerfile.go, supervisord-go, compose staging-go service
- Multi-stage Dockerfile builds Go server + ingestor (pure Go SQLite, no CGO)
- supervisord-go.conf runs meshcore-server + meshcore-ingestor + mosquitto + caddy
- staging-go compose service on port 81/1884 with staging-go profile
- Identical volume/config structure to Node.js deployment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-27 01:15:25 -07:00
Kpa-clawbot
a69d00c423 feat: add Docker Compose for prod/staging environments
Milestone 1 of #132. Adds docker-compose.yml with prod + staging services,
.env.example for port/data configuration, and Caddyfile.staging for HTTP-only
staging proxy. No changes to Dockerfile or server.js — same image, different
config.

Fixes #132 (partially)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-26 20:51:23 -07:00
you
4a5cd9fef4 fix: config.json load order — bind-mount first, data/ fallback
Broke MQTT by reading example config from data/ instead of the real
bind-mounted /app/config.json. Now checks app dir first.
2026-03-23 05:03:03 +00:00
you
16d6523b91 fix: config.json lives in /app/data/ volume, not baked into image
- entrypoint copies example config to /app/data/config.json on first run
- symlinks /app/config.json → /app/data/config.json so app code unchanged
- theme.json also symlinked from /app/data/ if present
- config persists across container rebuilds without extra bind mounts
- updated README with new config/theme instructions
2026-03-23 04:43:04 +00:00
you
a756517647 fix: set XDG_DATA_HOME so Caddy persists certs to mounted volume 2026-03-20 07:13:52 +00:00
you
fea8a7e0b5 feat: add Caddy to Docker container — automatic HTTPS
- Caddy reverse proxies :80/:443 → Node :3000
- Mount custom Caddyfile for your domain → auto Let's Encrypt TLS
- Caddy certs persisted in /data/caddy volume
- Ports: 80 (HTTP), 443 (HTTPS), 1883 (MQTT)
2026-03-20 06:07:38 +00:00
you
2e486e2a66 feat: Docker packaging — single container with Mosquitto + Node
- Dockerfile: Alpine + Node 22 + Mosquitto + supervisord
- Auto-copies config.example.json if no config.json mounted
- Named volume for data persistence (SQLite + Mosquitto)
- Ports: 3000 (web), 1883 (MQTT)
- .dockerignore excludes data, config, git, benchmarks
- README updated with Docker quickstart
2026-03-20 06:06:15 +00:00