## What this does
The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.
| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |
### Commit 1: the UI as static files, for any host
- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.
### Commit 2: the web container and the three modes
- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.
### Keeping new code on `CSApi`
- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
- which helper to use for each need;
- the lint rule;
- how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).
### Review follow-ups
- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.
### Audit fixes
- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.
### Audit notes, fixed
- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.
### Second review
- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.
## Upgrade notes
- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
- `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.
## Testing
- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
- The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
- Headers, checked with curl:
- `no-cache` on `/`, `/index.html` and `/deployment.js`;
- zstd on static assets;
- `/api/*` proxied (`Via: 1.1 Caddy`);
- a `101` WebSocket upgrade on `/ws`.
- In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
- there are no page errors and no horizontal overflow at 390 px;
- no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
- the session surviving a reload;
- a CSRF-protected `PATCH /api/account`;
- logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.
## Performance
There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.
## Notes
- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A Trivy scan of the published image found **19 HIGH and 20 MEDIUM** in
Go dependencies, plus a runtime base past its support window. There is
no Dependabot here, so this collects the bumps in one PR.
| dependency | from | to | why |
|---|---|---|---|
| `golang.org/x/net` (ingestor) | v0.27.0 | v0.61.0 | CVE-2024-45338 and
later advisories, fixed up to v0.60.0 |
| `golang.org/x/crypto` (server, `internal/users`) | v0.31.0 | v0.58.0 |
CVE-2025-22869, CVE-2025-47913 and later; `x/sys` follows to v0.49.0 |
| `github.com/eclipse/paho.mqtt.golang` | v1.5.0 | v1.5.1 |
CVE-2025-10543, an integer overflow in the UTF-8 string encoding |
| runtime image | `alpine:3.20` | `alpine:3.24` | 3.20 is out of
support; its musl 1.2.5-r3 carries CVE-2026-107891 |
The Alpine change covers `Dockerfile`, `Dockerfile.go`, the arm64 smoke
step in `deploy.yml` and the comment in
`docker/test-select-service-user.sh`.
**What changes besides the versions:**
- The new `x/` modules declare `go 1.26.0`, so the `go` directive of
`cmd/server`, `cmd/ingestor` and `internal/users` moves up from 1.22.
The builder image and CI already run 1.27.
- `x/sync` follows paho to v0.17.0.
**What is left:** only `GO-2026-5932`. It flags `x/crypto/openpgp` as
unmaintained; nothing here imports that package and no fix exists.
## Checks
- Every module passes `go mod tidy`, `go build` and `go vet`.
- `go test -race` passes in `internal/users`, `cmd/ingestor` and
`cmd/server`.
- A local image build on alpine 3.24.3:
- ships mosquitto 2.1.2, caddy v2.11.4 and supervisor 4.3.0
- keeps the caddy and mosquitto accounts at the same uid:gid (100:101,
101:102), so `test-select-service-user.sh` still describes the image
(116 assertions pass)
- comes up healthy, ingests an MQTT status through mosquitto and serves
the frontend
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #2125: the six non-blocking notes on its approval, with
note 3 as the one that changes behaviour.
**What changes**
- `.env` is now read last. The hand-over of the data directory moves
before it, and the only code that runs after the file is sourced reads
`prepare_service_user`'s positional parameters: the account, the
container's `RUN_AS_UID`/`RUN_AS_GID` (set or unset, and value) and the
file's path are parked in `$1..$6` before the `.`.
- The comment gives the real reason the account was already safe: the
`.` happens inside `load_env_file`, so a `set -- root` in `.env` lands
in that function's `$1`, not in `prepare_service_user`'s (note 1). It
also says that a trusted `.env` is root's code and this does not defend
against it (it can redefine functions or set `PATH`); what it cannot do
by assigning variables is change the account, where the hand-over goes,
or `RUN_AS_UID`/`RUN_AS_GID` as the services see them.
- `svc_log` uses `printf` (note 5); the entrypoint check also pins that
it does not assign `CORESCOPE_SERVICE_USER` (note 4); the test header is
reflowed (note 6); `TEST_SHELL` picks the inner shell so the suite runs
under dash, bash or busybox without editing.
**Why**
A sourced file shares the shell's variables: every global, the locals of
`load_env_file`, and, by dynamic scoping, the locals of its caller too
(checked under dash, bash 5.3 and busybox ash 1.37). So a trusted `.env`
that assigned `_uid_before` or `_uid_was` skipped the restore and left
`RUN_AS_UID=0` in the services' environment, as note 3 says. The same
class reached the hand-over: with `SERVICE_DATA_DIR=/` in `.env`, the
walk ran over `/`. Nothing but the entrypoint reads
`RUN_AS_UID`/`RUN_AS_GID`, so the effect was a wrong value in the
services' environment and a log line that said "ignoring" when it had
not.
**What I checked**
- `docker/test-select-service-user.sh`: 116 assertions, 0 failed, under
dash and bash. The new cases fail on master for the stated reasons and
nothing else does.
- New cases: `.env` running `set -- root` (note 1; this is the case that
catches sourcing the file inline, the names case alone does not); `.env`
assigning the restore's names, `_env_file`, `SERVICE_ENV_FILE` and
`RUN_AS_*`, with uid 1000 and gid 1001 so a swapped parameter shows;
`RUN_AS_GID` alone; `.env` pointing the three `SERVICE_*` paths
elsewhere; a logged path containing `\012`.
- Mutations, each applied to a copy and confirmed to make the suite
fail: hand-over moved back after `.env`; restore from `load_env_file`'s
locals (master); restore from `prepare_service_user`'s locals; file
sourced inline in `prepare_service_user` with the refusal log kept;
restore dropped; only the uid compared; gid restored from the uid's
parameter; set/unset flipped; the log reading `SERVICE_ENV_FILE` after
the `.`; the entrypoint reassigning the account. `svc_log` back to
`echo` fails under dash only, which is where note 5 applies.
- The image, built from this branch (linux/amd64), with a root-owned,
root-only-writable `.env` containing all of the above plus
`DISABLE_MOSQUITTO=true`, `DISABLE_CADDY=true` and `set -- root`: (A)
`RUN_AS_UID` unset: root, nothing re-owned, `DISABLE_*` honoured,
`RUN_AS_*` not in the services' environment. (B) `RUN_AS_UID=1000
RUN_AS_GID=1001` on A's data directory, root-owned `meshcore.db` inside:
services run as 1000, their environment has `RUN_AS_UID=1000
RUN_AS_GID=1001`, `meshcore.db` and the directory are 1000:1001, `.env`
stays root's, `/etc/passwd` stays root's. (C) `.env` owned by uid 1000:
refused with the reason, services run as 1000. The same image with
master's two scripts mounted over it, in B's shape: services'
environment `RUN_AS_UID=0 RUN_AS_GID=0`, `/etc/passwd` owned by 1000,
`.env` owned by 1000:1001.
- `sh test-all.sh` passes; no JS or Go change.
**Not checked**
- Windows: the suite stops at the symlink fixture, as before.
- busybox as `TEST_SHELL` on my machine: Ubuntu's busybox prefers its
own applets over `PATH`, so the stubs are bypassed and 28 cases fail, on
master and on this branch alike. Your WSL Debian run (103 passed) and
the image itself, which runs Alpine's ash with real binaries, are what
cover ash.
- Rootless Docker and `userns-remap`: unchanged, docs gap already noted.
Note 2: kept the names case as a second guard; the `set -- root` case is
the one that pins the design.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in
with `RUN_AS_UID`. A code-execution bug in either one then no longer
lands as root next to the mounted data and `config.json`. `supervisord`
stays root, for mosquitto and caddy.
| `RUN_AS_UID` | Go services run as |
|---|---|
| unset (default) | root, as before. Upgrades change nothing |
| set | `RUN_AS_UID` / `RUN_AS_GID` |
With `RUN_AS_UID` set:
- Each start hands the data directory to that uid (`chown -h`), except
`.env` and anything mounted into it on its own. Failures are logged.
- `.env` is read only if it is root-owned and only root can write it.
`RUN_AS_UID` and `RUN_AS_GID` are never taken from it.
- `/app` stays root's. **Depends on #2137.**
- A uid with no account in the image gets one at start; `caddy` and
`mosquitto` are reused. A uid that can't be provided fails startup
instead of falling back to root.
The logic is in `docker/select-service-user.sh`, tested by
`docker/test-select-service-user.sh`, which needs no Docker or root and
runs in CI. See also "Running the Go services unprivileged" in
`docs/deployment.md`.
Verified in the built image (master + this + #2137) across 14 cases.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
## Summary
Fixes#702 — `.env` file `DISABLE_MOSQUITTO`/`DISABLE_CADDY` ignored
when using `docker run`.
## Changes
### Entrypoint sources `/app/data/.env`
The entrypoint now sources `/app/data/.env` (if present) before the
`DISABLE_*` checks. This works regardless of how the container is
started — `docker run`, compose, or `manage.sh`.
```bash
if [ -f /app/data/.env ]; then
set -a
. /app/data/.env
set +a
fi
```
### `DISABLE_CADDY` added to compose files
Both `docker-compose.yml` and `docker-compose.staging.yml` now forward
`DISABLE_CADDY` to the container environment (was missing — only
`DISABLE_MOSQUITTO` was wired).
### Deployment docs updated
- `docs/deployment.md`: bare `docker run` is now the primary/recommended
approach with a full parameter reference table
- Documents the `/app/data/.env` convenience feature
- Compose and `manage.sh` marked as legacy alternatives
- `DISABLE_CADDY` added to the environment variable reference
### README quick start updated
Shows the full `docker run` command with `--restart`, ports, and
volumes. Includes HTTPS variant. Documents `-e` flags and `.env` file.
### v3.5.0 release notes
Updated the env var documentation to mention the `.env` file support.
## Testing
- All Go server tests pass
- All Go ingestor tests pass
- No logic changes to Go code — entrypoint shell script + docs only
---------
Co-authored-by: you <you@example.com>
## Summary
Implements the `DISABLE_CADDY` environment variable in the Docker
entrypoint, fixing #629.
## Problem
The `DISABLE_CADDY` env var was documented but had no effect — the
entrypoint only handled `DISABLE_MOSQUITTO`.
## Changes
### New supervisord configs
- **`supervisord-go-no-caddy.conf`** — mosquitto + ingestor + server (no
Caddy)
- **`supervisord-go-no-mosquitto-no-caddy.conf`** — ingestor + server
only
### Updated entrypoint (`docker/entrypoint-go.sh`)
Handles all 4 combinations:
| DISABLE_MOSQUITTO | DISABLE_CADDY | Config used |
|---|---|---|
| false | false | `supervisord.conf` (default) |
| true | false | `supervisord-no-mosquitto.conf` |
| false | true | `supervisord-no-caddy.conf` |
| true | true | `supervisord-no-mosquitto-no-caddy.conf` |
### Dockerfiles
Added COPY lines for the new configs in both `Dockerfile` and
`Dockerfile.go`.
## Testing
```bash
# Verify correct config selection
docker run -e DISABLE_CADDY=true corescope
# Should log: [config] Caddy reverse proxy disabled (DISABLE_CADDY=true)
docker run -e DISABLE_CADDY=true -e DISABLE_MOSQUITTO=true corescope
# Should log both disabled messages
```
Fixes#629
Co-authored-by: you <you@example.com>
## Summary
Fixes#450 — staging deployment flaky due to container not shutting down
cleanly.
## Root Causes
1. **Server never closed DB on shutdown** — SQLite WAL lock held
indefinitely, blocking new container startup
2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills
connections instead of draining them
3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM
then immediately SIGKILL (default 10s is often not enough for WAL
checkpoint)
4. **Supervisor didn't forward SIGTERM** — missing
`stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of
graceful shutdown
5. **Deploy scripts used default `docker stop` timeout** — only 10s
grace period
## Changes
### Go Server (`cmd/server/`)
- **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s
context timeout — drains in-flight requests before closing
- **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway`
frames to all connected clients
- **DB close on shutdown**: Explicitly closes DB after HTTP server stops
(was never closed before)
- **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close
— flushes WAL to main DB file and removes WAL/SHM lock files
### Go Ingestor (`cmd/ingestor/`)
- **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method,
called before `Close()`
- **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight
messages to drain
### Docker Compose (all 4 variants)
- Added `stop_grace_period: 30s` and `stop_signal: SIGTERM`
### Supervisor Configs (both variants)
- Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor
programs
### Deploy Scripts
- `deploy-staging.sh`: `docker stop -t 30` with explicit grace period
- `deploy-live.sh`: `docker stop -t 30` with explicit grace period
## Shutdown Sequence (after fix)
1. Docker sends SIGTERM to supervisord (PID 1)
2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s
each)
3. Server: stops poller → drains HTTP (15s) → closes WS clients →
checkpoints WAL → closes DB
4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL →
closes DB
5. Docker waits up to 30s total before SIGKILL
## Tests
All existing tests pass:
- `cd cmd/server && go test ./...` ✅
- `cd cmd/ingestor && go test ./...` ✅
---------
Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
## Summary
- add `DISABLE_MOSQUITTO` support in container startup by switching
supervisord config when disabled
- add a no-mosquitto supervisord config
(`docker/supervisord-go-no-mosquitto.conf`)
- fix Compose port mapping regression so host ports map to fixed
internal listener ports (`80`, `443`, `1883`)
- add compose variants without MQTT port publishing
(`docker-compose.no-mosquitto.yml`,
`docker-compose.staging.no-mosquitto.yml`)
- update `manage.sh` setup flow to ask `Use built-in MQTT broker?
[Y/n]`, skip MQTT port prompt when disabled, persist
`DISABLE_MOSQUITTO`, and use no-mosquitto compose files when
starting/stopping/restarting
- align `.env.example` staging keys with compose
(`STAGING_GO_HTTP_PORT`, `STAGING_GO_MQTT_PORT`)
- fix staging Caddyfile generation to use `STAGING_GO_HTTP_PORT`
- fix `.env.example` staging default comments to match actual values
(82/1885)
## Validation performed
- ✅ `bash -n manage.sh` passes.
- ✅ With `DISABLE_MOSQUITTO=true`, no-mosquitto compose overrides are
selected, Mosquitto is not started, and MQTT port is not published.
- ✅ With `DISABLE_MOSQUITTO=false`, standard compose files are used,
Mosquitto starts, and MQTT port mapping is present.
- ℹ️ Runtime Docker validation requires a running Docker host.
Fixes#267
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removes the separate config.json file bind mount from both compose
files. The data directory mount already covers it, and the Go server
searches /app/data/config.json via LoadConfig.
- Entrypoint symlinks /app/data/config.json for ingestor compatibility
- manage.sh setup creates config in data dir, prompts admin if missing
- manage.sh start checks config exists before starting, offers to create
- deploy.yml simplified — no more sudo rm or directory cleanup
- Backup/restore updated to use data dir path
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removes the separate config.json file bind mount from both compose
files. The data directory mount already covers it, and the Go server
searches /app/data/config.json via LoadConfig.
- Entrypoint symlinks /app/data/config.json for ingestor compatibility
- manage.sh setup creates config in data dir, prompts admin if missing
- manage.sh start checks config exists before starting, offers to create
- deploy.yml simplified — no more sudo rm or directory cleanup
- Backup/restore updated to use data dir path
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root causes from CI logs:
1. 'read /app/config.json: is a directory' — Docker creates a directory
when bind-mounting a non-existent file. The entrypoint now detects
and removes directory config.json before falling back to example.
2. 'unable to open database file: out of memory (14)' — old container
(3GB) not fully exited when new one starts. Deploy now uses
'docker compose down' with timeout and waits for memory reclaim.
3. Supervisor gave up after 3 fast retries (FATAL in ~6s). Increased
startretries to 10 and startsecs to 2 for server and ingestor.
Additional:
- Deploy step ensures staging config.json exists before starting
- Healthcheck: added start_period=60s, increased timeout and retries
- No longer uses manage.sh (CI working dir != repo checkout dir)
- Multi-stage Dockerfile builds Go server + ingestor (pure Go SQLite, no CGO)
- supervisord-go.conf runs meshcore-server + meshcore-ingestor + mosquitto + caddy
- staging-go compose service on port 81/1884 with staging-go profile
- Identical volume/config structure to Node.js deployment
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Milestone 1 of #132. Adds docker-compose.yml with prod + staging services,
.env.example for port/data configuration, and Caddyfile.staging for HTTP-only
staging proxy. No changes to Dockerfile or server.js — same image, different
config.
Fixes#132 (partially)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- entrypoint copies example config to /app/data/config.json on first run
- symlinks /app/config.json → /app/data/config.json so app code unchanged
- theme.json also symlinked from /app/data/ if present
- config persists across container rebuilds without extra bind mounts
- updated README with new config/theme instructions