From e7fd48a0543ea4427a53b3f613ba32ce2a871b4e Mon Sep 17 00:00:00 2001 From: Paul Picazo Date: Mon, 16 Nov 2026 05:56:09 -0800 Subject: [PATCH] feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What this does The frontend half of serving the UI apart from the server. The server half was #2362. It gives CoreScope three deployment modes. The server container is the same image in all three, and it still serves everything, the classic UI included. | Mode | The UI is served by | API | Server CORS setting | Logins | |---|---|---|---|---| | All-in-one container (today) | the server container | same origin | none | yes | | Split containers (new) | a web container that forwards the API | same origin | none | yes | | Static host or CDN (new) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | ### Commit 1: the UI as static files, for any host - **One place for API URLs:** `public/deployment.js` says where the API is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket URL. A lint rule keeps raw `/api` fetches out of the other files. The embedded UI keeps `apiBase: ''`, so it behaves as before, except that its WebSocket now uses `/ws` instead of `/`. - **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`, which is `public/` byte for byte except for the cache buster in `index.html` and `deployment.js`. - It is `dist/web` rather than `dist/` because the Go binaries are built into `dist/` and `make clean` removes that folder. - `API_BASE=same-origin` is for a host that forwards `/api` and `/ws` itself, as the web container does. - **Anonymous build (`AUTH=false`):** for another origin. It has no account features: login, account, admin and other writes link to the same page on the classic UI. It sends no cookies, and its three JSON POSTs go as `text/plain`, so there is no preflight. - **Error notices:** a blocked API answer or an HTML answer each show a notice. - **Channel badge:** `beta`, `next` and `dev` builds show a badge that opens a pre-filled issue, plus an "Open in classic" link. - **Runbook:** `deploy/static/README.md` covers Cloudflare Pages, previews, server CORS and rollback. ### Commit 2: the web container and the three modes - **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It packages the UI with `build.sh` when the image is built, serves it, and forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`). `SITE_ADDRESS` set to a domain name turns on automatic HTTPS. - **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules as `deploy/static/_headers`, and compresses responses with zstd or gzip. - **`docker-compose.split.yml`:** the stock server image (its own Caddy off) plus the web container: `CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f docker-compose.split.yml up -d --build`. - `CORESCOPE_IMAGE` is required and must match the checkout's version, so a release server is never paired with a newer UI by default. - The server's built-in broker, which accepts anonymous clients, is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - **Docs:** `docs/deployment.md` has a new "Deployment modes" section, and `DEPLOY.md` points to it. ### Keeping new code on `CSApi` - **Lint rule:** `.eslintrc.json` now rejects every direct network call in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`, `sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in `href`, `src` and `action` (assigned, set with `setAttribute` or written into markup), `window.open`, `location.assign`, `location.replace` and `location =`. `api-base.js` is the one file exempt. - **HTML pages:** `eslint public/*.js` never sees the inline scripts and markup of `public/*.html`, so `test-lint-network-calls.js` applies the same rule to them, at their page lines. `area-map.html`'s two fetches carry an `eslint-disable` with the reason: its Server field can name another server. - **Why it changed:** the earlier rule only caught a literal `/api…` passed straight to `fetch`. - **Check:** CI already fails on lint errors, and `tests/e2e/test-lint-network-calls.js` pins exactly what the rule reports: 25 bad statements reported, 15 good ones not, `api-base.js` exempt. - **AGENTS.md:** a new "API calls" section covers: - which helper to use for each need; - the lint rule; - how anonymous builds gate login-only features; - loading `api-base.js` in vm and jsdom tests (the region quick-picks suite missed that, and failed shard 1 on the first CI run). ### Review follow-ups - **Account tokens:** the channel badge's pre-filled issue drops the query on `#/account/…` routes, so mail-link tokens never reach a GitHub URL. "Open in classic" keeps the full route, since the classic UI needs the token. - **HTML answers:** on a packaged UI (static host, web container), `api()` rejects an HTML answer before status and retry handling. An HTML 404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server returned an unexpected response" after one request. It used to give a generic error, or six "warming up" retries for a 503. JSON errors and the server's JSON 503 warm-up behave as before. The embedded UI keeps its old handling (`CSApi.embedded`). - **Logins:** `docs/deployment.md`, `Dockerfile.web` and `docker-compose.split.yml` say `userManagement.publicBaseUrl` must be the web container's public origin. ### Audit fixes - **No broken logins build:** `build.sh` refuses `AUTH=true` with a cross-origin `API_BASE`. The server allows no credentialed CORS, so every API request of such a build was blocked; in Chromium all 12 were, and the pages stayed empty. - **No accidental deletes:** `build.sh` refuses an `OUT` that is neither empty nor a previous build. It used to wipe whatever folder or file `OUT` named. - **Lint:** the rule now also reports `/api/` paths in `href`, `src` and `action` attributes in single-quoted strings and template literals. `tests/e2e/test-lint-network-calls.js` pins exactly what it reports. - **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published on localhost only. - **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since v3.16.0 and v3.15.0 predate the static build. It also names the server version the static build needs: before #2362 the warm-up headers stay hidden and wildcard previews are refused. ### Audit notes, fixed - **Cache busting on every page:** the tool pages (`area-map.html`, `geofilter-builder.html`) load their scripts with `?v=__BUST__`. The server and `build.sh` replace it in every top-level page, and `_headers` and the Caddyfile make every page revalidate. Behind Caddy, a returning visitor could otherwise get a new tool page with an old `api-base.js`. - **HTTP/3:** the split compose publishes 443 over UDP too, for the HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`. - **WebSocket on `/`:** a Go test pins that an upgrade on `/` still works, for UIs cached from before `/ws`. ### Second review - **Trusted proxies:** the docs said to trust the compose subnet, and also to publish port 3000 if wanted. Together they let a client spoof its address: the subnet holds the gateway, where a client reaching a published 3000 through Docker's proxy arrives from. The compose file now fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web container's address (`172.30.83.200`), and the docs say to trust that address alone. - **Classic UI unchanged:** the HTML-answer notice above applies to packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as before, and an unknown `/api` path fails to parse as before, with no notice. - **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md says what the rule cannot see: a URL assembled elsewhere and passed in a variable. - **AGENTS.md:** `build.sh` is described as packaging (a copy, `deployment.js` and `__BUST__`, never a code transform), and rule 3 says `__BUST__` is filled in every top-level page. - **CI builds the web image:** the image-check job (no publish) builds `Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`, the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It also validates the split compose file. - **Cross-origin logins:** a hand-edited `deployment.js` with `auth: true` and another origin now logs a warning naming the setting, instead of failing with CORS errors alone. ## Upgrade notes - **WebSocket path:** the UI's live feed connects to `/ws` instead of `/`. The server has answered `/ws` since #1793 and still upgrades `/`, so browsers holding the previous UI keep working. - A reverse proxy that forwards WebSocket upgrades only for an exact `/` must forward `/ws` too. - The nginx example in `docs/deployment.md` (`location /`) already covers it. - **Static UI from another origin:** - The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for the warm-up banner and wildcard preview origins. - `build.sh` refuses `AUTH=true` there, since the server allows no credentialed CORS. - **Split containers:** - `CORESCOPE_IMAGE` is required. - With user accounts on, `userManagement.publicBaseUrl` must be the web container's public origin. - The built-in MQTT broker is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - The compose network is `172.30.83.0/24`, with the web container at `.200`. If Docker reports that the pool overlaps another network, set `SPLIT_NET`. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with new tests for busting every page and the WebSocket upgrade on `/`. - `sh test-all.sh` passes. New unit suites: - `test-static-build.js`: the build and its input checks, `same-origin` included, cross-origin logins refused, and an unrelated `OUT` left intact. - `test-web-container.js`: the Dockerfile, Caddyfile and compose file agree on packaging, forwarded paths, cache rules and the server port; the server image has no default and MQTT stays on localhost. - The `api-base`, anonymous mode, badge and `dev-static` suites. - `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes 47/47, the four pages included; it needs eslint, so CI runs it on E2E shard 1. - **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes 11/11. That is the static build on one origin against an API-only server on another, which CI runs on E2E shard 1. - **Split containers mode:** - The image builds, and `caddy validate` passes (Caddy 2.11.4). The new CI step passes locally, and fails when the Caddyfile stops forwarding `/api`. - Trusted proxies, with the stack up: the web container gets `172.30.83.200`. With only that address trusted, a spoofed `X-Forwarded-For` sent straight to a published 3000 is ignored, and one sent through Caddy is replaced with the client's address. With the subnet trusted, as the docs used to say, the spoofed address was believed: a deny entry for it rejected the upgrade. - With `docker-compose.split.yml` on the E2E fixture, seeded the way CI seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both through the web container and against the server directly. - Headers, checked with curl: - `no-cache` on `/`, `/index.html` and `/deployment.js`; - zstd on static assets; - `/api/*` proxied (`Via: 1.1 Caddy`); - a `101` WebSocket upgrade on `/ws`. - In Chromium: - every API request and the WebSocket stay on the web container's origin; - there are no page errors and no horizontal overflow at 390 px; - no badge shows on the `static` channel. - Logins through the web container: with user management on and `userManagement.publicBaseUrl` set to the web container's origin, a Chromium run passed every step, all on the web container's origin with no page errors: - registration, and activation from the mail link (which uses `publicBaseUrl`); - the session surviving a reload; - a CSRF-protected `PATCH /api/account`; - logout, then login with the form. - The same login with the backend's origin as `Origin` is refused with 403. The server checks `Origin` against `publicBaseUrl`, not `Host`, which is why the docs ask for it. ## Performance There are no hot-path changes. API calls go through `CSApi.url()`, a string concatenation. In the web container, Caddy serves files from disk and compresses them; API responses pass through with one proxy hop, the same as the all-in-one image's built-in Caddy. ## Notes - **Per-IP limits:** behind the web container, as behind the all-in-one image's Caddy, requests reach the server from the proxy's address. The per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`, the web container's fixed address. The docs say so. - **`depends_on`:** the web container starts with the server, not after its healthcheck, like the all-in-one image's Caddy. Gating on health would make every `up` wait for the first check (30 s) and fail outright when a large database loads for longer than the healthcheck allows; until the server listens, Caddy answers `/api` with 502. - **Not in this PR:** - publishing a `corescope-web` image (CI builds and smokes it, without publishing); - Cross-origin loginβ€”including a static frontend and API on sibling custom domainsβ€”is outside this PR. In the deployments supported by this PR, account support requires the frontend and API to share one browser origin, either served together or through a reverse proxy. πŸ€– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .eslintrc.json | 59 ++- .github/workflows/deploy.yml | 51 +++ AGENTS.md | 34 +- DEPLOY.md | 7 + Dockerfile.web | 38 ++ Makefile | 35 +- cmd/server/helpers_test.go | 32 ++ cmd/server/main.go | 64 +++- cmd/server/websocket_test.go | 28 ++ deploy/static/README.md | 174 +++++++++ deploy/static/_headers | 19 + docker-compose.split.yml | 99 +++++ docker/web/Caddyfile | 23 ++ docs/deployment.md | 80 ++++ public/account.js | 3 +- public/admin.js | 1 + public/analytics.js | 6 +- public/api-base.js | 85 +++++ public/app.js | 59 ++- public/area-filter.js | 2 +- public/area-map.html | 6 +- public/auth.js | 25 +- public/beta-ui.js | 93 +++++ public/channels.js | 2 +- public/customize-v2.js | 13 +- public/deployment.js | 10 + public/geofilter-builder.html | 8 +- public/index.html | 3 + public/live.js | 18 +- public/map.js | 23 +- public/mqtt-status-panel.js | 2 +- public/nav-drawer.js | 2 +- public/node-reach-coverage.js | 2 +- public/nodes.js | 4 +- public/packets.js | 13 +- public/path-inspector.js | 6 +- public/perf.js | 17 +- public/region-filter.js | 4 +- public/reported-nodes.js | 2 +- public/roles.js | 9 +- public/route-view.js | 6 +- public/rx-coverage.js | 8 +- public/style.css | 13 + public/warmup-banner.js | 2 +- scripts/dev-static.mjs | 96 +++++ scripts/non-unit-tests.json | 4 +- scripts/static/build.sh | 126 +++++++ test-all.sh | 6 + tests/e2e/test-lint-network-calls.js | 132 +++++++ tests/e2e/test-region-quick-picks.js | 4 + tests/e2e/test-split-anonymous-e2e.js | 191 ++++++++++ tests/unit/api-base-sandbox.js | 30 ++ tests/unit/test-1659-analytics-warmup.js | 2 + tests/unit/test-admin-dashboard-ui.js | 3 + tests/unit/test-aging.js | 2 + tests/unit/test-anonymous-mode.js | 163 +++++++++ tests/unit/test-api-base.js | 342 ++++++++++++++++++ tests/unit/test-area-filter.js | 2 + tests/unit/test-beta-ui.js | 156 ++++++++ tests/unit/test-dev-static.js | 89 +++++ tests/unit/test-fetch-all-nodes-pagination.js | 2 + tests/unit/test-frontend-helpers.js | 8 + tests/unit/test-hb-column-header-route.js | 2 + tests/unit/test-health-status.js | 2 + tests/unit/test-hop-path-detail.js | 2 + .../unit/test-issue-1136-observer-iata-map.js | 2 + .../unit/test-issue-1166-first-seen-column.js | 2 + tests/unit/test-issue-1356-map-a11y.js | 2 + .../test-issue-1407-cb-preset-propagation.js | 2 + .../test-issue-1412-customizer-no-override.js | 4 + tests/unit/test-issue-1418-cb-preset-ramp.js | 2 + .../unit/test-issue-1438-customizer-mcrole.js | 3 + .../unit/test-issue-1446-cb-preset-cascade.js | 8 + .../unit/test-issue-1470-node-tile-helper.js | 2 + .../unit/test-issue-1562-observers-summary.js | 2 + .../test-issue-1563-aggregate-and-inflight.js | 2 + .../unit/test-issue-1614-tile-url-function.js | 2 + tests/unit/test-issue-1849-trace-hashbytes.js | 2 + .../test-issue-1851-channel-message-scope.js | 3 + .../unit/test-issue-1862-map-region-filter.js | 3 + tests/unit/test-issue-1868-control-decode.js | 2 + .../unit/test-issue-1997-distance-building.js | 3 + tests/unit/test-issue-2001-map-scope-state.js | 2 + tests/unit/test-issue-2098-last-hop-region.js | 4 + ...-issue-2218-node-analytics-stale-render.js | 3 + .../test-issue-2230-account-stale-render.js | 2 + tests/unit/test-live-region-filter.js | 2 + tests/unit/test-live.js | 2 + tests/unit/test-map-clustering.js | 2 + tests/unit/test-my-repeaters-dashboard.js | 2 + tests/unit/test-nav-drawer-version-footer.js | 2 + .../unit/test-node-reach-coverage-debounce.js | 2 + .../unit/test-packet-detail-channel-escape.js | 2 + tests/unit/test-packet-detail-hash-size.js | 2 + tests/unit/test-packets.js | 2 + tests/unit/test-perf-disk-io-1120.js | 4 +- tests/unit/test-perf-render-1258.js | 4 +- tests/unit/test-pull-to-reconnect-1091.js | 2 + tests/unit/test-pull-to-reconnect.js | 2 + tests/unit/test-region-filter-url-param.js | 2 + tests/unit/test-reported-nodes.js | 2 + tests/unit/test-rx-coverage-viewport.js | 4 +- tests/unit/test-static-build.js | 198 ++++++++++ tests/unit/test-user-management-ui.js | 7 + tests/unit/test-warmup-banner.js | 4 + tests/unit/test-web-container.js | 117 ++++++ tests/unit/test-ws-stale-watchdog-1074.js | 2 + 107 files changed, 2861 insertions(+), 116 deletions(-) create mode 100644 Dockerfile.web create mode 100644 deploy/static/README.md create mode 100644 deploy/static/_headers create mode 100644 docker-compose.split.yml create mode 100644 docker/web/Caddyfile create mode 100644 public/api-base.js create mode 100644 public/beta-ui.js create mode 100644 public/deployment.js create mode 100644 scripts/dev-static.mjs create mode 100755 scripts/static/build.sh create mode 100644 tests/e2e/test-lint-network-calls.js create mode 100644 tests/e2e/test-split-anonymous-e2e.js create mode 100644 tests/unit/api-base-sandbox.js create mode 100644 tests/unit/test-anonymous-mode.js create mode 100644 tests/unit/test-api-base.js create mode 100644 tests/unit/test-beta-ui.js create mode 100644 tests/unit/test-dev-static.js create mode 100644 tests/unit/test-static-build.js create mode 100644 tests/unit/test-web-container.js diff --git a/.eslintrc.json b/.eslintrc.json index f6e644a5..7207011a 100644 --- a/.eslintrc.json +++ b/.eslintrc.json @@ -12,12 +12,14 @@ "CACHE_INVALIDATE_MS": "readonly", "CLIENT_CONFIG": "readonly", "CLIENT_TTL": "readonly", + "CSApi": "readonly", "CSAuth": "readonly", "ChannelColorPicker": "readonly", "ChannelColors": "readonly", "ChannelDecrypt": "readonly", "ChannelQR": "readonly", "Chart": "readonly", + "CSBeta": "readonly", "DIST_THRESHOLDS": "readonly", "DragManager": "readonly", "EXTERNAL_URLS": "readonly", @@ -298,6 +300,59 @@ "argsIgnorePattern": "^_", "varsIgnorePattern": "^_" } + ], + "no-restricted-syntax": [ + "error", + { + "selector": "CallExpression[callee.name='fetch']", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "CallExpression[callee.property.name='fetch'][callee.object.name=/^(window|self|globalThis)$/]", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "NewExpression[callee.name=/^(XMLHttpRequest|EventSource)$/]", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "CallExpression[callee.property.name='sendBeacon']", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "NewExpression[callee.name='WebSocket']:not([arguments.0.callee.property.name='wsUrl'])", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "Literal[value=/(href|src|action)=[\"']\\/api(\\/|$)/]", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "TemplateElement[value.raw=/(href|src|action)=[\"']\\/api(\\/|$)/]", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "AssignmentExpression:matches([left.property.name=/^(href|src|action|location)$/], [left.name='location']):matches([right.value=/^\\/api(\\/|$)/], [right.left.value=/^\\/api(\\/|$)/], [right.left.left.value=/^\\/api(\\/|$)/], [right.quasis.0.value.raw=/^\\/api(\\/|$)/])", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "CallExpression:matches([callee.name='open'], [callee.property.name='open'], [callee.object.name='location'][callee.property.name=/^(assign|replace)$/], [callee.object.property.name='location'][callee.property.name=/^(assign|replace)$/]):matches([arguments.0.value=/^\\/api(\\/|$)/], [arguments.0.left.value=/^\\/api(\\/|$)/], [arguments.0.left.left.value=/^\\/api(\\/|$)/], [arguments.0.quasis.0.value.raw=/^\\/api(\\/|$)/])", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + }, + { + "selector": "CallExpression[callee.property.name='setAttribute'][arguments.0.value=/^(href|src|action|formaction)$/]:matches([arguments.1.value=/^\\/api(\\/|$)/], [arguments.1.left.value=/^\\/api(\\/|$)/], [arguments.1.left.left.value=/^\\/api(\\/|$)/], [arguments.1.quasis.0.value.raw=/^\\/api(\\/|$)/])", + "message": "Every request to the CoreScope server goes through CSApi (public/api-base.js) so the UI also works from a static host or CDN: CSApi.fetch, CSApi.postJSON, CSApi.url, CSApi.wsUrl, or api() and CSAuth.request, which use it. See \"API calls\" in AGENTS.md." + } ] - } -} \ No newline at end of file + }, + "overrides": [ + { + "files": [ + "public/api-base.js" + ], + "rules": { + "no-restricted-syntax": "off" + } + } + ] +} diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 350c2864..a22c3425 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -553,6 +553,25 @@ jobs: sleep 1 done + - name: Start split frontend (static build + API-only server) + # The anonymous static build on :8788 against a stock server on :13591 + # that lists that origin in CORS_ALLOWED_ORIGINS, the way a static + # frontend (Cloudflare Pages) calls the containers. -public on a missing + # directory is API-only mode, so nothing is served same-origin. + if: matrix.shard == 1 + run: | + CHANNEL=dev COMMIT=e2e API_BASE=http://localhost:13591 AUTH=false CLASSIC_URL=http://localhost:13591 bash scripts/static/build.sh + fuser -k 13591/tcp 8788/tcp 2>/dev/null || true + CORS_ALLOWED_ORIGINS=http://localhost:8788 ./corescope-server -port 13591 -db test-fixtures/e2e-fixture.db -public /nonexistent & + echo $! > .server-split.pid + node scripts/dev-static.mjs --root dist/web --port 8788 & + echo $! > .static-split.pid + for i in $(seq 1 30); do + if curl -sf http://localhost:13591/api/healthz > /dev/null 2>&1 && curl -sf -o /dev/null http://localhost:8788/; then echo "split frontend ready after ${i}s"; break; fi + if [ "$i" -eq 30 ]; then echo "split frontend failed to start within 30s"; exit 1; fi + sleep 1 + done + - name: Run Playwright E2E tests (fail-fast) env: # On master pushes every suite records the frontend coverage of the @@ -587,6 +606,8 @@ jobs: # Allowlist: tests/a11y-allowlist.yaml (0 entries β€” hard pass policy). # Per-viewport summary printed at the end; any net>0 fails the build. suite 1 test-a11y-axe-1668.js BASE_URL=$B AXE_SCREENSHOT_DIR=/tmp/axe-1668 + # The static build on another origin than the API (scripts/static/build.sh). + suite 1 test-split-anonymous-e2e.js SPLIT_URL=http://localhost:8788 API_URL=http://localhost:13591 suite 2 test-filter-ux-e2e.js BASE_URL=$B suite 2 test-radio-setting-1869-e2e.js BASE_URL=$B suite 2 test-observer-neighbors-2270-e2e.js BASE_URL=$B @@ -719,6 +740,8 @@ jobs: suite 3 test-path-inspector-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B suite 3 test-pr-1490-live-map-gpu-animations-e2e.js CHROMIUM_REQUIRE=1 BASE_URL=$B suite 1 test-region-quick-picks.js + # The lint rule that keeps network calls on CSApi (needs eslint). + suite 1 test-lint-network-calls.js suite 3 test-table-sort.js suite 1 test-touch-targets.js CHROMIUM_REQUIRE=1 BASE_URL=$B suite 3 test-live-dedup.js BASE_URL=$B @@ -784,6 +807,9 @@ jobs: kill $(cat .server-um.pid) 2>/dev/null || true rm -f .server-um.pid fi + for f in .server-split.pid .static-split.pid; do + if [ -f "$f" ]; then kill $(cat "$f") 2>/dev/null || true; rm -f "$f"; fi + done - name: Upload shard results if: success() @@ -944,9 +970,34 @@ jobs: - name: Validate staging compose file run: docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging config --quiet + # Split containers: operators build the web image from their checkout and + # nothing publishes it, so without this a broken base image, package or + # Caddyfile would reach an operator first. + - name: Validate split compose file + run: CORESCOPE_IMAGE=corescope:check docker compose -f docker-compose.split.yml config --quiet + - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 + - name: Build and smoke the web image (Dockerfile.web) + env: + GIT_COMMIT: ${{ steps.meta.outputs.git_commit }} + run: | + set -euo pipefail + docker buildx build . -f Dockerfile.web --load -t corescope-web:check --build-arg GIT_COMMIT="$GIT_COMMIT" + # BACKEND is a closed port: /api must reach it (502), not the file server (404). + docker run -d --name web-check -p 8080:80 -e BACKEND=127.0.0.1:9 corescope-web:check + trap 'docker logs web-check 2>&1 | tail -20; docker rm -f web-check >/dev/null' EXIT + for i in $(seq 1 30); do curl -fs -o /dev/null http://localhost:8080/ && break; sleep 1; done + page=$(curl -fsS -D headers.txt http://localhost:8080/) + echo "$page" | grep -q "app.js?v=$GIT_COMMIT" || { echo "::error::index.html is not busted with the commit"; exit 1; } + if echo "$page" | grep -q __BUST__; then echo "::error::index.html still has __BUST__"; exit 1; fi + grep -qi '^cache-control: no-cache' headers.txt || { echo "::error::index.html may be cached"; exit 1; } + curl -fsS http://localhost:8080/deployment.js | grep -q "apiBase: ''" || { echo "::error::deployment.js is not same-origin"; exit 1; } + code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8080/api/healthz) + [ "$code" = 502 ] || { echo "::error::/api/healthz answered $code, not 502 from the closed BACKEND"; exit 1; } + echo "web image serves the packaged UI and forwards /api βœ…" + - name: Set up QEMU (arm64 runtime stage + arm64 smoke) uses: docker/setup-qemu-action@v3 diff --git a/AGENTS.md b/AGENTS.md index fd35106a..0e3f1726 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,8 @@ Guide for AI agents working on this codebase. Read this before writing any code. ## Architecture -Go backend + static frontend. No build step. No framework. No bundler. +Go backend + static frontend. No build step: `public/` is what the browser runs. No framework. No bundler. +`scripts/static/build.sh` is packaging, not a build: it copies `public/` for a static host or the web container, writes `deployment.js` and fills in `__BUST__`, and never transforms code. **⚠️ The Node.js server (server.js) is DEPRECATED and has been removed. All backend code is in Go.** **⚠️ DO NOT create or modify any Node.js server files. All backend changes go in `cmd/server/` or `cmd/ingestor/`.** @@ -18,7 +19,9 @@ cmd/server/ β€” Go API server (REST + WebSocket broadcast + static file s decoder.go β€” MeshCore packet decoder cmd/ingestor/ β€” Go MQTT ingestor (separate binary, writes to shared SQLite DB) public/ β€” Frontend (vanilla JS, one file per page) β€” ACTIVE, NOT DEPRECATED - app.js β€” SPA router, shared globals, theme loading + app.js β€” SPA router, shared globals, theme loading, api() (cached GETs) + deployment.js β€” where the API is (rewritten by scripts/static/build.sh for a static host) + api-base.js β€” CSApi: the ONLY code that builds API URLs or opens connections (see "API calls") roles.js β€” ROLE_COLORS, TYPE_COLORS, health thresholds, shared helpers nodes.js β€” Nodes list + side pane + full detail page map.js β€” Leaflet map with markers, legend, filters @@ -33,9 +36,10 @@ public/ β€” Frontend (vanilla JS, one file per page) β€” ACTIVE, NOT style.css β€” Main styles, CSS variables for theming live.css β€” Live page styles home.css β€” Home page styles - index.html β€” SPA shell, script/style tags with __BUST__ placeholder (auto-replaced at server startup) + index.html β€” SPA shell, script/style tags with __BUST__ placeholder (filled in automatically; see rule 3) test-fixtures/ β€” Real data SQLite fixture from staging (used for E2E tests) -scripts/ β€” Tooling (coverage collector, fixture capture, frontend instrumentation) +scripts/ β€” Tooling (coverage collector, fixture capture, frontend instrumentation; + static/build.sh packages the UI for a static host or the web container) ``` ### Data Flow @@ -105,7 +109,7 @@ Every change must consider performance impact BEFORE implementation. This codeba No proof = no merge. ### 1. No commit without tests -Every change that touches logic MUST have tests. For Go backend: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...`, or `make test` for all 14 modules. For frontend: `node tests/unit/test-packet-filter.js && node tests/unit/test-aging.js && node tests/unit/test-frontend-helpers.js`. If you add new logic, add tests. No exceptions. +Every change that touches logic MUST have tests. For Go backend: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...`, or `make test` for all 14 modules. For frontend: `node tests/unit/test-packet-filter.js && node tests/unit/test-aging.js && node tests/unit/test-frontend-helpers.js`, and `npx eslint public/*.js` (CI fails on any error). If you add new logic, add tests. No exceptions. The SQLite driver (`github.com/mattn/go-sqlite3`) is cgo. **`CGO_ENABLED=0` still *builds*** β€” that is the trap. It links a stub, and the binary dies on the first query with `go-sqlite3 requires cgo to work. This is a stub`, so a green build proves nothing. A plain `GOOS=linux go build` likewise cannot cross-compile it. Use `make build` / `make crossbuild` β€” the latter needs [`zig`](https://ziglang.org/download/) as the cross C compiler and produces static musl binaries. Never re-add `CGO_ENABLED=0`. @@ -113,7 +117,7 @@ The SQLite driver (`github.com/mattn/go-sqlite3`) is cgo. **`CGO_ENABLED=0` stil After pushing, verify the change works in an actual browser. Use `browser profile=openclaw` against the running instance. Take a screenshot if the change is visual. If you can't validate it, say so β€” don't claim it works. ### 3. Cache busters are automatic β€” do NOT manually edit them -Cache busters are injected automatically by the Go server at startup. The `__BUST__` placeholder in `index.html` is replaced with a Unix timestamp when the server reads the file. No manual bumping needed β€” every server restart picks up new asset versions. Do NOT replace `__BUST__` with hardcoded timestamps. +Cache busters are injected automatically. The Go server replaces the `__BUST__` placeholder in every top-level page (`index.html`, `area-map.html`, `geofilter-*.html`) with a Unix timestamp at startup, so every server restart picks up new asset versions. `scripts/static/build.sh` replaces it with the commit when it packages the UI. No manual bumping needed. Do NOT replace `__BUST__` with hardcoded timestamps, and give every local script or stylesheet a new page loads `?v=__BUST__`. ### 4. Verify API response shape before building UI Before writing client code that consumes an API endpoint, check what the endpoint ACTUALLY returns. Use `curl` or check the server code. Don't assume fields exist β€” grouped packets (`groupByHash=true`) have different fields than raw packets. This has caused multiple breakages. @@ -196,6 +200,22 @@ If you need to know how something works β€” a flag, a field, a timing, a behavio ## Frontend Conventions +### API calls (api-base.js) +The UI runs embedded in the server, in the web container, and from a static host or CDN on another origin (see "Deployment modes" in `docs/deployment.md`). `public/deployment.js` says where the API is, and `public/api-base.js` (`window.CSApi`) is the only code that turns a path into a URL or opens a connection. Everything else goes through it: + +| Need | Use | +|------|-----| +| GET JSON, cached | `api('/nodes?…')` (app.js) | +| Any other request | `CSApi.fetch('/api/…', opts)` | +| JSON POST | `CSApi.postJSON('/api/…', body)`: sends `text/plain`, so a cross-origin POST has no preflight (the server decodes JSON whatever the Content-Type; `cmd/server/simple_post_test.go`) | +| Logged-in request | `CSAuth.request(method, '/api/…', body)` | +| Link, download, `src` | `CSApi.url('/api/…')` | +| WebSocket | `new WebSocket(CSApi.wsUrl('/ws'))` | + +- **Never** call `fetch`, `XMLHttpRequest`, `EventSource`, `navigator.sendBeacon` or `new WebSocket(url)` directly, or put a bare `/api/…` path in an `href`, `src` or `action` (assigned, set with `setAttribute` or written into markup) or in `window.open`, `location.assign`, `location.replace` or `location =`. It works embedded and breaks on a static host. The `no-restricted-syntax` rule in `.eslintrc.json` rejects all of these in `public/*.js` (only `api-base.js` is exempt) and CI fails on it: run `npx eslint public/*.js` before pushing. That command never sees the inline scripts and markup of `public/*.html`; `tests/e2e/test-lint-network-calls.js` applies the same rule to them, and pins what the rule reports: extend it when you extend the rule. The rule sees a path at the start of a string, a concatenation or a template; a URL assembled elsewhere and passed in a variable gets past it, so it does not replace review. Do not silence it with `eslint-disable`. The one exception is `area-map.html`, whose Server field can name another server. +- **Logins and writes:** the anonymous static build (`CSApi.anonymous`) sends no cookies. A feature that needs a login or a write shows `CSBeta.classicOnlyHtml('…')` (or links with `CSApi.classicHref(hash)`) instead, as `account.js`, `admin.js` and `perf.js` do. +- **Tests:** a test that runs a frontend file outside the browser must load `deployment.js` and `api-base.js` first, or `CSApi` is undefined and its fetches fail silently inside their `try` blocks. vm sandboxes: `tests/unit/api-base-sandbox.js`. jsdom: `w.eval` both files before the file under test (`tests/e2e/test-region-quick-picks.js`). + ### Theming All colors MUST use CSS variables. Never hardcode `#hex` values outside of `:root` definitions. The customizer controls colors via `THEME_CSS_MAP` in customize.js. If you add a new color, add it as a CSS variable and map it in the customizer. @@ -378,6 +398,7 @@ One logical change per commit. Each commit is deployable. Each commit has its te | Gave up testing locally | 2 | Basic tests work on ARM β€” only heavy coverage scripts crash | | Copy-pasted functions for "coverage" | 1 | Test the real code, not copies in a helper file | | Subagent timed out mid-work | 4 | Give clear scope, don't try to run slow pipelines locally | +| Frontend file tested without api-base.js | 1 | Load `deployment.js` + `api-base.js` first (see "API calls") | ## File Naming - Tests: `test-{feature}.js` in `tests/unit/` (standalone, listed in `test-all.sh`) or `tests/e2e/` (browser, listed in `scripts/non-unit-tests.json`) β€” never in repo root @@ -397,5 +418,6 @@ Existing patterns: `#/nodes/{pubkey}?section=node-neighbors`, `#/analytics?tab=c - Don't add framework abstractions (React, Vue, etc.) - Don't hardcode colors β€” use CSS variables - Don't make per-packet server API calls from the frontend +- Don't call `fetch` / `WebSocket` / XHR directly from the frontend β€” go through `CSApi` (see "API calls") - Don't push without running tests - Don't start implementing without plan approval diff --git a/DEPLOY.md b/DEPLOY.md index 040cd098..444ed565 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -24,6 +24,13 @@ curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-c docker compose up -d ``` +### Other deployment modes + +The UI can also run in a container of its own (`docker-compose.split.yml`, with +`CORESCOPE_IMAGE` set to the server image of your checkout's version) or from a +static host or CDN such as Cloudflare Pages. See +[Deployment modes](docs/deployment.md#deployment-modes). + ## Image Tags | Tag | Description | diff --git a/Dockerfile.web b/Dockerfile.web new file mode 100644 index 00000000..e22799c9 --- /dev/null +++ b/Dockerfile.web @@ -0,0 +1,38 @@ +# CoreScope web container: the static UI on its own, served by Caddy, which +# forwards /api and /ws to a CoreScope server container. The UI and the API +# then share one origin, so the server needs no CORS settings. Logins work as +# on the classic UI once the server's userManagement.publicBaseUrl is this +# container's public origin. docker-compose.split.yml runs the two together; +# see "Deployment modes" in docs/deployment.md. +# +# Runtime environment: +# BACKEND host:port of the CoreScope server. Default corescope:3000. +# SITE_ADDRESS :80 (default) for plain HTTP, or a domain name for Caddy's +# automatic HTTPS (publish 80 and 443, and keep /data). + +FROM alpine:3.24 AS build +RUN apk add --no-cache bash +WORKDIR /src +COPY public/ public/ +COPY scripts/static/ scripts/static/ +COPY deploy/static/ deploy/static/ +# GIT_COMMIT is the cache buster on every asset (?v=); without one, the build +# time, as the server does at start. CHANNEL beta, next or dev shows a badge. +ARG GIT_COMMIT= +ARG CHANNEL=static +RUN COMMIT="$(printf '%.12s' "${GIT_COMMIT:-$(date +%s)}")" \ + API_BASE=same-origin AUTH=true CHANNEL="$CHANNEL" OUT=/srv \ + bash scripts/static/build.sh && \ + rm /srv/_headers + +FROM alpine:3.24 +RUN apk add --no-cache caddy +COPY --from=build /srv /srv +COPY docker/web/Caddyfile /etc/caddy/Caddyfile +ENV BACKEND=corescope:3000 \ + SITE_ADDRESS=:80 \ + XDG_DATA_HOME=/data \ + XDG_CONFIG_HOME=/config +EXPOSE 80 443 +VOLUME ["/data"] +CMD ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"] diff --git a/Makefile b/Makefile index 345d628f..cc633614 100644 --- a/Makefile +++ b/Makefile @@ -15,6 +15,8 @@ # make crossbuild # linux/amd64 + linux/arm64, static, into dist/ # make test / vet / fmt-check # across all modules # make docker-build # multi-arch image via buildx +# make web API_BASE=… AUTH=… # the UI alone, for a static host, into dist/web +# make dev-split # UI on :8788 calling the API on :3000 cross-origin GO ?= $(shell command -v go) GIT_VERSION ?= $(shell git describe --tags --match "v*" 2>/dev/null || echo unknown) @@ -59,12 +61,12 @@ DOCKER_IMAGE ?= ghcr.io/kpa-clawbot/corescope DOCKER_TAG ?= $(GIT_VERSION) DOCKER_PLATFORMS ?= linux/amd64,linux/arm64 -.PHONY: all build crossbuild test vet fmt-check tidy clean docker-build docker-push help +.PHONY: all build crossbuild test vet fmt-check tidy clean web dev-split docker-build docker-push help all: build help: - @echo "targets: build crossbuild test vet fmt-check tidy clean docker-build docker-push" + @echo "targets: build crossbuild test vet fmt-check tidy clean web dev-split docker-build docker-push" @echo " build-{$(shell echo $(CMDS) | tr ' ' ',')}" # -- host builds --------------------------------------------------------------- @@ -119,6 +121,35 @@ tidy: clean: rm -rf $(DIST) +# -- static frontend ------------------------------------------------------------ +# +# web packages public/ into dist/web for a static host or a CDN. It reads +# API_BASE, AUTH, CHANNEL, CLASSIC_URL and COMMIT; see scripts/static/build.sh. +# The Go binaries share dist/, which is why the UI gets a folder of its own. +web: + bash scripts/static/build.sh + +# dev-split runs the UI the way a static host serves it: the anonymous +# build on :8788 calling an API-only server on :3000 cross-origin, with no +# proxy. The server change is the one env var, exactly as in production. +# DEV_SPLIT_DB=path/to.db picks the data (default: the E2E fixture); it runs on a +# migrated copy in dist/, so the original is never touched. +DEV_SPLIT_DB ?= test-fixtures/e2e-fixture.db +DEV_SPLIT_API_PORT ?= 3000 +DEV_SPLIT_STATIC_PORT ?= 8788 + +dev-split: build-server build-migrate + cp $(DEV_SPLIT_DB) $(DIST)/dev-split.db + $(DIST)/corescope-migrate -db $(DIST)/dev-split.db + CHANNEL=dev COMMIT=dev AUTH=false \ + API_BASE=http://localhost:$(DEV_SPLIT_API_PORT) CLASSIC_URL=http://localhost:$(DEV_SPLIT_API_PORT) \ + bash scripts/static/build.sh + @echo "UI: http://localhost:$(DEV_SPLIT_STATIC_PORT) API: http://localhost:$(DEV_SPLIT_API_PORT) (Ctrl-C stops both)" + @trap 'kill 0' INT TERM EXIT; \ + CORS_ALLOWED_ORIGINS=http://localhost:$(DEV_SPLIT_STATIC_PORT) \ + $(DIST)/corescope-server -port $(DEV_SPLIT_API_PORT) -db $(DIST)/dev-split.db -public /nonexistent & \ + node scripts/dev-static.mjs --root $(DIST)/web --port $(DEV_SPLIT_STATIC_PORT) + # -- docker -------------------------------------------------------------------- docker-build: diff --git a/cmd/server/helpers_test.go b/cmd/server/helpers_test.go index b2bff47a..f146ffc5 100644 --- a/cmd/server/helpers_test.go +++ b/cmd/server/helpers_test.go @@ -8,6 +8,7 @@ import ( "net/http/httptest" "os" "path/filepath" + "regexp" "strings" "testing" ) @@ -447,6 +448,37 @@ func TestSpaHandlerCacheBust(t *testing.T) { }) } +// Every top-level page gets the same cache buster as index.html, so a tool +// page (area-map.html, geofilter-builder.html) served fresh never runs a +// script cached from before the server started. +func TestSpaHandlerBustsEveryTopLevelPage(t *testing.T) { + dir := t.TempDir() + os.WriteFile(filepath.Join(dir, "index.html"), []byte(``), 0644) + os.WriteFile(filepath.Join(dir, "tool.html"), []byte(``), 0644) + handler := spaHandler(dir, http.FileServer(http.Dir(dir))) + + get := func(p string) *httptest.ResponseRecorder { + w := httptest.NewRecorder() + handler.ServeHTTP(w, httptest.NewRequest("GET", p, nil)) + return w + } + bust := regexp.MustCompile(`v=(\d+)`) + index, tool := get("/"), get("/tool.html") + if strings.Contains(tool.Body.String(), "__BUST__") { + t.Fatalf("tool.html kept its placeholder: %s", tool.Body.String()) + } + im, tm := bust.FindStringSubmatch(index.Body.String()), bust.FindStringSubmatch(tool.Body.String()) + if im == nil || tm == nil || im[1] != tm[1] { + t.Errorf("tool.html buster %v, index.html buster %v: want the same timestamp", tm, im) + } + if got := tool.Header().Get("Cache-Control"); got != "no-cache, no-store, must-revalidate" { + t.Errorf("tool.html Cache-Control = %q", got) + } + if got := tool.Header().Get("Content-Type"); !strings.HasPrefix(got, "text/html") { + t.Errorf("tool.html Content-Type = %q", got) + } +} + func TestWriteJSON(t *testing.T) { w := httptest.NewRecorder() writeJSON(w, map[string]interface{}{"key": "value"}) diff --git a/cmd/server/main.go b/cmd/server/main.go index ac880ada..4958be1d 100644 --- a/cmd/server/main.go +++ b/cmd/server/main.go @@ -606,16 +606,22 @@ func main() { // It reads index.html once at creation time and replaces the __BUST__ placeholder // with a Unix timestamp so browsers fetch fresh JS/CSS after each server restart. func spaHandler(root string, fs http.Handler) http.Handler { - // Pre-process index.html: replace __BUST__ with a cache-bust timestamp - indexPath := filepath.Join(root, "index.html") - rawHTML, err := os.ReadFile(indexPath) - if err != nil { - log.Printf("[static] warning: could not read index.html for cache-bust: %v", err) - rawHTML = []byte("

CoreScope

index.html not found

") - } + // Pre-process every top-level page, index.html and the tool pages + // (area-map.html, geofilter-builder.html): replace __BUST__ with a + // cache-bust timestamp, so each loads the scripts of this start. bustValue := fmt.Sprintf("%d", time.Now().Unix()) - indexHTML := []byte(strings.ReplaceAll(string(rawHTML), "__BUST__", bustValue)) - log.Printf("[static] cache-bust value: %s", bustValue) + pages := bustPages(root, bustValue) + indexHTML, ok := pages["/index.html"] + if !ok { + log.Printf("[static] warning: could not read index.html for cache-bust") + indexHTML = []byte("

CoreScope

index.html not found

") + } + log.Printf("[static] cache-bust value: %s (%d pages)", bustValue, len(pages)) + servePage := func(w http.ResponseWriter, body []byte) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") + w.Write(body) + } return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Defense-in-depth: explicitly reject path-traversal attempts before @@ -628,20 +634,21 @@ func spaHandler(root string, fs http.Handler) http.Handler { http.Error(w, "bad request", http.StatusBadRequest) return } - // Serve pre-processed index.html for root and /index.html - if r.URL.Path == "/" || r.URL.Path == "/index.html" { - w.Header().Set("Content-Type", "text/html; charset=utf-8") - w.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") - w.Write(indexHTML) + // Serve the pre-processed index.html for the root, and each + // pre-processed page (/index.html included) at its own path + if r.URL.Path == "/" { + servePage(w, indexHTML) + return + } + if page, ok := pages[r.URL.Path]; ok { + servePage(w, page) return } path := filepath.Join(root, r.URL.Path) if _, err := os.Stat(path); os.IsNotExist(err) { // SPA fallback β€” serve pre-processed index.html - w.Header().Set("Content-Type", "text/html; charset=utf-8") - w.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") - w.Write(indexHTML) + servePage(w, indexHTML) return } // Disable caching for JS/CSS/HTML @@ -652,6 +659,29 @@ func spaHandler(root string, fs http.Handler) http.Handler { }) } +// bustPages reads each .html file at the top of root, keyed by its URL path +// ("/index.html"), with every __BUST__ replaced by bust. +func bustPages(root, bust string) map[string][]byte { + pages := map[string][]byte{} + entries, err := os.ReadDir(root) + if err != nil { + log.Printf("[static] warning: could not list %s for cache-bust: %v", root, err) + return pages + } + for _, e := range entries { + if e.IsDir() || !strings.EqualFold(filepath.Ext(e.Name()), ".html") { + continue + } + raw, err := os.ReadFile(filepath.Join(root, e.Name())) + if err != nil { + log.Printf("[static] warning: could not read %s for cache-bust: %v", e.Name(), err) + continue + } + pages["/"+e.Name()] = []byte(strings.ReplaceAll(string(raw), "__BUST__", bust)) + } + return pages +} + // isSafeStaticPath rejects request paths that contain traversal sequences // or backslashes β€” defense-in-depth for the SPA static handler so a future // router with SkipClean(true) cannot expose the filesystem. Empty input is diff --git a/cmd/server/websocket_test.go b/cmd/server/websocket_test.go index fbdd6210..17bf337d 100644 --- a/cmd/server/websocket_test.go +++ b/cmd/server/websocket_test.go @@ -454,3 +454,31 @@ func TestHubDefaultPingInterval(t *testing.T) { t.Fatalf("pingInterval = %v, want 30s (public/app.js WS_STALE_MS depends on it)", got) } } + +// UIs cached from before the UI moved its WebSocket to /ws still connect on +// "/": wsOrStatic upgrades any path the static handler serves. Keep that +// until those clients are gone; this pins it. +func TestWebSocketUpgradeOnRootStillWorks(t *testing.T) { + hub := NewHub() + static := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte("static")) + }) + srv := httptest.NewServer(wsOrStatic(hub, static)) + defer srv.Close() + + conn, resp, err := websocket.DefaultDialer.Dial("ws"+srv.URL[4:]+"/", nil) + if err != nil { + t.Fatalf("dial /: %v", err) + } + defer conn.Close() + if resp.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("dial /: status %d, want 101", resp.StatusCode) + } + deadline := time.Now().Add(2 * time.Second) + for hub.ClientCount() != 1 && time.Now().Before(deadline) { + time.Sleep(10 * time.Millisecond) + } + if hub.ClientCount() != 1 { + t.Errorf("expected the root connection to register with the hub, got %d clients", hub.ClientCount()) + } +} diff --git a/deploy/static/README.md b/deploy/static/README.md new file mode 100644 index 00000000..a7d53f00 --- /dev/null +++ b/deploy/static/README.md @@ -0,0 +1,174 @@ +# Frontend on a static host or CDN + +The CoreScope UI is static files. This directory and `scripts/static/build.sh` +let it be served from Cloudflare Pages or any other static host, while the +existing container keeps serving the API and the classic, embedded UI. This is +the third of the deployment modes in +[docs/deployment.md](../../docs/deployment.md#deployment-modes); the other two +run the UI in a container. + +This is the **anonymous** build (`AUTH=false`): it reads everything a logged-out +visitor can read, live updates included, and links to the classic site for +logging in and admin work. On the server it needs one setting, +`CORS_ALLOWED_ORIGINS` (see [Server](#server)), on a version that includes #2362: +`edge` now, and releases cut from `master` after it. Older servers accept the +origin too, but they hide the `Retry-After` and `X-CoreScope-Load-Status` +headers from it, so the warm-up banner never shows, and they refuse wildcard +preview origins. + +## How it works + +- `public/deployment.js` says where the API is. The embedded UI ships it with + `apiBase: ''` (same origin, as today). `build.sh` rewrites it for the + static build. +- `public/api-base.js` (`window.CSApi`) is the only place that builds API and + WebSocket URLs. A lint rule keeps raw `fetch('/api…')` out of every other file. +- The anonymous build sends no cookies, sends its three JSON `POST`s as + `text/plain` (so the browser sends no preflight), and hides login, account and + admin controls in favour of an "available on classic" link. + +## Building + +```bash +make web API_BASE=https://analyzer.example.org AUTH=false +``` + +writes `dist/web`: `public/` byte for byte, except that every top-level page +(`index.html` and the tool pages) gets the cache buster and `deployment.js` says +where the API is. Serve that folder at the root of a hostname; the UI uses +absolute paths such as `/icons/…`. `_headers` holds the cache rules: the pages +and `deployment.js` always revalidate, while everything they load carries +`?v=`. Cloudflare Pages and Netlify read it; other hosts need the same +rules in their own config. The variables are in step 6 below and at the top of +`scripts/static/build.sh`. + +## Deploying from a fork + +This repository is the software, not any one CoreScope instance: it has no +Pages project and deploys nothing. Each instance (a region, the flagship) +deploys from its own fork. Pages needs that anyway: its GitHub App is installed +on a repository you administer. + +The fork carries **no code changes**. Every setting lives in the Pages project +variables, and deploying means moving a branch to a commit that already exists +upstream. + +### One-time setup + +1. Fork this repository into an account or org your maintainers administer. +2. In the fork, turn GitHub Actions off (Settings β†’ Actions). Otherwise syncing + `master` runs this repository's CI and deploy workflows there, which fail + without its secrets. Pages builds do not use Actions. +3. Create the deploy branches `stage` and `prod`, and protect `prod` so only + maintainers can move it. +4. Install the Cloudflare Pages GitHub App on the fork only. +5. Create one Pages project per deploy branch, on the Free plan: + + | Project | Production branch | Serves | API | + |---|---|---|---| + | e.g. `corescope-web-stage` | `stage` | your staging frontend | your staging server | + | e.g. `corescope-web-prod` | `prod` | your production frontend | your production server | + + For both: + - build command: `bash scripts/static/build.sh`; + - build output directory: `dist/web`; root directory: blank (the + repository root holds files over Pages' 25 MiB limit); + - preview branches: None, or All to get a preview of every other branch + pushed to the fork (see Previews below). + + One project on `prod` is enough if you have no staging server. +6. Environment variables (Production environment, and Preview too if previews + are on) for each project: + + | Name | Example | Notes | + |---|---|---| + | `API_BASE` | `https://analyzer.example.org` | The CoreScope server. https, lowercase, no path | + | `AUTH` | `false` | Required, no default. Must be `false`: the server allows no credentialed CORS, so `build.sh` refuses `true` with a cross-origin `API_BASE` | + | `CHANNEL` | `beta` | `beta`, `next` or `dev` show a badge; `static` shows none | + | `CLASSIC_URL` | `https://analyzer.example.org` | Optional. Target of the "Open in classic" links | + | `SKIP_DEPENDENCY_INSTALL` | `1` | `build.sh` needs no npm packages | + + `COMMIT` defaults to the first 7 characters of `CF_PAGES_COMMIT_SHA`. +7. Add a custom domain to each project (for example `beta.example.org`). The + UI uses absolute paths such as `/icons/…`, so it must be served at the root + of a hostname. + +### Deploying + +Keep each frontend on the version its server runs: `prod` on the release tag +your production containers run, `stage` on upstream `master` if staging runs +the master image. Upgrade the server first, then move the branch. Move branches +by fast-forward, so they stay identical to upstream commits (`upstream` is this +repository, `region` is your fork, and `vX.Y.Z` is the release your production +server runs): + +```bash +git fetch upstream --tags +git push region vX.Y.Z:refs/heads/prod +git push region upstream/master:refs/heads/stage +``` + +Releases up to v3.16.0 predate `scripts/static/build.sh` and have nothing to +build, so `prod` can only follow releases cut after it. + +Only move `stage` to a `master` commit whose CI passed. A pull request into +`prod` also works if you want a review step, but its merge commits make the +branch drift from upstream. Each project builds only when its branch moves. + +### Previews + +With preview branches on, every other branch pushed to the fork, and every pull +request between branches of the fork, gets its own deployment at +`..pages.dev` plus a branch alias such as +`my-branch..pages.dev`. Set `CHANNEL=dev` in the Preview environment +so previews show the Dev badge. Pull requests to this repository come from +forks and get no preview. + +A preview is code from a branch, so keep previews anonymous (`AUTH=false`), +pointed at a staging server if you have one. + +## Server + +Append the static origin to the embed CORS list and restart: + +```bash +# Keep every origin already listed; add the static one exactly. Never "*". +CORS_ALLOWED_ORIGINS=https://existing-embed.example.org,https://beta.example.org +``` + +- The env var replaces `corsAllowedOrigins` from `config.json`, so it must list + every origin, old and new. +- An entry is an exact origin: lowercase, scheme included, no trailing slash. +- For previews, also list `https://.pages.dev` and + `https://*..pages.dev` (wildcard labels need a server with #2362; see + `config.example.json`). The wildcard matches one label only, and the server + refuses `https://*.pages.dev`. +- The same list admits the origin to `/ws`. A `*` entry does not work for `/ws`. + +Check it: + +```bash +curl -sI -H "Origin: https://beta.example.org" https://analyzer.example.org/api/stats | grep -i access-control-allow-origin +``` + +It must print the origin, and no `Access-Control-Allow-Credentials` line. + +## Local and CI + +```bash +make dev-split +``` + +The UI on `http://localhost:8788` calls an API-only server on +`http://localhost:3000` cross-origin, with no proxy. CI runs +`tests/e2e/test-split-anonymous-e2e.js` the same way on E2E shard 1. Pull requests +to this repository get no preview deployment; this is how to try one. + +## Roll back + +- **The UI:** in the Pages dashboard, roll back to the previous deployment, or + move the deploy branch back to the previous release tag + (`git push --force-with-lease region vX.Y.Z:refs/heads/prod`). +- **Access from the static origin:** remove it from `CORS_ALLOWED_ORIGINS` and + restart. The static UI then shows "Can't reach the CoreScope API"; the classic + UI and embeds are unaffected. diff --git a/deploy/static/_headers b/deploy/static/_headers new file mode 100644 index 00000000..56aa174b --- /dev/null +++ b/deploy/static/_headers @@ -0,0 +1,19 @@ +# Copied into the static build by scripts/static/build.sh. Cloudflare Pages +# and Netlify read it; other hosts ignore it and need the same rules in their +# own config (the web container's are in docker/web/Caddyfile). The pages and +# deployment.js change on every deploy without a ?v= buster, so a browser must +# always revalidate them. Every local script and stylesheet a page loads +# carries ?v= (vendor/ files carry their version in the path), so +# those may use the host's default caching. +/ + Cache-Control: no-cache, no-store, must-revalidate +/index.html + Cache-Control: no-cache, no-store, must-revalidate +/area-map.html + Cache-Control: no-cache, no-store, must-revalidate +/geofilter-builder.html + Cache-Control: no-cache, no-store, must-revalidate +/geofilter-docs.html + Cache-Control: no-cache, no-store, must-revalidate +/deployment.js + Cache-Control: no-cache, no-store, must-revalidate diff --git a/docker-compose.split.yml b/docker-compose.split.yml new file mode 100644 index 00000000..bdf90546 --- /dev/null +++ b/docker-compose.split.yml @@ -0,0 +1,99 @@ +# CoreScope β€” split containers: the UI in its own web container and the +# server in the usual image. +# +# The web container (Dockerfile.web, Caddy) serves the static UI and forwards +# /api and /ws to the server, so both share one origin: no CORS settings. With +# user management on, set the server's userManagement.publicBaseUrl to the web +# container's public origin (scheme and port included): logins and other +# writes are checked against it. The server container is unchanged and still +# serves everything, the classic UI included, on port 3000 inside the compose +# network; publish it too if you want to reach that directly. For per-IP +# limits, the server trusts the web container's fixed address: see networks. +# +# Usage: CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge \ +# docker compose -f docker-compose.split.yml up -d --build +# Docs: docs/deployment.md, "Deployment modes" +# +# Variables (shell or .env): +# CORESCOPE_IMAGE required: the server image of the same version as this +# checkout, which the web image is built from. :edge on +# master, :vX.Y.Z on a release tag. There is no default, so +# a release server is never paired with a newer UI unasked. +# DATA_DIR the server's data directory. Default ./data +# HTTP_PORT default 80; HTTPS_PORT default 443 (TCP and UDP) +# SITE_ADDRESS :80 (default), or a domain name for automatic HTTPS, which +# needs the default ports: certificate issuance and the +# HTTP/3 advertisement both assume 80 and 443 +# GIT_COMMIT the UI's cache buster; default the build time +# CHANNEL static (default), or beta, next, dev for a badge +# MQTT_BIND where the server's built-in broker is published. Default +# 127.0.0.1: it accepts anonymous clients, so set 0.0.0.0 +# only to let gateways on a network you trust publish to it. +# MQTT_PORT default 1883 +# SPLIT_NET the compose network's /24, as its first three octets. +# Default 172.30.83, which puts the web container at +# 172.30.83.200. Change it only if Docker reports that the +# pool overlaps another network, and update the server's +# trustedProxies to match. + +services: + corescope: + image: ${CORESCOPE_IMAGE:?set CORESCOPE_IMAGE to the server image of this checkout's version, e.g. ghcr.io/kpa-clawbot/corescope:edge on master or :vX.Y.Z on a release tag} + ports: + - "${MQTT_BIND:-127.0.0.1}:${MQTT_PORT:-1883}:1883" + volumes: + - ${DATA_DIR:-./data}:/app/data + environment: + - DISABLE_CADDY=true + - DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false} + restart: unless-stopped + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/healthz"] + interval: 30s + timeout: 5s + retries: 3 + + web: + build: + context: . + dockerfile: Dockerfile.web + args: + GIT_COMMIT: ${GIT_COMMIT:-} + CHANNEL: ${CHANNEL:-static} + image: corescope-web:latest + ports: + - "${HTTP_PORT:-80}:80" + - "${HTTPS_PORT:-443}:443" + # HTTP/3: Caddy advertises it on 443 when SITE_ADDRESS is a domain. + - "${HTTPS_PORT:-443}:443/udp" + volumes: + - web-caddy-data:/data + environment: + - BACKEND=corescope:3000 + - SITE_ADDRESS=${SITE_ADDRESS:-:80} + networks: + default: + # The one address the server's trustedProxies lists; see networks. + ipv4_address: "${SPLIT_NET:-172.30.83}.200" + depends_on: + - corescope + restart: unless-stopped + +networks: + # A fixed subnet gives the web container a fixed address, so the server's + # webSocket.trustedProxies and userManagement.trustedProxies can list it + # alone (172.30.83.200 by default) and still match after a recreate. Never + # list the whole subnet: it holds the gateway address too, and a client + # that reaches a published port 3000 through Docker's proxy arrives from + # there, so its own X-Forwarded-For would be believed. + default: + ipam: + config: + - subnet: "${SPLIT_NET:-172.30.83}.0/24" + # Automatic addresses come from the lower half, so no container + # can take the web container's. + ip_range: "${SPLIT_NET:-172.30.83}.0/25" + +volumes: + # Caddy's TLS certificates when SITE_ADDRESS is a domain name. + web-caddy-data: diff --git a/docker/web/Caddyfile b/docker/web/Caddyfile new file mode 100644 index 00000000..7e259cc8 --- /dev/null +++ b/docker/web/Caddyfile @@ -0,0 +1,23 @@ +# The web container (Dockerfile.web): the static UI from /srv, with the API +# and the WebSocket forwarded to the CoreScope server, so both share this +# origin. BACKEND is the server's host:port; SITE_ADDRESS is ":80" for plain +# HTTP, or a domain name for automatic HTTPS. +# +# Caddy adds X-Forwarded-For. To keep the server's per-IP limits behind this +# proxy, list its address in webSocket.trustedProxies (and +# userManagement.trustedProxies) in the server's config.json. + +{$SITE_ADDRESS::80} { + encode zstd gzip + + @backend path /api/* /ws + reverse_proxy @backend {$BACKEND} + + # The same rules as deploy/static/_headers: these change on every deploy + # without a ?v= buster, so a browser must always revalidate them. + @nocache path / /index.html /area-map.html /geofilter-builder.html /geofilter-docs.html /deployment.js + header @nocache Cache-Control "no-cache, no-store, must-revalidate" + + root * /srv + file_server +} diff --git a/docs/deployment.md b/docs/deployment.md index 4c4d5c6e..78309d63 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -5,6 +5,7 @@ Comprehensive guide to deploying and operating CoreScope. For a quick start, see ## Table of Contents - [System Requirements](#system-requirements) +- [Deployment modes](#deployment-modes) - [Docker Deployment](#docker-deployment) - [Configuration Reference](#configuration-reference) - [MQTT Setup](#mqtt-setup) @@ -33,6 +34,85 @@ CoreScope runs well on Raspberry Pi 4/5 (ARM64). The Go server uses ~300 MB RAM --- +## Deployment modes + +The UI is static files (`public/`) that call the server's API. The three modes +differ only in what serves those files. The server container is the same image +in all three, and it still serves the classic UI itself. + +| Mode | The UI is served by | API | Server CORS setting | Logins | +|---|---|---|---|---| +| [All-in-one container](#all-in-one-container) | the server container | same origin | none | yes | +| [Split containers](#split-containers) | a web container (`Dockerfile.web`) that forwards the API | same origin | none | yes | +| [Static host or CDN](#static-host-or-cdn) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | + +### All-in-one container + +The default, and what the rest of this guide describes: one image runs the +server, the ingestor and, optionally, Mosquitto and Caddy. The server serves the +UI from `/app/public` and the API on one origin. See +[Docker Deployment](#docker-deployment). + +### Split containers + +`docker-compose.split.yml` runs the server image unchanged next to a web +container built from `Dockerfile.web`. The web container is Caddy: it serves the +UI, packaged by `scripts/static/build.sh` when the image is built, and forwards +`/api/*` and `/ws` to the server. The browser sees one origin, so the server +needs no CORS setting. Logins work as on the classic UI when account +configuration uses the public web origin, as described below. + +```bash +CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge \ + docker compose -f docker-compose.split.yml up -d --build +``` + +| Variable | Default | | +|---|---|---| +| `HTTP_PORT`, `HTTPS_PORT` | `80`, `443` | Published by the web container; 443 over TCP and UDP (HTTP/3) | +| `SITE_ADDRESS` | `:80` | A domain name turns on Caddy's automatic HTTPS and HTTP/3. Keep the default ports then: certificate issuance and the HTTP/3 advertisement both assume 80 and 443 | +| `CORESCOPE_IMAGE` | none, required | The server image of the same version as the checkout the web image is built from: `:edge` on `master`, `:vX.Y.Z` on a release tag | +| `DATA_DIR` | `./data` | The server's data directory | +| `GIT_COMMIT` | the build time | The UI's cache buster | +| `CHANNEL` | `static` | `beta`, `next` or `dev` shows a badge | +| `MQTT_BIND`, `MQTT_PORT` | `127.0.0.1`, `1883` | Where the server's built-in broker is published. It accepts anonymous clients, so set `MQTT_BIND=0.0.0.0` only to let gateways on a network you trust publish to it | +| `SPLIT_NET` | `172.30.83` | The compose network's /24, as its first three octets; the web container is `.200` in it. Change it only if Docker reports that the pool overlaps another network, and update `trustedProxies` to match | + +- When `userManagement.enabled` is true, set `userManagement.publicBaseUrl` in + the server's config to the public web origin, including its scheme and port + (for example, `http://localhost:8080` for a web container published on port + 8080). Login and other state-changing requests compare the browser's `Origin` + or `Referer` with this configured origin; forwarding the public `Host` header + alone is not enough. This value also determines mail links. See + [account configuration](user-guide/accounts.md#2-configure). +- The web container finds the server through `BACKEND` (`corescope:3000` in the + compose file). Port 3000 stays inside the compose network, and the server + still serves everything there, the classic UI included; publish it if you want + to reach that directly. +- The web image is built from your checkout, so keep the checkout and + `CORESCOPE_IMAGE` on the same version: a newer UI may call endpoints an older + server lacks. To update only the UI, rebuild the web container: + `docker compose -f docker-compose.split.yml up -d --build web`. +- Behind the web container, as behind the all-in-one image's Caddy, every + request reaches the server from the proxy's address, so per-IP limits are off + until `webSocket.trustedProxies` and `userManagement.trustedProxies` list it. + The compose file fixes that address at `172.30.83.200`: list it alone. Never + list the compose subnet. It also holds the gateway address, and a client that + reaches a published port 3000 through Docker's proxy (from the host itself or + over IPv6, for example) arrives from there, so its own `X-Forwarded-For` + would be believed. + +### Static host or CDN + +`make web` (or `scripts/static/build.sh`) writes the anonymous build to +`dist/web` for Cloudflare Pages or any static host. It calls the server +cross-origin, so the server lists the UI's origin in `CORS_ALLOWED_ORIGINS`. +It sends no cookies; logins and admin pages link to the classic UI on the +server. Setup, previews, server settings and rollback: +[deploy/static/README.md](../deploy/static/README.md). + +--- + ## Docker Deployment ### Quick Start (one command) diff --git a/public/account.js b/public/account.js index 9345dbf3..e8ed6063 100644 --- a/public/account.js +++ b/public/account.js @@ -119,7 +119,7 @@ (window.CSSettingsSync ? '

Settings sync

' : '') + '

My data

' + '' + - '' + + '' + '

Delete account