mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 17:58:06 +00:00
## What this does The frontend half of serving the UI apart from the server. The server half was #2362. It gives CoreScope three deployment modes. The server container is the same image in all three, and it still serves everything, the classic UI included. | Mode | The UI is served by | API | Server CORS setting | Logins | |---|---|---|---|---| | All-in-one container (today) | the server container | same origin | none | yes | | Split containers (new) | a web container that forwards the API | same origin | none | yes | | Static host or CDN (new) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | ### Commit 1: the UI as static files, for any host - **One place for API URLs:** `public/deployment.js` says where the API is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket URL. A lint rule keeps raw `/api` fetches out of the other files. The embedded UI keeps `apiBase: ''`, so it behaves as before, except that its WebSocket now uses `/ws` instead of `/`. - **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`, which is `public/` byte for byte except for the cache buster in `index.html` and `deployment.js`. - It is `dist/web` rather than `dist/` because the Go binaries are built into `dist/` and `make clean` removes that folder. - `API_BASE=same-origin` is for a host that forwards `/api` and `/ws` itself, as the web container does. - **Anonymous build (`AUTH=false`):** for another origin. It has no account features: login, account, admin and other writes link to the same page on the classic UI. It sends no cookies, and its three JSON POSTs go as `text/plain`, so there is no preflight. - **Error notices:** a blocked API answer or an HTML answer each show a notice. - **Channel badge:** `beta`, `next` and `dev` builds show a badge that opens a pre-filled issue, plus an "Open in classic" link. - **Runbook:** `deploy/static/README.md` covers Cloudflare Pages, previews, server CORS and rollback. ### Commit 2: the web container and the three modes - **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It packages the UI with `build.sh` when the image is built, serves it, and forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`). `SITE_ADDRESS` set to a domain name turns on automatic HTTPS. - **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules as `deploy/static/_headers`, and compresses responses with zstd or gzip. - **`docker-compose.split.yml`:** the stock server image (its own Caddy off) plus the web container: `CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f docker-compose.split.yml up -d --build`. - `CORESCOPE_IMAGE` is required and must match the checkout's version, so a release server is never paired with a newer UI by default. - The server's built-in broker, which accepts anonymous clients, is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - **Docs:** `docs/deployment.md` has a new "Deployment modes" section, and `DEPLOY.md` points to it. ### Keeping new code on `CSApi` - **Lint rule:** `.eslintrc.json` now rejects every direct network call in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`, `sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in `href`, `src` and `action` (assigned, set with `setAttribute` or written into markup), `window.open`, `location.assign`, `location.replace` and `location =`. `api-base.js` is the one file exempt. - **HTML pages:** `eslint public/*.js` never sees the inline scripts and markup of `public/*.html`, so `test-lint-network-calls.js` applies the same rule to them, at their page lines. `area-map.html`'s two fetches carry an `eslint-disable` with the reason: its Server field can name another server. - **Why it changed:** the earlier rule only caught a literal `/api…` passed straight to `fetch`. - **Check:** CI already fails on lint errors, and `tests/e2e/test-lint-network-calls.js` pins exactly what the rule reports: 25 bad statements reported, 15 good ones not, `api-base.js` exempt. - **AGENTS.md:** a new "API calls" section covers: - which helper to use for each need; - the lint rule; - how anonymous builds gate login-only features; - loading `api-base.js` in vm and jsdom tests (the region quick-picks suite missed that, and failed shard 1 on the first CI run). ### Review follow-ups - **Account tokens:** the channel badge's pre-filled issue drops the query on `#/account/…` routes, so mail-link tokens never reach a GitHub URL. "Open in classic" keeps the full route, since the classic UI needs the token. - **HTML answers:** on a packaged UI (static host, web container), `api()` rejects an HTML answer before status and retry handling. An HTML 404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server returned an unexpected response" after one request. It used to give a generic error, or six "warming up" retries for a 503. JSON errors and the server's JSON 503 warm-up behave as before. The embedded UI keeps its old handling (`CSApi.embedded`). - **Logins:** `docs/deployment.md`, `Dockerfile.web` and `docker-compose.split.yml` say `userManagement.publicBaseUrl` must be the web container's public origin. ### Audit fixes - **No broken logins build:** `build.sh` refuses `AUTH=true` with a cross-origin `API_BASE`. The server allows no credentialed CORS, so every API request of such a build was blocked; in Chromium all 12 were, and the pages stayed empty. - **No accidental deletes:** `build.sh` refuses an `OUT` that is neither empty nor a previous build. It used to wipe whatever folder or file `OUT` named. - **Lint:** the rule now also reports `/api/` paths in `href`, `src` and `action` attributes in single-quoted strings and template literals. `tests/e2e/test-lint-network-calls.js` pins exactly what it reports. - **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published on localhost only. - **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since v3.16.0 and v3.15.0 predate the static build. It also names the server version the static build needs: before #2362 the warm-up headers stay hidden and wildcard previews are refused. ### Audit notes, fixed - **Cache busting on every page:** the tool pages (`area-map.html`, `geofilter-builder.html`) load their scripts with `?v=__BUST__`. The server and `build.sh` replace it in every top-level page, and `_headers` and the Caddyfile make every page revalidate. Behind Caddy, a returning visitor could otherwise get a new tool page with an old `api-base.js`. - **HTTP/3:** the split compose publishes 443 over UDP too, for the HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`. - **WebSocket on `/`:** a Go test pins that an upgrade on `/` still works, for UIs cached from before `/ws`. ### Second review - **Trusted proxies:** the docs said to trust the compose subnet, and also to publish port 3000 if wanted. Together they let a client spoof its address: the subnet holds the gateway, where a client reaching a published 3000 through Docker's proxy arrives from. The compose file now fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web container's address (`172.30.83.200`), and the docs say to trust that address alone. - **Classic UI unchanged:** the HTML-answer notice above applies to packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as before, and an unknown `/api` path fails to parse as before, with no notice. - **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md says what the rule cannot see: a URL assembled elsewhere and passed in a variable. - **AGENTS.md:** `build.sh` is described as packaging (a copy, `deployment.js` and `__BUST__`, never a code transform), and rule 3 says `__BUST__` is filled in every top-level page. - **CI builds the web image:** the image-check job (no publish) builds `Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`, the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It also validates the split compose file. - **Cross-origin logins:** a hand-edited `deployment.js` with `auth: true` and another origin now logs a warning naming the setting, instead of failing with CORS errors alone. ## Upgrade notes - **WebSocket path:** the UI's live feed connects to `/ws` instead of `/`. The server has answered `/ws` since #1793 and still upgrades `/`, so browsers holding the previous UI keep working. - A reverse proxy that forwards WebSocket upgrades only for an exact `/` must forward `/ws` too. - The nginx example in `docs/deployment.md` (`location /`) already covers it. - **Static UI from another origin:** - The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for the warm-up banner and wildcard preview origins. - `build.sh` refuses `AUTH=true` there, since the server allows no credentialed CORS. - **Split containers:** - `CORESCOPE_IMAGE` is required. - With user accounts on, `userManagement.publicBaseUrl` must be the web container's public origin. - The built-in MQTT broker is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - The compose network is `172.30.83.0/24`, with the web container at `.200`. If Docker reports that the pool overlaps another network, set `SPLIT_NET`. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with new tests for busting every page and the WebSocket upgrade on `/`. - `sh test-all.sh` passes. New unit suites: - `test-static-build.js`: the build and its input checks, `same-origin` included, cross-origin logins refused, and an unrelated `OUT` left intact. - `test-web-container.js`: the Dockerfile, Caddyfile and compose file agree on packaging, forwarded paths, cache rules and the server port; the server image has no default and MQTT stays on localhost. - The `api-base`, anonymous mode, badge and `dev-static` suites. - `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes 47/47, the four pages included; it needs eslint, so CI runs it on E2E shard 1. - **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes 11/11. That is the static build on one origin against an API-only server on another, which CI runs on E2E shard 1. - **Split containers mode:** - The image builds, and `caddy validate` passes (Caddy 2.11.4). The new CI step passes locally, and fails when the Caddyfile stops forwarding `/api`. - Trusted proxies, with the stack up: the web container gets `172.30.83.200`. With only that address trusted, a spoofed `X-Forwarded-For` sent straight to a published 3000 is ignored, and one sent through Caddy is replaced with the client's address. With the subnet trusted, as the docs used to say, the spoofed address was believed: a deny entry for it rejected the upgrade. - With `docker-compose.split.yml` on the E2E fixture, seeded the way CI seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both through the web container and against the server directly. - Headers, checked with curl: - `no-cache` on `/`, `/index.html` and `/deployment.js`; - zstd on static assets; - `/api/*` proxied (`Via: 1.1 Caddy`); - a `101` WebSocket upgrade on `/ws`. - In Chromium: - every API request and the WebSocket stay on the web container's origin; - there are no page errors and no horizontal overflow at 390 px; - no badge shows on the `static` channel. - Logins through the web container: with user management on and `userManagement.publicBaseUrl` set to the web container's origin, a Chromium run passed every step, all on the web container's origin with no page errors: - registration, and activation from the mail link (which uses `publicBaseUrl`); - the session surviving a reload; - a CSRF-protected `PATCH /api/account`; - logout, then login with the form. - The same login with the backend's origin as `Origin` is refused with 403. The server checks `Origin` against `publicBaseUrl`, not `Host`, which is why the docs ask for it. ## Performance There are no hot-path changes. API calls go through `CSApi.url()`, a string concatenation. In the web container, Caddy serves files from disk and compresses them; API responses pass through with one proxy hop, the same as the all-in-one image's built-in Caddy. ## Notes - **Per-IP limits:** behind the web container, as behind the all-in-one image's Caddy, requests reach the server from the proxy's address. The per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`, the web container's fixed address. The docs say so. - **`depends_on`:** the web container starts with the server, not after its healthcheck, like the all-in-one image's Caddy. Gating on health would make every `up` wait for the first check (30 s) and fail outright when a large database loads for longer than the healthcheck allows; until the server listens, Caddy answers `/api` with 502. - **Not in this PR:** - publishing a `corescope-web` image (CI builds and smokes it, without publishing); - Cross-origin login—including a static frontend and API on sibling custom domains—is outside this PR. In the deployments supported by this PR, account support requires the frontend and API to share one browser origin, either served together or through a reverse proxy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
127 lines
6.2 KiB
Bash
Executable File
127 lines
6.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Packages public/ for a static host (Cloudflare Pages, a CDN, the web
|
|
# container) instead of the CoreScope server. This is a packaging step, not a
|
|
# build step: the output is public/ byte for byte, except that index.html gets
|
|
# a cache buster and deployment.js says where the API is.
|
|
#
|
|
# A static host runs it as the build command (output dir dist/web); the web
|
|
# container (Dockerfile.web) runs it when the image is built. Locally:
|
|
# make web, or make dev-split to try it against a local server.
|
|
#
|
|
# Environment:
|
|
# API_BASE required. https://host[:port] of the CoreScope server
|
|
# (http only for localhost / 127.0.0.1), or same-origin for a
|
|
# host that forwards /api and /ws to the server itself.
|
|
# AUTH required, true or false; there is no default, so nobody turns
|
|
# logins on by accident. false builds the anonymous UI: no
|
|
# cookies, no login or admin controls. It is the only kind that
|
|
# works on another origin, because the server allows no
|
|
# credentialed CORS. true needs API_BASE=same-origin: the UI
|
|
# then uses the server's logins exactly as embedded.
|
|
# CHANNEL beta | next | dev | static. Default static.
|
|
# COMMIT shown in the UI and used as the cache buster. Default: the
|
|
# first 7 characters of CF_PAGES_COMMIT_SHA (set by Pages), else
|
|
# git rev-parse --short HEAD.
|
|
# CLASSIC_URL optional. The embedded UI, for "Open in classic" links.
|
|
# OUT output directory. Default dist/web.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/../.."
|
|
|
|
fail() { echo "build: $*" >&2; exit 1; }
|
|
|
|
OUT="${OUT:-dist/web}"
|
|
CHANNEL="${CHANNEL:-static}"
|
|
API_BASE="${API_BASE:-}"
|
|
AUTH="${AUTH:-}"
|
|
CLASSIC_URL="${CLASSIC_URL:-}"
|
|
COMMIT="${COMMIT:-}"
|
|
if [ -z "$COMMIT" ] && [ -n "${CF_PAGES_COMMIT_SHA:-}" ]; then COMMIT="${CF_PAGES_COMMIT_SHA:0:7}"; fi
|
|
if [ -z "$COMMIT" ]; then COMMIT="$(git rev-parse --short HEAD 2>/dev/null || true)"; fi
|
|
|
|
# Every value lands inside a JS string literal, so each one is held to a
|
|
# shape that cannot break out of it.
|
|
origin_ok() {
|
|
[[ "$1" =~ ^https://[a-z0-9.-]+(:[0-9]{1,5})?$ ]] && return 0
|
|
[[ "$1" =~ ^http://(localhost|127\.0\.0\.1)(:[0-9]{1,5})?$ ]]
|
|
}
|
|
[ -n "$API_BASE" ] || fail "API_BASE is required (the CoreScope server, e.g. https://analyzer.example.org, or same-origin)"
|
|
case "$AUTH" in
|
|
true|false) ;;
|
|
*) fail "AUTH must be true or false (false for the anonymous build)" ;;
|
|
esac
|
|
API_BASE="${API_BASE%/}"
|
|
if [ "$API_BASE" = same-origin ]; then
|
|
[ "$AUTH" = true ] || fail "same-origin needs AUTH=true: a UI on the server's own origin uses its logins"
|
|
API_BASE=''
|
|
else
|
|
origin_ok "$API_BASE" || fail "API_BASE must be https://host[:port] in lowercase, with no path (http only for localhost), or same-origin: $API_BASE"
|
|
# A build with logins sends the session cookie with every request, and the
|
|
# server never answers a credentialed cross-origin request, so the browser
|
|
# would block all of them.
|
|
[ "$AUTH" = false ] || fail "AUTH=true needs API_BASE=same-origin: the server allows no credentialed CORS, so every API request of a cross-origin build with logins would be blocked (use AUTH=false)"
|
|
fi
|
|
if [ -n "$CLASSIC_URL" ]; then
|
|
CLASSIC_URL="${CLASSIC_URL%/}"
|
|
origin_ok "$CLASSIC_URL" || fail "CLASSIC_URL must be https://host[:port], with no path: $CLASSIC_URL"
|
|
fi
|
|
case "$CHANNEL" in
|
|
beta|next|dev|static) ;;
|
|
*) fail "CHANNEL must be beta, next, dev or static: $CHANNEL" ;;
|
|
esac
|
|
[[ "$COMMIT" =~ ^[A-Za-z0-9._-]{1,40}$ ]] || fail "COMMIT must be 1-40 characters of [A-Za-z0-9._-]: '$COMMIT'"
|
|
|
|
# OUT is replaced wholesale, so it must be new, empty or a previous build:
|
|
# a typo in OUT must not wipe an unrelated folder.
|
|
MARKER='/* Written by scripts/static/build.sh. */'
|
|
if [ -e "$OUT" ]; then
|
|
[ -d "$OUT" ] || fail "OUT is not a directory: $OUT"
|
|
if [ -n "$(ls -A "$OUT")" ] && [ "$(head -n 1 "$OUT/deployment.js" 2>/dev/null)" != "$MARKER" ]; then
|
|
fail "refusing to replace $OUT: it is neither empty nor a previous build (no deployment.js written by this script)"
|
|
fi
|
|
fi
|
|
|
|
rm -rf "$OUT"
|
|
mkdir -p "$OUT"
|
|
cp -R public/. "$OUT/"
|
|
|
|
# The same substitution spaHandler makes at server start in every top-level
|
|
# page, with the commit in place of a timestamp so a rebuild of one commit is
|
|
# byte-identical.
|
|
for page in "$OUT"/*.html; do
|
|
sed -i.bak "s/__BUST__/$COMMIT/g" "$page" && rm -f "$page.bak"
|
|
done
|
|
|
|
classic_js=null
|
|
[ -n "$CLASSIC_URL" ] && classic_js="'$CLASSIC_URL'"
|
|
cat > "$OUT/deployment.js" <<EOF
|
|
$MARKER
|
|
window.CS_DEPLOYMENT = { channel: '$CHANNEL', commit: '$COMMIT', version: null, apiBase: '$API_BASE', auth: $AUTH, classicUrl: $classic_js };
|
|
EOF
|
|
|
|
cp deploy/static/_headers "$OUT/_headers"
|
|
|
|
# -- checks ---------------------------------------------------------------------
|
|
if grep -rl "__BUST__" "$OUT" >/dev/null; then fail "a __BUST__ placeholder is left in $(grep -rl "__BUST__" "$OUT" | head -1)"; fi
|
|
[ ! -e "$OUT/functions" ] && [ ! -e "$OUT/_worker.js" ] || fail "functions/ and _worker.js are not allowed: the frontend is static only"
|
|
count=$(find "$OUT" -type f | wc -l)
|
|
[ "$count" -lt 20000 ] || fail "$count files; Cloudflare Pages allows fewer than 20000"
|
|
# 25 MiB in bytes: busybox find (the web image) has no M suffix.
|
|
big=$(find "$OUT" -type f -size +26214400c | head -1)
|
|
[ -z "$big" ] || fail "$big is over 25 MiB, the Cloudflare Pages per-file limit"
|
|
# Byte for byte public/, plus _headers. deployment.js is new, and the
|
|
# top-level pages differ only by the cache buster.
|
|
expected=$( (cd public && find . -type f; echo ./_headers) | sort)
|
|
actual=$(cd "$OUT" && find . -type f | sort)
|
|
[ "$expected" = "$actual" ] || fail "the output is not exactly public/ plus _headers:
|
|
$(diff <(echo "$expected") <(echo "$actual") || true)"
|
|
while IFS= read -r f; do
|
|
case "$f" in
|
|
./deployment.js|./_headers) ;;
|
|
./*/*) cmp -s "public/$f" "$OUT/$f" || fail "$f differs from public/$f" ;;
|
|
./*.html) sed "s/__BUST__/$COMMIT/g" "public/$f" | cmp -s - "$OUT/$f" || fail "$f is not public/$f with the cache buster" ;;
|
|
*) cmp -s "public/$f" "$OUT/$f" || fail "$f differs from public/$f" ;;
|
|
esac
|
|
done <<< "$actual"
|
|
|
|
echo "build: $OUT ($count files) channel=$CHANNEL commit=$COMMIT apiBase=${API_BASE:-same-origin} auth=$AUTH"
|