Commit Graph
13 Commits
Author SHA1 Message Date
4ebb5ccc11 feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.

Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.

Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.

Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.

Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).

Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-16 16:15:36 +01:00
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00
efitenandClaude Opus 5.5 fad303a4f4 docs: split the Docker minimum into running and building the image (#2361)
`docs/deployment.md` listed Docker 20.10+ as the minimum, without
separating running the published image from building it. The Dockerfile
needs BuildKit (`# syntax=docker/dockerfile:1` on line 1, `RUN --mount`
from line 71), and BuildKit is the default builder only from Docker
23.0. On 20.10 to 22.x the legacy builder rejects `RUN --mount` unless
`DOCKER_BUILDKIT=1` is set. Found while reviewing #2359.

Changes:
- `docs/deployment.md`: the requirements table gets two Docker rows,
20.10+ to run the published image, and 23.0+ (or 20.10+ with
`DOCKER_BUILDKIT=1`) to build it, plus one sentence on why.
- `docs/go-migration.md`: that guide builds the image, so its
prerequisite becomes 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`).

Not verified: I did not run a build on Docker 20.10 or 22.x; the
legacy-builder behaviour comes from Docker's documentation, not from a
test here. Podman and buildah are not covered.

Docs only.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:00:52 +01:00
n30nexandn30nex b3a9f2a9e9 docs: make scheduled backups portable and bounded (#2353)
Fixes #2352. Follow-up to #2345.

The older deployment guide recommended copying a live telemetry
database. It now points to the consistent-snapshot procedure. The
scheduled-download example uses a trailing-X temporary filename, bounds
connection and total download time, and explains the weak-key denylist.

Validation: the parent reran the extracted command against eight
disposable HTTP cases: valid SQLite, unauthorized, server error, empty
content, HTML, corrupt SQLite, truncated transfer and timeout. Failed
downloads removed partial files and preserved existing completed
backups; the timeout exited 28. Shell syntax and diff checks passed. GNU
tools were exercised; BSD/macOS was not run natively.

TDD exemption: pure documentation, no application or repository test
files changed. No runtime/browser behaviour, dependencies or
configuration changed. Independent reviews are complete; the one
timeout-wording finding was addressed in 33d629b5 without changing the
validated command. Exact-head CI passed, including the browser shards.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 11:59:39 +01:00
n30nexandn30nex 8437d22356 docs: use consistent snapshots for backup and restore (#2345)
The deployment guide copied a running `meshcore.db`, which can omit
committed WAL data or produce an inconsistent backup. It now uses the
existing authenticated `/api/backup` snapshot, checks the download
before publishing it, and explains how to restore with all database
users stopped while preserving the old database and sidecars for
rollback.

Fixes #703, following the [maintainer's remaining documentation
scope](https://github.com/Kpa-clawbot/CoreScope/issues/703#issuecomment-6163256620).

- Protects API credentials in a header file and keeps incomplete
downloads out of the completed backup set.
- Links the separate account-backup guide; configuration and secrets
remain separate backup responsibilities.
- Replaces unsafe live-copy scheduling instructions with the same
checked download procedure.

Validation:
- Exact shell example with real curl/SQLite: valid download, HTTP
401/500, empty, HTML, corrupt database, and truncated transfer. Failures
preserved the previous backup and published no new `.db`.
- Existing `TestBackup*` passed. Chromium downloaded an authenticated
snapshot from a disposable instance; a row committed only in its
nonempty WAL survived and `quick_check` returned `ok`.
- Disposable stopped-file restore and rollback recovered the original
WAL-only row and preserved the original files byte-for-byte.
- `git diff --check` and privacy preflight passed.

TDD exemption: pure docs; no application behavior changed and no test
files modified. No new config/customizer settings. Live operator
services were not used for validation.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:14 +01:00
a164a44e7f fix(docker): run the Go services as an unprivileged user (#2125)
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in
with `RUN_AS_UID`. A code-execution bug in either one then no longer
lands as root next to the mounted data and `config.json`. `supervisord`
stays root, for mosquitto and caddy.

| `RUN_AS_UID` | Go services run as |
|---|---|
| unset (default) | root, as before. Upgrades change nothing |
| set | `RUN_AS_UID` / `RUN_AS_GID` |

With `RUN_AS_UID` set:
- Each start hands the data directory to that uid (`chown -h`), except
`.env` and anything mounted into it on its own. Failures are logged.
- `.env` is read only if it is root-owned and only root can write it.
`RUN_AS_UID` and `RUN_AS_GID` are never taken from it.
- `/app` stays root's. **Depends on #2137.**
- A uid with no account in the image gets one at start; `caddy` and
`mosquitto` are reused. A uid that can't be provided fails startup
instead of falling back to root.

The logic is in `docker/select-service-user.sh`, tested by
`docker/test-select-service-user.sh`, which needs no Docker or root and
runs in CI. See also "Running the Go services unprivileged" in
`docs/deployment.md`.

Verified in the built image (master + this + #2137) across 14 cases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
2026-10-15 09:49:29 +02:00
Alex B e6323ec587 fix(store): account path, decode-cache and dedup-key bytes so maxMemoryMB eviction triggers (#2035)
trackedBytes undercounted the packet store by about 2.5x, so packetStore.maxMemoryMB never triggered: a tx was charged at creation, before pickBestObservation set its path, so the byPathHop and spTxIndex costs were never added, and eviction then re-estimated with the path known and subtracted more than had been added, drifting the total downwards. The ParsedDecoded cache, the obsKeys dedup key and several per-observation strings were not estimated at all.

StoreTx.accountedBytes now records what was charged, rechargeTx returns the delta after every pickBestObservation, and eviction subtracts accountedBytes instead of re-estimating. Measured by the author on a production database copy: trackedMB 151 against 402 MB of heap in use before, 351 against 396 MB after, with GC cycles dropping from ~0.71/s to ~0.011/s over 12 h on their instance.

Reviewed by auditing the accounting rather than the arithmetic: all six production pickBestObservation sites recharge, all three subtraction sites read accountedBytes, every recharge site holds s.mu (Load from :857, the two ingest paths at :2778 and :3142 with deferred unlocks), observations are charged only after acceptance, and no charged tx is discarded during the chunk merge. That lock audit is the independent check, because CI's race job covers the ingestor only.

Operator impact, both from the estimate growing rather than any limit moving: where maxMemoryMB is set, eviction now caps the real store size, and the cold load clamp drops about a third of the boot walk (124420 to 84374 packets at 650 MB). Where it is unset, which is the default, the change is inert. docs/go-migration.md claimed the Go server ignored the setting, which was never true, and is corrected here.

Merged by the interim maintainer without a second human reviewer: CI (run 35258839322) plus the review above are the independent checks.
2026-09-17 21:58:16 +02:00
8ce5291b7c fix(map): apply the CARTO key on every map surface (rebase of #1916 onto #1919) (#1926)
Continues #1916 by @nullrouten0. The commit is theirs, authorship
unchanged; I rebased it onto master and resolved the fallout from #1919.
Opening it here rather than force-pushing to someone else's branch.

## Why this is needed after #1919

#1919 shipped the Carto key, but only for the five `BASE_STYLES` entries
in `map-tile-providers.js`. Four map surfaces still request unkeyed
tiles and get them back stamped `API KEY REQUIRED` at HTTP 200:

- `public/roles.js` — `getTileUrl()` returns the bare `TILE_LIGHT`
constant in light mode and never consults the registry (the dark branch
has consulted it since #1461). Affects `analytics.js:2200` and
`nodes.js:94`, so the analytics map and the node-detail map stay
watermarked in light theme even with a key configured.
- `public/customize-v2.js:1838` and `:2047` — the two geo-filter maps in
the customizer.
- `public/geofilter-builder.html` — standalone page, outside the SPA, so
it fetches `/api/config/client` itself.

@nullrouten0 had already found and fixed all four, plus written the
docs, before #1919 was merged. Their diagnosis of the light-mode branch
is in the PR verbatim.

## What I changed while rebasing

1. **`carto.token` → `carto.key`.** #1919 shipped `key` and operators
already have it in their configs; renaming now would break them
silently. All of #1916's code, tests and docs follow suit.
2. **Dropped the duplicate key getter.** `_getCartoKeyParam` did the
same job as master's `_getCartoKey`; kept master's.
3. **Merged the two test suites.** Master's five cases plus four of
#1916's that master does not cover: the `api_key`-is-ignored assertion,
lazy resolution after async config, and both `MC_tileUrlById` cases. 42
passed, 0 failed.
4. **Corrected one factual claim in the docs.** #1916 stated CARTO
offers no referrer or origin restriction. CARTO does ask for a domain
when the key is issued. Whether that is enforced per request is not
something I verified, so the note now says exactly that rather than
asserting either way.
5. Merged the two config comments, keeping the operationally useful
part: unkeyed tiles return **200** with a watermark, so nothing errors
and no healthcheck fires. Verify by looking at a tile, not at a status
code.

## Verification

Rebased onto `7aa60c03`. No regressions:

| | base 7aa60c03 | this branch |
|---|---|---|
| `test-issue-1420-tile-providers.js` | 38 passed, 0 failed | **42
passed, 0 failed** |
| `test-issue-1614-tile-url-function.js` | 3 passed, 0 failed | 3
passed, 0 failed |
| `test-issue-1470-node-tile-helper.js` | 3 passed, 3 failed | 3 passed,
3 failed (pre-existing) |
| `test-frontend-helpers.js` | 625 passed, 2 failed | 625 passed, 2
failed (pre-existing) |

Also verified end to end on a live instance running this change with a
real key: the tile that came back carried the `API KEY REQUIRED`
watermark before and came back clean after.

@nullrouten0 — this is your work and I would rather you had the credit
and the merge. Say the word and I will close this and hand the rebase
back to you, or push it to your branch instead if you prefer #1916 to
stay the vehicle.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE

Co-authored-by: nullrouten <nullrouten@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 11:50:45 +02:00
Eldoon Nemar d7cd9203ca Fixes #1165: add OSM/Stamen tile providers with per-provider Leaflet layer control. (#1533)
List of changes too long to describe, so I'll hit high level.

- Config now supports the json map tiles that were suggested by
@Kpa-clawbot.
- Leaflet map layer button appears in the top right of live.js and
map.js (because all the work was already done on live.js... Added bonus)
- Allows users to enter creds for OSM and Stamen to get enterprise
related perks, in the config file
- Added a default light map under customizer. Still suggest removing
them all together and relying on the config
- You can enable OSM and Stamen in the config without a license, but at
your own risk!!!
- Config comment explains where to register and the providers for osm,
as well as the general limits per X interval
- Updated tests (28) to address the changes made to the maps

### TDD Exemption

**Reason**: Net-new UI surfaces (per `AGENTS.md`)

This PR introduces a net-new UI surface (the multi-provider map tile
selector). Under the `AGENTS.md` exemption for net-new UI surfaces, the
absence of an initial failing (red) commit is permitted, as the UI was
built first. However, the underlying public APIs are fully covered.

The following tests serve as the first assertions for these new APIs:
- `window.MC_createLayerControl`: Asserted in `MC_createLayerControl
handles Auto mode and explicit layers correctly`
- `window.MC_setDarkTileProvider` & `window.MC_getDarkTileProvider`:
Asserted in `MC_setDarkTileProvider persists to localStorage...`
- `window.MC_setLightTileProvider` & `window.MC_getLightTileProvider`:
Asserted in `MC_setLightTileProvider persists to localStorage...`
- `window.MC_initTileRegistry`: Asserted in `MC_initTileRegistry(true)
dispatches mc-tile-provider-changed`
- `applyTileFilter`: Asserted in `applyTileFilter sets invert CSS for
inverted dark provider...`
- Cross-tab synchronization: Asserted in `Cross-tab storage event
re-dispatches mc-tile-provider-changed`
2026-06-04 06:53:30 -07:00
63bfa3d910 feat(security): detect CDN-fronted deployment + document bypass requirement (closes #1561) (#1564)
Closes #1561. Follow-up to #1551.

## Why

#1551 added `Cache-Control: no-store` to all `/api/*` responses. That's
sufficient for CDNs that honour origin headers (Varnish, nginx). It is
**not** sufficient for Cloudflare zones where Cache Rules / Page Rules
override origin Cache-Control.

Field evidence from the meshat.se diagnosis (2026-06-04): observers
behind Cloudflare were returning `cf-cache-status: HIT` with `age` up to
~6 hours despite the origin emitting `no-store`. The CDN was caching per
zone policy and ignoring the upstream directive — exactly the failure
mode #1551 cannot reach. The application has no way to inject CDN rules;
the only durable fix is operator-side.

This PR makes that operator step discoverable and verifiable.

## What

### Server-side detection (log-only)

`cmd/server/cdn_detection.go` adds a middleware wired into the `/api/*`
chain after `noStoreAPIMiddleware`. On the **first** request bearing any
CDN-typical header (`CF-Connecting-IP`, `CF-Ray`, `X-Forwarded-For`,
`X-Real-IP`, `Fastly-Client-IP`, `True-Client-IP`) it logs:

```
[security] WARNING: detected request via CDN (CF-Ray header present).
Ensure /api/* is bypassed in your CDN config — see docs/deployment-behind-cdn.md.
Cached API responses cause observer-flap and incorrect dashboards.
```

`sync.Once` guarantees the warning fires at most once per process boot.
The middleware never blocks, never modifies the response, never adds
headers. Detection is observational only — operators who run behind a
CDN without bypass have a real bug; the warning is appropriate.

### Operator documentation

`docs/deployment.md` gains a new **"Behind a CDN (Cloudflare, Fastly)"**
section covering:

1. Curl verification command + healthy vs unhealthy output examples
2. Cloudflare Cache Rule creation (URI Path starts-with `/api/` → Bypass
cache)
3. Legacy Page Rules equivalent
4. Fastly note
5. Re-verification
6. Meaning of the startup log warning
7. Why we can't fix this server-side

`docs/deployment-behind-cdn.md` is the canonical path the log message
references — it's a short TL;DR that links back to the full section.

### Healthcheck script

`scripts/check-cdn-bypass.sh` — POSIX sh, no dependencies beyond curl +
grep + awk. Operators run:

```sh
scripts/check-cdn-bypass.sh https://your-domain.example.com
```

Exits `0` with `OK: no CDN caching detected ...` or `1` with a precise
diagnostic naming the offending header (`cf-cache-status: HIT` or stale
`age`).

## TDD

- **Red commit `e90ccaba`** (`test(security): RED ...`) —
`cmd/server/cdn_detection_test.go` (4 Go tests + 6 subtests for each
header) and `scripts/test-check-cdn-bypass.sh` (3 shell harness cases).
Middleware stub returns `next` unchanged so tests compile and fail on
assertions, not build errors.
- **Green commit `5e6a60b5`** (`feat(security): GREEN ...`) — real
middleware, wiring in `routes.go`, healthcheck script, doc.

## Deliverables

| File | Status | Purpose |
|------|--------|---------|
| `cmd/server/cdn_detection.go` | new | middleware + sync.Once warning |
| `cmd/server/cdn_detection_test.go` | new | 4 Go tests (1 stand-alone +
1 silence + 1 once + 1 table-driven over 6 headers) |
| `cmd/server/routes.go` | modified | `r.Use(cdnDetectionMiddleware)`
after no-store |
| `docs/deployment.md` | modified | TOC entry + "Behind a CDN" section |
| `docs/deployment-behind-cdn.md` | new | canonical path referenced by
log message + script output |
| `scripts/check-cdn-bypass.sh` | new | operator-runnable healthcheck |
| `scripts/test-check-cdn-bypass.sh` | new | shell harness with fake
curl |

## What this PR explicitly does NOT do

- Does not block requests based on CDN detection (log-only).
- Does not enforce CDN bypass (impossible — operator-controlled).
- Does not spoof, strip or modify CDN headers.
- Does not add CSP / HSTS / other security headers (out of scope).
- Warning is not configurable — operators behind a CDN without bypass
have a real bug, surfacing it is correct.

## Verification

- `go test ./...` in `cmd/server/` — full suite green.
- `sh scripts/test-check-cdn-bypass.sh` — 3/3 pass.
- Preflight checklist — all 11 gates clean (PII, branch scope, red
commit, CSS vars, CSS self-fallback, LIKE-on-JSON, sync migration,
async-migration annotation, XSS sinks, img/SVG ratio, themed-img/SVG,
fixture coverage).

---------

Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: clawbot <bot@clawbot.invalid>
2026-06-04 13:14:09 +00:00
Kpa-clawbotandyou 26c47df814 fix: entrypoint .env support + deployment docs for bare docker run (#704)
## Summary

Fixes #702 — `.env` file `DISABLE_MOSQUITTO`/`DISABLE_CADDY` ignored
when using `docker run`.

## Changes

### Entrypoint sources `/app/data/.env`
The entrypoint now sources `/app/data/.env` (if present) before the
`DISABLE_*` checks. This works regardless of how the container is
started — `docker run`, compose, or `manage.sh`.

```bash
if [ -f /app/data/.env ]; then
  set -a
  . /app/data/.env
  set +a
fi
```

### `DISABLE_CADDY` added to compose files
Both `docker-compose.yml` and `docker-compose.staging.yml` now forward
`DISABLE_CADDY` to the container environment (was missing — only
`DISABLE_MOSQUITTO` was wired).

### Deployment docs updated
- `docs/deployment.md`: bare `docker run` is now the primary/recommended
approach with a full parameter reference table
- Documents the `/app/data/.env` convenience feature
- Compose and `manage.sh` marked as legacy alternatives
- `DISABLE_CADDY` added to the environment variable reference

### README quick start updated
Shows the full `docker run` command with `--restart`, ports, and
volumes. Includes HTTPS variant. Documents `-e` flags and `.env` file.

### v3.5.0 release notes
Updated the env var documentation to mention the `.env` file support.

## Testing
- All Go server tests pass
- All Go ingestor tests pass
- No logic changes to Go code — entrypoint shell script + docs only

---------

Co-authored-by: you <you@example.com>
2026-04-11 20:43:16 -07:00
you cde62166cb docs: v3.5.0 release notes + API documentation across README, deployment guide, FAQ
- Release notes for 95 commits since v3.4.1
- OpenAPI/Swagger docs: /api/spec and /api/docs called out everywhere
- Deployment guide: new API Documentation section
- README: API docs link added
- FAQ: 'Where is the API documentation?' entry
- Test plans for v3.4.2 validation
2026-04-08 06:55:25 +00:00
Kpa-clawbotandyou a068e3e086 feat: zero-config defaults + deployment docs (M3-M4, #610) (#631)
## Zero-Config Defaults + Deployment Docs

Make CoreScope start with zero configuration — no `config.json`
required. The ingestor falls back to sensible defaults (local MQTT
broker, standard topics, default DB path) when no config file exists.

### What changed

**`cmd/ingestor/config.go`** — `LoadConfig` no longer errors on missing
config file. Instead it logs a message and uses defaults. If no MQTT
sources are configured (from file or env), defaults to
`mqtt://localhost:1883` with `meshcore/#` topic.

**`cmd/ingestor/main.go`** — Removed redundant "no MQTT sources" fatal
(now handled in config layer). Improved the "no connections established"
fatal with actionable hints.

**`README.md`** — Replaced "Docker (Recommended)" section with a
one-command quickstart using the pre-built image. No build step, no
config file, just `docker run`.

**`docs/deployment.md`** — New comprehensive deployment guide covering
Docker, Compose, config reference, MQTT setup, TLS/HTTPS, monitoring,
backup, and troubleshooting.

### Zero-config flow

```
docker run -d -p 80:80 -v corescope-data:/app/data ghcr.io/kpa-clawbot/corescope:latest
```

1. No config.json found → defaults used, log message printed
2. No MQTT sources → defaults to `mqtt://localhost:1883`
3. Internal Mosquitto broker already running in container → connection
succeeds
4. Dashboard shows empty, ready for packets

### Review fixes (commit 13b89bb)

- Removed `DISABLE_CADDY` references from all docs — this env var was
never implemented in the entrypoint
- Fixed `/api/stats` example in deployment guide — showed nonexistent
fields (`mqttConnected`, `uptimeSeconds`, `activeNodes`)
- Improved MQTT connection failure message with actionable
troubleshooting hints

Closes #610

---------

Co-authored-by: you <you@example.com>
2026-04-05 15:04:49 -07:00