Closes the recurring "sync migration on large table" regression class
(#791-style, #1483-style).
## Problem
Pattern that keeps repeating:
1. A perf/feature PR adds `CREATE INDEX` / `ALTER TABLE` / `UPDATE ...
WHERE` in a migration file (typically `cmd/ingestor/db.go`).
2. Local dev DB has ~100 rows. Migration returns in milliseconds. CI is
green.
3. Reviewers approve on plan correctness; nobody knows what the prod
table size is.
4. First prod boot at scale (Cascadia: ~2600 nodes, 80K+ obs; previous
prod: 1.9M+ obs) pins the ingestor at `[migration] Adding index...` for
minutes.
5. Healthcheck times out → container restart → loop. Operator pages.
Hotfix.
Most recent case: `obs_observer_ts_idx_v1` in v3.8.3 — release notes
already document an "expect a longer first boot" warning because we knew
it would hit prod hard.
## What this PR adds
**Async helper (`cmd/ingestor/async_migration.go`):**
- `Store.RunAsyncMigration(ctx, name, fn)` — registers the migration as
`pending_async` in a new `_async_migrations` bookkeeping table, returns
to caller immediately, schedules `fn` in a goroutine on the shared
backfill `WaitGroup`, transitions to `done` (or `failed` with error
captured) on completion.
- `Store.AsyncMigrationStatus(name)` and
`Store.WaitForAsyncMigrations()` for tests/shutdown.
- Idempotent: `done` rows short-circuit; `pending_async`/`failed` rows
are retried on next boot.
**Retroactive #1483 conversion (`cmd/ingestor/db.go`):**
- `obs_observer_ts_idx_v1` (the composite `(observer_idx, timestamp)`
index build on `observations`) is now scheduled via `RunAsyncMigration`
from `OpenStore()` so the ingestor accepts packets immediately while the
index builds in the background.
- Legacy `_migrations` gate is preserved by the async fn → DBs that
already completed the sync build stay no-op.
**Annotation convention (`MIGRATIONS.md`):**
Every new `CREATE INDEX` / `ALTER TABLE` / data-rewrite in a migration
file must do ONE of:
1. Run via `Store.RunAsyncMigration(...)` (preferred for backfills).
2. Carry a `// PREFLIGHT: async=true reason="..."` comment directly
above the migration block.
3. Include a `PREFLIGHT-MIGRATION-SCALE: <30s N=<scale>` line in the PR
body.
**TDD pair:**
- Red commit `2c6744cc` — `TestRunAsyncMigration_PendingThenDone`
against a stub helper. Build passes, assertion fails (`async migration
fn did not start within 2s`).
- Green commit `38354f32` — real helper + retroactive fix + docs. Test
green.
**Fixtures (`cmd/ingestor/testdata/preflight-migrations/`):**
- `bad_sync_migration.go` — known-bad sample with no annotation.
- `good_annotated_migration.go` — known-good sample with annotation.
The preflight gate script can be unit-tested against these.
## Gate location (NOT in this PR)
The actual `check-async-migrations.sh` lives in the OpenClaw skills
directory at `~/.openclaw/skills/pr-preflight/scripts/` (separate from
the repo) and is wired into `run-all.sh`. It greps the diff for
new/modified migration blocks and hard-fails (exit 1) on any sync schema
mutation lacking one of the three opt-outs above. The fixtures in this
PR give maintainers a reproducible target.
## Why annotation-discipline, not size detection
You cannot determine table size from a diff. The gate enforces that
every author who adds a schema migration must consciously decide which
bucket it falls into and write that down. That is the cheapest possible
intervention that breaks the cycle.
## Testing
- `go test ./...` in `cmd/ingestor` — all tests pass including the new
`TestRunAsyncMigration_PendingThenDone`.
- Manual: red commit fails on assertion (not build), green commit passes
— verifiable by `git checkout 2c6744cc --
cmd/ingestor/async_migration.go && go test -run TestRunAsync
./cmd/ingestor` from the green commit.
## Preflight overrides
None — clean run after the convention is applied.
---------
Co-authored-by: clawbot <bot@openclaw.local>
Co-authored-by: clawbot <bot@openclaw>
Follow-up to v3.8.3 security train. Found by non-XSS input-validation
audit.
Three findings closed in one PR — all defense-in-depth: medium is
genuinely DoS-only (no data exposure), lows tighten log hygiene and SPA
path handling so future router changes can't silently expose the
filesystem.
## Findings addressed
### MEDIUM — unbounded `limit` on list endpoints
- **What:** four list endpoints accepted `limit=999999999` and passed
the value straight to SQL `LIMIT ?` and Go `make(..., 0, limit)`.
- **Where:** `cmd/server/routes.go` — handlePackets (incl. multi-node
branch), handleNodes, handleChannelMessages, handleAnalyticsSubpaths,
handleAnalyticsSubpathsBulk per-group lim, handleDroppedPackets.
- **Fix:** new `clampLimit(raw, def, max)` helper in
`cmd/server/clamp_limit.go` plus `queryLimit(r, def, max)` HTTP wrapper.
Caps: packets/nodes/channels/dropped = 500, analytics buckets /
bulk-health = 200. Already-clamped endpoints (handleBulkHealth) migrated
to the helper for uniformity. Silent clamp — no response-shape change.
Negative / zero / non-numeric → default.
### LOW — log injection via newline in advert name
- **What:** advert `name` field allows `\n` / `\t` (sanitizeName
intentionally preserves them for display). Logged at two MQTT-ingest
sites, an attacker with publish ACL could forge log lines.
- **Where:** `cmd/ingestor/main.go:659,690`.
- **Fix:** new `sanitizeLogString` in `cmd/ingestor/sanitize_log.go`
strips control bytes < 0x20 and DEL with `?`. Wrapped at the two log
call sites that interpolate `name=` and `observer=`. Stored display
values untouched.
### LOW — SPA static handler depends on default mux path-cleaning
- **What:** `cmd/server/main.go:469` joins `r.URL.Path` to root; safe
today only because gorilla/mux runs `path.Clean` and `http.FileServer`
rejects `..`. A future `SkipClean(true)` or router swap would silently
expose the filesystem.
- **Where:** `cmd/server/main.go` (spaHandler).
- **Fix:** new `isSafeStaticPath` rejects requests whose decoded or raw
path contains `..`, `%2e%2e`, `\\`, or `%5c` with a 400. Legit asset
names with dots (`/app.js`, `/customize-v2.js`, `/themes/dark.css`) are
unaffected.
## TDD
- Commit 1 (red): adds `TestClampLimit`, `TestSpaHandlerPathTraversal`,
`TestSanitizeLogString` with stub helpers — tests fail on assertions
(not build errors), proving they gate the change.
- Commit 2 (green): production fix. Revert the green commit and the red
commit's assertions fail.
## Audit reference
Source: non-XSS input-validation audit dated 2026-06-03 (workspace).
Sibling PR `fix/xss-r2-trace-obs-anl` owns the XSS findings — not
included here.
---------
Co-authored-by: clawbot <clawbot@users.noreply.github.com>
## This PR fixes the stored XSS in full (closes#1536)
Mesh-advertised node names (`adv_name`) and observer names were rendered
into the dashboard DOM **without HTML-escaping** in multiple places —
the same class as the publicly disclosed MeshCore dashboard XSS
(CVE-2026-45323). `adv_name` has no protocol-level validation and the Go
`sanitizeName()` keeps `< > " &`, so a payload like `<img src=x
onerror=...>` reaches the frontend intact and executes.
**I audited every name/sender/text/channel render in `public/` and this
PR escapes all unescaped sinks. There are no known remaining XSS sinks
of this class after this change.**
### Sinks fixed (all escaped via the existing global `escapeHtml`, plus
a local helper for the standalone `area-map.html`)
| File | Sink |
|------|------|
| `app.js` | global search dropdown — node name + channel name |
| `nodes.js` | nodes-table row name; node-detail Leaflet popups (×2) |
| `observers.js` | observers-table name cell |
| `packets.js` | observer-name cells via `obsNameOnly` (×4) + observer
multi-select checkbox label |
| `live.js` | node-filter `<option>` + map marker tooltip |
| `analytics.js` | topology map node tooltip |
| `route-view.js` | hop + union marker tooltips (×2) |
| `area-map.html` | node popups (×2) — added a local `escapeHtml` (file
is standalone) |
### Already-safe (verified, not changed)
`map.js` popups (`safeEsc`), live-feed text (`escapeHtml(preview)`),
packet-detail text, channel messages (`channels.js`), `route-render.js`
popups, `hop-display.js`.
### Why escape at the sink (not the backend)
`sanitizeName()` only strips control chars; HTML-escaping stored names
server-side would be lossy and corrupt legitimate names containing `& <
>`, and break the `meshcore://` deep-links / exports. Output-encoding at
render is the correct OWASP fix and matches `meshcore-card` v0.3.3.
### Tests
- Added 6 `escapeHtml` regression tests including the CVE payload `<img
src=x onerror=alert(1)>` and an attribute-breakout payload.
- `node test-frontend-helpers.js`: **568 passed / 32 failed** — the 32
are pre-existing sandbox limitations (e.g. `AreaFilter is not defined`),
identical to the untouched baseline (562/32). Zero new failures.
### Cache busting
Automatic — the server rewrites `__BUST__` in `index.html` with a
restart timestamp, so no manual bump is needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: CoreScope Bot <bot@meshcore>
Co-authored-by: Kpa-clawbot <bot@clawbot.local>
### Description
This PR addresses several visual and UX issues on the Live page,
specifically focusing on mobile viewport constraints and filter
accessibility.
**Changes:**
1. **Dropdown Clipping Fix**: Previously, the Node, Region, and Area
filters were nested inside `.live-toggles`. On narrow screens,
`.live-toggles` becomes a horizontally scrolling container (`overflow-x:
auto`), which unintentionally clipped the absolute-positioned dropdown
menus for these filters. They have been moved to `.live-controls-body`
as siblings, allowing their dropdowns to correctly break out and overlay
the map.
2. **Cog Positioning**: The settings cog (`#liveControlsToggle`) has
been pushed to the far right of the metrics header using `margin-left:
auto`, creating a cleaner visual separation.
3. **Filter Spacing**: When the controls panel is expanded, a `12px` top
margin is now applied to push the filter buttons further away from the
metrics row for better touch targets and readability.
4. **Test Updates**: The E2E Playwright test for the Area dropdown was
updated to click the cog menu first, matching the new DOM structure.
5. **Area outside cog**: Resolves the initial issue of the area dropdown
being outside of the cog on a mobile display
### Performance Justification
This is a pure HTML/CSS structural refactor. There are no additional
per-item calculations or API calls introduced. Moving the DOM nodes out
of the scrolling container has zero impact on render loop complexity,
and no new JavaScript event listeners were added to the hot path.
### Testing
- [x] Unit tests pass (`npm test`)
- [x] Playwright E2E tests pass (updated to reflect the cog interaction)
- [x] Verified visually in browser (Desktop and Mobile viewports)
Made the suggested changes as listed in the fix path provided by
@Kpa-clawbot
Fix path:
`style.css:1244` `.field-table .section-row td` → `color: var(--text)`
(or new `--section-header-fg`).
`style.css:2620-2631` `.copy-link-btn` → `color: var(--text);`
background/border via `--accent-bg` / `--accent-border` tokens with safe
defaults.
`live.css:987` `.vcr-scope-btn.active` → same token swap; ensure text
remains `--text` on the tinted bg.
`nodes.js:212` `.multibyte-badge` → move inline styles to style.css,
`color:var(--text)`, keep `--accent-bg` background.
When creating the defaults for `--accent-bg` and `--accent-border`, I
chose to go with the default style values embedded in nodes.js, as that
was the safest bet.
We should probably extend the custom themes to include these variables
as well as not to confuse users if they see it. This also causes the
delima of, sometimes the `--accent` is use as the background for
objects, and not `--accent-bg`, example:
`btn active` has background set to` --accent` and border set to
`--accent`.
If we don't extend the config to accept accent-bg and accent-border, we
risk users still making accents of light blue that will be drown out
with the defaults we've set.
Also updated the badge above the multi-byte badge that contains X bytes
of the nodes public key, where X is determined by the path byte length.
This was done because it had styles set that were easy to add to the
styles.css file, to clean up coe. The node-type badge above it is
unfortunately driven by javascript in the nodes.js page, and requires
syling.
**Note:** Accidentally added ghost changes into this push for a second
time. They can be ignored as they were previously merged and shouldn't
have been seen as new.
Rename of ghost to inferred hops as described as partial fix for issue
#1505
Update of ghostDesc in live.js, also mentioned as partial fix for issue
#1505
## Summary
Fixes a visual bug on the Live page where the navigation bar layout
would break, causing the right-side icons (search, theme toggle,
hamburger menu) to be pushed into the middle of the screen.
## Cause
The Live page dynamically injects a "📌" button to let users lock the
auto-hiding header. However, `live.js` was appending this button as a
direct child of the outer `.nav-bar` container.
Because `.nav-bar` uses flexbox with `justify-content: space-between` to
separate the left, center, and right sections, adding a 4th top-level
child threw off the distribution of space, squeezing `.nav-right` toward
the center.
## Changes
- **DOM Placement (`live.js`)**: Modified the injection logic to target
`.nav-right` and use `appendChild()` so the pin button is cleanly nested
at the far right of the existing right-side cluster (past the hamburger
menu).
- **CSS Cleanup (`live.css`)**: Removed `margin-left: auto;` from
`.nav-pin-btn` as it is no longer necessary and could cause spacing
issues inside the `.nav-right` flex container.
## Verification
- Verified the pin button renders seamlessly on the far right of the
Live page.
- Confirmed the outer `.nav-bar` layout strictly maintains its
left/center/right alignment.
- Confirmed there are no test regressions (the E2E test
`test-issue-1510-live-nav-pin-e2e.js` selects by ID and continues to
pass flawlessly).
Closes#1522
## Summary
- Call `history.replaceState` in `doTrace()` after the hash is
validated, so the URL becomes `#/tools/trace/<hash>` and can be shared
directly.
## Change
`public/traces.js` — one line added:
```js
history.replaceState(null, '', `#/tools/trace/${encodeURIComponent(hash)}`);
```
The read path (`init()` picks up the hash from the URL on load) already
existed — only the write path was missing.
When the browser backgrounds the tab,drops frames due to DOM bloat, or
user goes to another page; the uncapped delta time (`dt`) in the
`requestAnimationFrame` loop caused the physics engine to simulate
massive time jumps, making packets appear to fast-forward at 8x speed.
This commit:
- Clamps `dt` to a maximum of 32ms in both the path animation and node
pulse loops to ensure graceful slowdowns during lag.
- Restricts the `VCR.speed` multiplier strictly to `REPLAY` mode so live
packets are not accidentally accelerated.
Kpa-clawbot