mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-25 21:03:42 +00:00
980c5c451539df5fa8ff4247e90b4ce72ff1adba
38
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efitenandClaude Opus 5
|
25f8426d32 |
test: make every suite in tests/e2e run, and tell the truth when it does (#2053)
Closes #2037. Step 1 (#2045) wired in the nine suites that already passed. This is steps 2 and 3: the four that ran and failed, and the five that could not run at all. After this, the count of suites in `tests/e2e` invoked by nothing goes from 18 to 0. ## Step 2 — the four that ran and failed Triaged against a fixture server in CI with full output kept, not the four-line tail the first probe saved. | suite | verdict | |---|---| | `test-packets-scope-column.js` | passes on master today. It failed on 2026-09-18, so something between the two fixed it. Wired in unchanged rather than investigated. | | `test-node-reach-e2e.js` | test wrong. It waited for the reach map whenever any *link* had GPS; `public/node-reach.js` builds the map only when the *node* has coordinates. Guaranteed 10s timeout on a node with positioned neighbours and no position of its own. | | `test-channel-modal-e2e.js` | both failures test-side. The Add button's visible label was shortened to "+ Add" with the accessible name moved to `aria-label` (`public/channels.js:746`); and `.ch-section-mychannels` is conditional on the visitor having added a channel, which has not happened at that point in the suite. | | `test-touch-targets.js` | three test-side, two a product finding. | The three test-side touch-target failures were the harness measuring controls that are not shown: `.compare-btn` (the CTA was removed in #1646, and `style.css` says so), `.ch-back-btn` (`display:none` outside the mobile channels layout), and `.filter-toggle-btn` (`display:none` on mobile since #1461; the control shown is the navbar mirror, which `mobile-page-actions.js:70` builds as a `.nav-btn`, so it was already measured). All three are dropped from the table with the reason recorded in the file. The remaining two are **not** a test problem: `.nav-btn` and `.ch-icon-btn` are each declared twice in `public/style.css`, 48px in the touch-target block and 44px in their own component rule, and the later one wins. Rather than lower the blanket or hide the failures, the suite now has `DEFAULT_MIN = 48` plus a `MIN_OVERRIDES` table holding those two at their effective 44, so a third selector dropping to 44 still fails the build. The contradiction is **#2052**, with both ways out costed; the override entries should go when it is settled. ## Step 3 — the five that could not run None is deleted. I checked each selector and seam against the product before deciding, and every one still targets a surface that exists and that nothing else covers. Four were written against `@playwright/test`, a runner the project neither installs nor uses anywhere else. Adopting a second runner for twelve tests costs more than porting them, and the precedent is already set: `test-path-inspector-coverage-e2e.js` exists, as its own header says, because `test-path-inspector-e2e.js` could not run. So they are ported to the plain-node Chromium pattern the other 109 suites use. - **`test-issue-1522-trace-url-sync-e2e.js`** — the trace hash in the URL, both directions. `test-e2e-playwright.js` covers that the page loads and searches; it never looks at the URL, which is the whole of #1522. - **`test-marker-outline-weight.js`** — the canvas pulse ring never thins below 2px. There is no CSS rule to read and axe cannot see inside a canvas, so sampling the seam is the only way. Added a guard that the ring was actually visible, so the weight check cannot pass vacuously on a pulse that never rendered. - **`test-pr-1490-live-map-gpu-animations-e2e.js`** — the queue drains, the engine sleeps again, the fading trails stay under the cap of 5, and the canvas sits on `animationsPane` rather than under the markers. - **`test-path-inspector-e2e.js`** — reduced to what nothing else covers: the map side pane, the `/#/traces/<hash>` redirect, the tools landing. Its standalone-page test duplicated the wired coverage suite and is dropped. Its "switching candidate clears prior polyline" case ended after the click with a comment and no assertion, which is the same green-but-empty problem this issue is about; it now compares path counts, and skips loudly when the fixture yields too few candidates. The fifth, **`test-table-sort.js`**, needed `jsdom`, which was declared nowhere. It is a unit test of `public/table-sort.js` filed under `tests/e2e`, so: `jsdom` is a devDependency (lockfile updated, `npm ci` stays consistent), the file moved to `tests/unit/`, and the `domIntegration` group in `scripts/non-unit-tests.json` is gone with its only member. It runs 22 tests. 20 passed immediately; 2 had rotted, because #1648 M2 replaced the up/down glyphs with Phosphor sprites and the direction moved out of `textContent` into the `<use href>`. Those two now read the sprite ref and the `aria-sort` value, so they also guard the accessible announcement. ## Verification `tests/unit/test-table-sort.js` 22/22 and `test-test-inventory.js` pass locally; the E2E suites need a fixture server, which I cannot build here (no cgo toolchain since #1992), so CI is their first run as committed. The triage above was measured in CI, not assumed. ## Not done The per-assertion skips named in the second comment on #2037 are untouched: the two flaky packet-detail cases, the fixture-data ones, and the two `clientRxCoverage` suites that skip wholesale while reporting success. Those need a fixture deployment with coverage enabled, which is its own change. I have not opened it. Option 2 from the issue, making `test-test-inventory.js` require a `deploy.yml` line for every `tests/e2e` file, is also not here. It is the right guard and it is now enforceable, since the list is finally at zero, but it belongs in its own change where a red build means what it says. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
5c016a210a |
fix(analytics): show a building state on the distance index's 202, and stop caching it (#2051)
Closes #1997. ## What was wrong `/api/analytics/distance` answers `202 {status:"building", retry_after_seconds:5}` with no `summary` until the lazy index (#1011) has been built. 1. `renderDistanceTab` read `data.summary.totalHops` straight away. The TypeError is caught by the tab's own try/catch, so it never reaches `window.onerror`: it is painted into the tab as `Failed to load distance analytics: Cannot read properties of undefined (reading 'totalHops')`. 2. `api()` caches any `res.ok` body, and `res.ok` is true for 202, so the placeholder was stored for the `analyticsRF` TTL. Even a correct retry read the cached "building" body back. That is the half that made the broken state outlast the index build. ## What changed - `public/app.js:173` skips the cache write when `res.status === 202`. A 200 still caches, unchanged. - `public/analytics.js` renders a building notice and retries itself, honouring `retry_after_seconds` clamped to [1s, 30s]. The timer is cleared on tab switch and in `destroy()`, and a new render supersedes a pending retry, so two renders cannot write into the same tab. The notice uses `.text-center`/`.text-muted` rather than the `.spinner` class used at `analytics.js:1683`, because `.spinner` has no CSS anywhere in the repo and renders nothing. ## Tests `tests/unit/test-issue-1997-distance-building.js` (7 assertions, wired into `test-all.sh`) pins the two pure decisions the renderer makes and `api()`'s refusal to cache a 202 while still caching a 200. Red-run on the unfixed sources: 6 of 7 fail, and the "a 200 is still cached" control stays green. `tests/e2e/test-issue-1997-distance-building-e2e.js` (classified in `scripts/non-unit-tests.json`, invoked from `deploy.yml` with `CHROMIUM_REQUIRE=1`) serves both responses by route interception, so it does not depend on whether the server under test has an index built. It asserts: the building state appears, the tab does not paint the error text, a retry arrives with no interaction, the retry replaces the placeholder once the server answers 200, and no retry fires after leaving the tab. Check (2) deliberately asserts on the rendered text and not on `pageerror`: the TypeError is caught, so a `pageerror` assertion would pass on the broken build too. ## Verification No local cgo toolchain here since #1992, so I could not build a server to run the E2E against. Instead I ran its five steps in Playwright against a live instance with this branch's `public/app.js` and `public/analytics.js` injected in place of the deployed ones (both files are byte-identical between that instance and upstream master, so the injection is faithful): | check | deployed build | this branch | |---|---|---| | (1) building state shown | fail | pass | | (2) not rendered as data | fail | pass | | (3) retried on its own | fail (1 request) | pass (2 requests) | | (4) real payload after retry | fail | pass | | (5) no retry after leaving the tab | pass | pass | (5) passes on the broken build too: it schedules no retry at all, so it is a control and only means anything together with (3). The committed E2E suite has not been run as committed. CI is its first real run. ## Not done - The server still recomputes on every 202 poll rather than signalling readiness. - No other analytics tab was audited for the same assume-a-summary pattern. - One pre-existing unit suite (`test-preflight-xss-gate.js`) fails on this Windows machine with a cp1252 `UnicodeEncodeError` from its Python helper, on a clean tree as well as with this change. Unrelated, and green on Linux CI. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
liquidraver
|
84eba41592 |
fix(live): anchor the legend toggle to .live-page so the VCR bar stops eating its clicks (#2049)
The PACKET TYPES legend could not be dismissed on Live: clicking the palette toggle did nothing, while document.querySelector('#legendToggleBtn').click() from the console worked. That asymmetry was the diagnosis, since a synthesised click skips hit testing.
Residual half of #1833. That fix corrected the button's offset to calc(var(--vcr-bar-height) + 10px) but left the anchor: .legend-toggle-btn was position: fixed, so the offset resolved against the viewport, while the .vcr-bar it clears is absolute inside .live-page, whose height subtracts --bottom-nav-reserve (56px + safe-area at <=768). The two anchors disagreed by exactly the reserve, dropping the button into the bar's band, and .vcr-bar at z-index 1000 against the button's 500 took every real click. The window is 641-768px: below 641 both buttons are display:none, above 768 the reserve is 0, which is why neither the desktop nor the phone layout test saw it.
Both buttons are now position: absolute, resolving against the same containing block as the bar. .feed-show-btn carried the identical defect but missed the bar by 16px because --legend-toggle-stack parks it a row higher, so that change prevents a break rather than repairs one.
Three review rounds, each verified rather than argued:
- The new E2E was classified in non-unit-tests.json but had no deploy.yml line, so it would never have run (#2037). Now wired in, and it covers both buttons.
- The '.feed-show-btn is display:none at <=768' claim was wrong (it is <=640), which the author measured and corrected.
- Its first real execution failed on its own setup: getComputedStyle().getPropertyValue() on an unregistered custom property returns the unevaluated declaration ('calc(56px + 0px)'), so the non-zero check could not work. It now asserts the measured gap instead, and deliberately does not pin 56 because the gap measures 58 on CI.
Final run: 14 passed, 0 failed. The discriminating assertion is (f), each button's offset net of the bar being identical at 720 and 1440 (11px and 123px slack); under the old anchor those differ by the reserve. (g) pins the desktop layout unmoved.
Merged by the interim maintainer without a second human reviewer: CI and the review above are the independent checks.
Fixes #1833
|
||
|
liquidraver
|
bbf54cfe1b |
fix(rx-coverage): open at the configured map default, with its own saved viewport (#2033)
The coverage page opened at a hardcoded [51.0, 4.8] zoom 8 regardless of deployment, ignoring /api/config/map (#2032). It now follows the same precedence as the main map (URL hash, then saved position, then /api/config/map, then [37.6, -122.1] zoom 9) and persists its own position across visits, syncing lat/lon/zoom into the hash so a view is shareable. The saved position lives under its own key, rx-coverage-view, and the page never reads or writes the main map's map-view: sharing the configured default was the bug, sharing the session position was not. Both suites assert map-view stays untouched after a pan, so reintroducing a shared write fails instead of passing quietly. Also fixed here: selectedRx is now percent-encoded into the hash, and a generation counter stops a late /api/config/map response or a stale 150ms layout timer from building a map for a page that was already left. Reviewed twice. Verified by mutation rather than by reading: writing map-view too, ignoring the saved coverage position, and dropping the /api/config/map fetch each make the unit suite exit 1, so it covers the feature, the fix and the rejected alternative. The deploy.yml invocation was confirmed to land inside Run Playwright E2E tests (fail-fast) by parsing the workflow, and CI run 35261689694 is the E2E suite's first real execution: Go prints 'RX coverage viewport regressions OK' and Playwright prints 'RX coverage viewport browser regressions OK'. Worth recording for the next reviewer: that E2E asserted localStorage.getItem('map-view'), so wiring it into deploy.yml without updating the assertion would have turned the job red on its first ever run. It was registered in scripts/non-unit-tests.json but invoked by nothing, the gap tracked as #2037. Merged by the interim maintainer without a second human reviewer: CI and the mutation checks above are the independent checks. Fixes #2032 |
||
|
Alex B
|
893773338e |
chore(tests): move root test-*.js into tests/unit and tests/e2e (#2036)
Moves 290 root test-*.js into tests/unit (177, listed in test-all.sh) and tests/e2e (113, classified in scripts/non-unit-tests.json), per #1981 and PR-D of #1385. Root goes from 348 entries to 48. test-all.sh and test-fixtures/ stay put. The inventory guard now fails if a test reappears in the root or sits in the wrong folder.
Verified independently of the diff: the invoked sets are unchanged (test-all.sh 177 before and after, deploy.yml 96 before and after, both identical as sets), and a full local run of test-all.sh on master and on the branch produced 4702 output lines each whose only differences are absolute paths, stack-trace line numbers shifted by the REPO_ROOT line, the inventory wording and two perf ratios. The guard was mutation-checked: a test back in the root, a unit suite in tests/e2e, and a suite dropped from test-all.sh each make it exit 1. CI run 35246304316 ran 97 suites from tests/e2e and is green.
Follow-up
|
||
|
Sylvain Rabot
|
a2ea18f778 |
perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992)
Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite
3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4),
and pays the resulting cross-compilation cost with `zig cc`.
Draft because the riskiest part of this deletes rows — see [Please
review this part first](#please-review-this-part-first) — and because
three things remain unverified at the bottom.
`modernc.org/sqlite` is a transpilation of the C amalgamation. This repo
is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out
neighbour/topology/analytics queries per request, and it pays for that
transpilation on exactly those paths. Head-to-head on the same
120k-transmission / 240k-observation database, running our own hot-path
SQL under both drivers (Apple M4, `-count=5`, medians):
| workload | modernc | mattn | |
|---|---:|---:|---|
| chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms |
**2.3×** |
| aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×**
|
| 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** |
Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the
chunk load.
**Superseded by a production run.** @efiten measured both drivers on a
real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 —
as server-only containers against the same live volume, one at a time,
with round 2 reversing the order so the page cache favours the old
driver:
| | audit 7d | audit 24h | background fill (13 chunks) | start →
/api/health |
|---|---:|---:|---:|---:|
| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |
Warm, the old driver improves to 13.46s on the 7d audit and still loses
by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against
0.037s — nothing.
**So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.**
The shape the harness predicted holds — scans and joins gain, small
lookups do not — which is more reassuring than the magnitude would have
been. Quote these numbers.
**The counterweight**, cold and native on that machine: a build goes
from **52s to 163s**. An instance that builds its own image pays that
per deploy.
## The build is cgo now, and one thing about that is a trap
**`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary
dies on its first query with `go-sqlite3 requires cgo to work. This is a
stub`. A green build is not evidence of anything here, which is why
`AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely
cannot cross-compile any more.
A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc
-target {x86_64,aarch64}-linux-musl` and links static, so each artifact
stays a single self-contained file and the `alpine:3.20` runtime no
longer depends on the base image's libc at all.
`-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl
objects zig links in, and without it the server binary is 19.8MB instead
of 12.1MB.
The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU
for compilation — and gains a checksum-pinned zig plus BuildKit cache
mounts. The mounts are not a nicety: without them an image build
recompiles the amalgamation from cold and takes over half an hour.
## Please review this part first
`internal/dbschema/dedup_index.go` **deletes observation rows**. It is
the one part of this change that can lose data, and it exists because
the migration exposed a real bug rather than causing one.
`stmtInsertObservation` resolves its `ON CONFLICT` against
`idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created
inside the branch that creates the `observations` table for the first
time. Any database whose table predates that branch never got one, so
the UPSERT had no conflict target. modernc failed on the first insert;
mattn fails at `OpenStore`. Same bug, found earlier.
Creating the index unconditionally repairs it — but the index is what
was supposed to prevent duplicates, so a database that never had it can
already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this
repo holds one.** So duplicates are collapsed first. Refusing is not the
safer option: without the index the ingestor cannot prepare its UPSERT,
so it cannot start at all.
Replaying that UPSERT faithfully is subtler than it looks, and a first
cut of this got it wrong twice:
- `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a
group in id order the survivor keeps the **last** non-NULL value. Taking
the first silently discarded newer readings.
- The UPSERT names exactly five columns (`snr`, `rssi`, `score`,
`raw_hex`, `resolved_path`). Every other column must keep the surviving
row's own value; merging those too invents history the ingestor would
never have written.
Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split
apart, a writer inserting a duplicate in the gap fails the index
creation while leaving the deletions committed — rows destroyed and no
index to show for it.
Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**,
holding the write lock throughout, once, at ingestor startup before MQTT
subscribe. Materialising the duplicate-group scan once rather than per
column took that from 9.7s; the pathological case (400k of 600k rows
duplicated) is 5.7s, slightly worse than the 4.2s it was before that
change.
## Four more behavioural differences
Full detail in `docs/sqlite-driver-migration.md`. Briefly:
**Statement preparation is eager.** modernc's `newStmt` stored the SQL
and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`,
so SQL naming a missing table fails at *open*. 59 server tests failed on
this alone, all fixtures with partial schemas. `OpenDB` keeps failing
loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the
fixtures now declare what they are prepared against via
`ensurePreparable`. This also exposed nine `nodes(pubkey …)`
declarations across seven files, where production has only ever had
`public_key` — lazy compilation had hidden the mismatch for as long as
it existed.
**`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to
NORMAL and executes the pragma unconditionally, where SQLite's own
default (what modernc left alone) is FULL. In WAL mode that weakens
durability under power loss. Pinned in `dbschema.WriterDSN`, which both
writers now share — `cmd/migrate` kept a bare path at first and so
quietly wrote at NORMAL, which is what a second copy of a DSN buys you.
**The DSN dialects are mutually invisible.** modernc understood only
`_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither
errors on the other's form — a driver-only rename would have dropped
every pragma in silence. `_journal_mode=WAL` is also gone from the
server's read handle: modernc ignored it, mattn honours it, and setting
`journal_mode` on a read-only connection is a write. Dropping
`_busy_timeout` with it costs nothing, since mattn already defaults to
5000ms — which means the read handle finally *gets* the busy timeout it
had silently lacked.
**`mode=ro` survives for a non-obvious reason.** mattn always passes
`READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what
makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the
#1283/#1289 invariant holds with no build flags — but it depends on the
`file:` prefix. `cmd/decrypt` had been building its DSN without one, so
its `mode=ro` had never applied and a missing path was created
read-write. Fixed in passing; never a migration regression.
## What did not change
No modernc-specific API was in use: no `RegisterFunction`, no
`*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No
`time.Time` is ever bound as a query argument, so driver time handling
is not in play. Both drivers convert declared
`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so
`/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an
earlier draft "fixed" that with a `CAST` and would have been the
regression.
## Tests and CI
New regression tests, each written because something got through without
it:
- `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge
ordering. The original test used complementary NULLs, which passes
whichever direction you pick, which is why the bug survived it.
- `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation
must roll the deletions back.
- `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma,
read back through the store's own connection. A separate `sqlite3`
session or the startup log line would prove nothing.
- `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which
now rests on a detail of the driver's C wrapper.
- `TestEnsurePreparableMatchesPrepareStatements` — fails when a new
prepared statement outgrows the fixture helper.
CI gains test execution for `cmd/migrate` and `internal/dbschema`, which
had none and both open the database. A PR-time two-arch build plus an
arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without
it nothing on a PR would exercise zig, static musl linking or arm64, and
the first signal would arrive on master. `cache-dependency-path` widens
from 2 of the 5 tracked `go.sum` files to all of them.
`make test` passes across all 14 modules, `cmd/server` also under `-race
-count=2` with no failures and no races. `gofmt` and `go vet` clean.
Release-routing and Dockerfile COPY-invariant gates pass.
## Verified by running
- All 8 cross-builds static and correct-architecture; both arches of the
container image built, exported and run under QEMU, serving
`/api/health` and `/api/nodes` against a 2.9M-observation production
snapshot.
- The `migrate` binary repairing that snapshot's duplicate on bare
Alpine.
- `CGO_ENABLED=0` producing a binary that builds and then fails on first
query.
## Not verified
- ~~The 2–2.3× figures come from a standalone harness, not this load
under the old driver.~~ **Closed** by @efiten's production run above,
which also corrected the multiplier.
- SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no
total `ORDER BY`.
- Sustained live ingest through the new writer DSN, and the duplicate
collapse against a database an ingestor is actively writing to. Verified
against a static snapshot only, and the collapse is measured at 4.1s on
2.4M synthetic rows with 5 duplicates — well short of an 11M-row
instance. @efiten has offered a staging instance taking real MQTT
traffic; **this is the item to close before the PR leaves draft.**
An earlier revision of this branch shipped the dedup merge in the wrong
direction with a green test suite, and review then found three more
things in the same file: the repair gated on an error string, a
non-atomic TEMP table drop aimed at the wrong connection, and a deletion
whose only record was a row count. All fixed in
|
||
|
n30nex
|
cd9b4c04d0 |
test: unify frontend test runs and prevent inventory drift (#1965)
Make `test-all.sh` the authoritative standalone frontend runner for npm and CI. Restore stale assertions and reject missing, duplicate, removed or undocumented inventory entries. Fixes #1858. Rebased onto `a2f039d4`. Retains release-routing, map scope-state and Scope Audit stylesheet tests, adds `test-packets-local-channels.js` to the sorted runner, and classifies `test-neighbor-map-btn-clip-e2e.js` under browser. Its separate CI browser step is preserved. Inventory: 280 root suites, 167 standalone, 113 requiring separate setup. The icon repair fixes two suites red on master: `test-issue-1648-m2-emoji-scan.js` and `test-issue-1648-m6-final-sweep.js`. Node/live configured-scope confirmations now use the existing accessible Phosphor check sprite. Values, visibility conditions and scanner assertions are preserved. - Red evidence: `89e45a9` inventory assertions; `4e255df` accessible confirmation assertion. The latest rebase also reproduced both unclassified-file failures before adding their entries. This follow-up only changes runner/classification configuration and counts; no test files modified. - Local validation: all 167 standalone suites; 27 M2 browser checks and 16 neighbor geometry checks in Chromium. Syntax, whitespace, PII, CSS-variable and XSS checks passed. - Browser coverage includes populated/empty/null configured scopes in node and live views. - No new dependencies, requests, application settings or Go changes. Workflow outside the unit step matches master. - Windows validation uses process-local UTF-8 settings. Encoding and node-reach confirmation follow-ups remain separate, as requested. ## Preflight override External `run-all.sh` is unavailable; applicable repository checks were run directly. |
||
|
|
a2004351d3 |
fix(#1684): staging disk monitor + cleanup cron (#1686)
## Summary Adds a staging VM disk-usage monitor + daily cleanup cron, fixing the gap surfaced by #1684 (staging hit 100% disk during a hot-patch, no alert, no cleanup). ## What landed - **`scripts/staging/disk-monitor.sh`** — parses `df -P <mount>`, classifies usage `<80 ok / >=80 warn / >=90 error / >=95 alert`, emits to stderr + journald via `logger -p`, exits non-zero on `error|alert` so the systemd unit surfaces as failed. - **`scripts/staging/disk-cleanup.sh`** — daily prune of `/tmp` snapshot patterns (`*.db`, `staging-snap.*`, `cs-*`, `node-compile-cache`) older than 7d + `docker builder/image prune --filter until=72h --filter label!=keep`. Honors `CORESCOPE_CLEANUP_DRY_RUN=1`. - **`scripts/staging/test-disk-monitor.sh`** — pure-bash unit tests for the testable helpers (22 cases covering threshold boundaries, df parsing, invalid input, severity→priority mapping). - **`DEPLOY.md`** — install one-liner with full inline systemd unit + timer content (15-min monitor, daily 03:30 cleanup). Uses `<STAGING_HOST>` placeholder. - **`.github/workflows/deploy.yml`** — wires `test-disk-monitor.sh` into the Go build & test job. ## TDD - Commit `26185967` (RED): tests against stub helpers — `PASS=5 FAIL=17` on assertions. - Commit `d31a1082` (GREEN): real helpers — `PASS=22 FAIL=0`. ## Phase 3 — `staging-snap.db` root cause `grep -rn staging-snap.db cmd/ public/ scripts/` → **zero hits**. The 4.4 GB orphan was a manual debug artifact, not committed code. The cleanup retention rule prevents recurrence. Partial fix for #1684 — leaves issue open for operator to verify install on staging and confirm alert fires at 85%. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: clawbot <bot@openclaw.dev> |
||
|
Kpa-clawbotandopenclaw-bot
|
55e4d957b1 |
M1: emoji → Phosphor Icons — top-nav, mobile nav, Compare (#1648) (#1649)
Red commit:
|
||
|
|
63bfa3d910 |
feat(security): detect CDN-fronted deployment + document bypass requirement (closes #1561) (#1564)
Closes #1561. Follow-up to #1551. ## Why #1551 added `Cache-Control: no-store` to all `/api/*` responses. That's sufficient for CDNs that honour origin headers (Varnish, nginx). It is **not** sufficient for Cloudflare zones where Cache Rules / Page Rules override origin Cache-Control. Field evidence from the meshat.se diagnosis (2026-06-04): observers behind Cloudflare were returning `cf-cache-status: HIT` with `age` up to ~6 hours despite the origin emitting `no-store`. The CDN was caching per zone policy and ignoring the upstream directive — exactly the failure mode #1551 cannot reach. The application has no way to inject CDN rules; the only durable fix is operator-side. This PR makes that operator step discoverable and verifiable. ## What ### Server-side detection (log-only) `cmd/server/cdn_detection.go` adds a middleware wired into the `/api/*` chain after `noStoreAPIMiddleware`. On the **first** request bearing any CDN-typical header (`CF-Connecting-IP`, `CF-Ray`, `X-Forwarded-For`, `X-Real-IP`, `Fastly-Client-IP`, `True-Client-IP`) it logs: ``` [security] WARNING: detected request via CDN (CF-Ray header present). Ensure /api/* is bypassed in your CDN config — see docs/deployment-behind-cdn.md. Cached API responses cause observer-flap and incorrect dashboards. ``` `sync.Once` guarantees the warning fires at most once per process boot. The middleware never blocks, never modifies the response, never adds headers. Detection is observational only — operators who run behind a CDN without bypass have a real bug; the warning is appropriate. ### Operator documentation `docs/deployment.md` gains a new **"Behind a CDN (Cloudflare, Fastly)"** section covering: 1. Curl verification command + healthy vs unhealthy output examples 2. Cloudflare Cache Rule creation (URI Path starts-with `/api/` → Bypass cache) 3. Legacy Page Rules equivalent 4. Fastly note 5. Re-verification 6. Meaning of the startup log warning 7. Why we can't fix this server-side `docs/deployment-behind-cdn.md` is the canonical path the log message references — it's a short TL;DR that links back to the full section. ### Healthcheck script `scripts/check-cdn-bypass.sh` — POSIX sh, no dependencies beyond curl + grep + awk. Operators run: ```sh scripts/check-cdn-bypass.sh https://your-domain.example.com ``` Exits `0` with `OK: no CDN caching detected ...` or `1` with a precise diagnostic naming the offending header (`cf-cache-status: HIT` or stale `age`). ## TDD - **Red commit `e90ccaba`** (`test(security): RED ...`) — `cmd/server/cdn_detection_test.go` (4 Go tests + 6 subtests for each header) and `scripts/test-check-cdn-bypass.sh` (3 shell harness cases). Middleware stub returns `next` unchanged so tests compile and fail on assertions, not build errors. - **Green commit `5e6a60b5`** (`feat(security): GREEN ...`) — real middleware, wiring in `routes.go`, healthcheck script, doc. ## Deliverables | File | Status | Purpose | |------|--------|---------| | `cmd/server/cdn_detection.go` | new | middleware + sync.Once warning | | `cmd/server/cdn_detection_test.go` | new | 4 Go tests (1 stand-alone + 1 silence + 1 once + 1 table-driven over 6 headers) | | `cmd/server/routes.go` | modified | `r.Use(cdnDetectionMiddleware)` after no-store | | `docs/deployment.md` | modified | TOC entry + "Behind a CDN" section | | `docs/deployment-behind-cdn.md` | new | canonical path referenced by log message + script output | | `scripts/check-cdn-bypass.sh` | new | operator-runnable healthcheck | | `scripts/test-check-cdn-bypass.sh` | new | shell harness with fake curl | ## What this PR explicitly does NOT do - Does not block requests based on CDN detection (log-only). - Does not enforce CDN bypass (impossible — operator-controlled). - Does not spoof, strip or modify CDN headers. - Does not add CSP / HSTS / other security headers (out of scope). - Warning is not configurable — operators behind a CDN without bypass have a real bug, surfacing it is correct. ## Verification - `go test ./...` in `cmd/server/` — full suite green. - `sh scripts/test-check-cdn-bypass.sh` — 3/3 pass. - Preflight checklist — all 11 gates clean (PII, branch scope, red commit, CSS vars, CSS self-fallback, LIKE-on-JSON, sync migration, async-migration annotation, XSS sinks, img/SVG ratio, themed-img/SVG, fixture coverage). --------- Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: clawbot <bot@clawbot.invalid> |
||
|
|
e4a21fc9ab |
feat(preflight): hard-fail gate on unescaped node-controlled HTML sinks (#1543)
## Summary Closes the "XSS regression in newly-added sink" class. Follow-up to #1537 (10 stored-XSS sinks in node names) and the post-#1537 audit (TRACE-1, OBS-1, ANL-1 — 3 additional HIGH XSS in files #1537 didn't touch). After those fixes land, the project still has **zero automated catch for the next one**. Every future PR can re-introduce the same class freely. This PR closes that gap with a hard-fail pr-preflight gate that runs at PR-creation time and in CI. ## What the gate does A NEW or MODIFIED line in the PR diff under `public/**/*.{js,html}` is flagged when it matches any of these sink patterns: | Pattern | What it catches | |---|---| | `.innerHTML = \`…\`` / `'…'` | template-literal or string-concat HTML injection | | `insertAdjacentHTML(…, \`…\`)` | DOM-adjacent injection | | `.bindPopup(\`…\`)` / `.bindTooltip(\`…\`)` | Leaflet popup/tooltip injection (the OBS-1 class) | | `.setAttribute('on<event>', …)` | inline event-handler injection | | `.setAttribute('href'\|'src'\|'action'\|'formaction', <interp>)` | `javascript:` URI class | For each flagged line, the gate then walks the dynamic substring (`${…}`, post-`+`, or `setAttribute` value arg) and only fires if it interpolates an identifier from the node-controlled allowlist (`name`, `observer`, `sender`, `pubkey`, `body`, `hash`, …). This keeps the regex off static CSS classes like `text-center`. A flagged line is accepted (no fail) when ANY of: - **(a)** wrapped in `escapeHtml(` / `escapeAttr(` / `safeEsc(` / local `esc(` — the audited helpers - **(b)** a same-PR `test*.js` file DOM-greps the audit payload (`' onfocus=` or `onerror=alert`) AND references the sink file's basename - **(c)** the PR body carries `PREFLIGHT-XSS-OPTOUT: <file>:<line> reason="…"` — explicit author opt-out logged for reviewer attention Otherwise: **HARD FAIL** with `file:line: flagged: <token>` plus a suggested fix. ## Split - **Skill directory** (local, no PR): - `~/.openclaw/skills/pr-preflight/scripts/check-xss-sinks.sh` — canonical gate - `~/.openclaw/skills/pr-preflight/data/xss-node-controlled-fields.txt` — allowlist (27 identifiers, easy to extend without a repo PR) - wired into `~/.openclaw/skills/pr-preflight/scripts/run-all.sh` - **This PR** (in repo): - `testdata/preflight-xss/` — fixtures (`bad-1..bad-3`, `good-1..good-2`, `test-good-2.js`) - `scripts/check-xss-sinks.sh` — local mirror of the canonical gate, so CI can exercise the gate without depending on the skill dir - `test-preflight-xss-gate.js` — Node test wrapper that asserts bad fixtures fail (exit 1) and good fixtures pass (exit 0) - `public/app.js` — `escapeHtml` docstring marked CANONICAL with links to the enforcing gate - `.github/workflows/deploy.yml` — invoke `node test-preflight-xss-gate.js` alongside the existing `test-xss-escape-sinks.js` ## TDD red → green | | Commit | Test result | |---|---|---| | **Red** | `test(preflight-xss): RED — fixtures + assertion wrapper for XSS sink gate` | `test-preflight-xss-gate.js` exits 1 — bad fixtures unexpectedly pass because `scripts/check-xss-sinks.sh` is a no-op stub. Genuine assertion failure (not a build error). | | **Green** | `feat(preflight): GREEN — implement XSS-sink check + escapeHtml docstring` | stub replaced with real check; all 5 fixtures behave as expected. | The red commit ships a working stub script so the test runs to completion and fails on an **assertion**, not on a missing-file error. ## Coverage proof — would the gate have caught the originals? - **PR #1537 (10 sinks):** synthetic file from the deleted lines of #1537 → gate flags `n.name` in `innerHTML \`tpl\`` and two `bindPopup(\`…${n.name}\`)` lines. Yes, the gate would have caught these the moment they hit a PR diff. - **Post-#1537 audit:** - **TRACE-1** (`traces.js` `${e.message}` / `${urlHash}` in innerHTML): yes — the `hash`/`urlHash` tokens are allowlisted and the innerHTML template-literal pattern matches. - **OBS-1** (`observer-detail.js` URL fragment + MQTT fields into innerHTML / bindPopup): yes — the `observer`, `text`, `hash` tokens are allowlisted and both sink patterns match. - **ANL-1** (`analytics.js` attribute-mutation roundtrip): yes for `setAttribute('on*', …)` and `setAttribute('href', \`…${interp}…\`)` patterns. (Note: pure innerHTML lines with only `${e.message}` are not node-controlled and are intentionally not flagged.) ## Allowlist (initial 27 identifiers) ``` adv_name name observer observer_name sender from_node channel channel_name model firmware client_version radio iata hopNames nodeLabel obsName n.name o.name obs.name public_key pubkey area_key region_name text body message preview hash urlHash ``` Extend in `~/.openclaw/skills/pr-preflight/data/xss-node-controlled-fields.txt` whenever a new node-controlled field surfaces in an audit — no repo PR required. ## Hard rules respected - No build step, no ESLint plugin, no AST analysis — grep + heuristics + opt-out escape valves - Hard fail (exit 1), not warning-only (exit 2) - PII preflight grep on every commit + this PR body - Same split as the sibling migration-gate PR ## Three-axis merge-readiness - **Mergeable:** yes — branch is clean off `origin/master`, no conflicts - **CI:** will report on push; red commit expected to fail, green commit expected to pass - **Threads:** none open yet (new PR) --------- Co-authored-by: meshcore-bot <bot@local> Co-authored-by: mc-bot <bot@meshcore.local> Co-authored-by: corescope-bot <bot@corescope> |
||
|
efitenandClaude Sonnet 4.6
|
8151185ede |
fix(ci): Dockerfile COPY invariant check — prevent missing internal/<pkg> Docker failures (#1316) (#1432)
## Summary - Adds `scripts/check-dockerfile-internal-pkgs.sh`: reads `replace => ../../internal/<pkg>` directives from `cmd/server/go.mod` and `cmd/ingestor/go.mod`, then verifies each referenced package has the correct number of `COPY internal/<pkg>/` lines in `Dockerfile` (one per builder section that needs it) - Wired into CI as a step in the `go-test` job, before CSS lint — runs on every PR, adds ~0.1s - Prevents the recurring failure pattern (#1316): new `internal/<pkg>` added to go.mod but COPY line forgotten in Dockerfile; non-Docker CI passes, Docker build fails after merge with a cryptic module error Key details: - Counts COPY occurrences per package: if a pkg is referenced in both go.mods (both binaries need it), it must appear in at least 2 builder sections - Anchored regex: only matches actual `replace` directives (not comments) - Anchored grep: skips commented-out `COPY internal/...` lines Closes #1316. ## Test plan - [ ] Run `bash scripts/check-dockerfile-internal-pkgs.sh` locally — exits 0 on current Dockerfile - [ ] Manually remove a `COPY internal/perfio/` line from Dockerfile → script exits 1 with a clear error - [ ] CI step visible in the `go-test` job on this PR 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Kpa-clawbotandopenclaw-bot
|
e267fb754d |
fix(ci): aggregate e2e pass/fail across all suites instead of broken digits-before-slash regex (#1298)
RED |
||
|
Kpa-clawbotandOpenClaw Bot
|
43203b09b7 |
fix(#1249): IATA badge missing on fixture + mobile clipping (#1252)
Failing test commit: `bdb4eefb` (added in #1189 R1) — original CI failure: https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995819598 Fixes #1249. ## Root cause Two independent bugs surfaced by the same E2E test: 1. **Fixture join broken.** `scripts/capture-fixture.sh` wrote the text observer hash into `observations.observer_idx`, but the v3 join in `cmd/server` is `observers.rowid = observations.observer_idx`. The join silently nulled out `observer_id` / `observer_iata` for every packet. 2. **Mobile clipping.** `.col-observer` had `data-priority=3` (hides at ≤1024px) and was in the narrow-viewport `defaultHidden` list, so at 375px the cell collapsed to `display:none` and `.badge-iata` had a 0×0 box. ## Changes - `test-fixtures/e2e-fixture.db`: remap `observer_idx` text hash → integer rowid (500/500 rows resolved). - `scripts/capture-fixture.sh`: build an `observer_id → rowid` map before insert; skip rows whose observer isn't in the fixture. Comment explains the trap. - `public/packets.js`: bump `.col-observer` priority `3 → 1` and drop `observer` from narrow-viewport `defaultHidden`. ## Verification All three sub-tests in `test-observer-iata-1188-e2e.js` pass locally against the freshened fixture. `curl /api/packets?limit=5` returns real IATA codes (OAK / MRY / SFO) instead of empty strings. Co-authored-by: OpenClaw Bot <bot@openclaw.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
eae1b915ca |
feat: load Aldrich webfont for #1137 logo SVG (#1138)
Adds Aldrich webfont so the merged #1137 logo renders in the intended typeface. ## Problem The inline SVG logo merged in #1137 declares `font-family="Aldrich, monospace"` in `public/index.html` and `public/home.js`, but the page never loaded the Aldrich font face. Browsers silently fell back to monospace. ## Fix Self-hosted webfont: - `public/fonts/aldrich-regular.woff2` — Regular 400, ~16KB, downloaded from Google Fonts (latin subset). Self-hosted to avoid third-party CDN dependency, privacy concern, and FOUT delay. - `@font-face` declaration added at the top of `public/style.css` with `font-display: swap`. Aldrich only ships in 400; the SVG `font-weight="700"` on the wordmark synthesizes bold (matches the design intent of #1137). ## TDD - Red commit: E2E test asserting `document.fonts.check('1em Aldrich')` is true and the navbar SVG `<text>` `font-family` contains "Aldrich". Without the font face declaration, both assertions fail on an assertion (not a build error). - Green commit: adds the woff2 + `@font-face` rule, both assertions pass. ## Files - `public/fonts/aldrich-regular.woff2` (new, 16460 bytes) - `public/style.css` — `@font-face` rule - `test-e2e-playwright.js` — new test --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
494d3022f9 |
Partial fix for #1128: close audit gaps (z-scale, css-var lint, multi-viewport E2E, Bug 1+5 polish) (#1133)
## Partial fix for #1128 — closes the gaps PR #1131 left behind PR #1131 was a partial fix for the packets-page layout chaos (merged 2026-05-06 ~01:55 UTC, then the issue was reopened by the maintainer). #1131 shipped Bug 4 (`--surface` definition), the `.path-popover` flip + lower z-index, the debounced re-measure for Bug 1, the `.filter-bar` row-gap + `.multi-select-trigger` truncation for Bug 3, the new z-index TOKENS, and a single-viewport E2E with five individual-component assertions. This PR closes everything else the issue body and the `specs/packets-layout-audit.md` audit asked for. ### What changed (per gap) **Gap A — apply the z-index scale (audit Section 2)** #1131 added `--z-dropdown` / `--z-popover` / `--z-modal` / `--z-tooltip` but explicitly left existing literal values in place. This PR renumbers the 7 dropdowns/popovers the audit named: | Selector | Before | After | |---|---:|---:| | `.col-toggle-menu` | 50 | `var(--z-dropdown)` (100) | | `.multi-select-menu` | 90 | `var(--z-dropdown)` | | `.region-dropdown-menu` | 90 | `var(--z-dropdown)` | | `.node-filter-dropdown` | 100 | `var(--z-dropdown)` | | `.fux-saved-menu` | `var(--z-tooltip)` (9200) | `var(--z-dropdown)` | | `.fux-ac-dropdown` | `var(--z-tooltip)` | `var(--z-dropdown)` | | `.hop-conflict-popover` | `var(--z-tooltip)` | `var(--z-popover)` (300) | `.fux-ctx-menu` deliberately retains the tooltip band — context menus must float above all toolbar UI. `.region-filter-options-menu` no longer exists in the source (was renamed `.region-dropdown-menu`). The `style.css` doc-block at the top is rewritten to record the applied scale and to point operators at the new lint. **Gap B — CSS-var lint (audit Section 5 #1, "single highest-value addition")** Adds `scripts/check-css-vars.js` (~70 lines). Walks `public/*.css`, extracts every `var(--name)` reference WITHOUT a fallback, asserts the name is defined in some `public/*.css`. References WITH a fallback are tolerated. Wired into CI in the `go-test` job before the JS unit tests. The red commit (`608d81f`) shipped this lint exiting 1 against the master tree — three undefined vars that bypassed earlier review: ``` public/style.css:2628 var(--text-primary) public/style.css:2675 var(--bg-hover) public/style.css:2924 var(--primary) ``` The green commit (`1369d1e`) defines those three as aliases in the :root block (`--text-primary` → `--text`, `--bg-hover` → `--hover-bg`, `--primary` → `--accent`). Light + dark themes inherit through the existing tokens. **Gap C — multi-viewport E2E (issue acceptance criterion)** Adds `test-issue-1128-multi-viewport-e2e.js` — sister of the existing single-viewport test. At each of three viewports (1280×900, 1080×800, 768×1024): - takes a screenshot to `e2e-screenshots/issue-1128-<viewport>.png` - asserts no two `.filter-group` siblings vertically overlap - on desktop+laptop, opens the Saved menu and the Types multi-select and asserts the dropdown does not vertically overlap any `.filter-group` below it Plus three viewport-agnostic assertions: - dropdown selectors compute z-index in `[100,199]` (`.col-toggle-menu`, `.multi-select-menu`, `.region-filter-options-menu`, `.fux-saved-menu`, `.fux-ac-dropdown`) - `.path-hops .hop / .hop-named / .arrow` compute `line-height ≤ 18px` - `.col-path` computes `height ≤ 28px` Wired into the e2e-test job after the existing #1128 test. **Gap D — Bug 5 polish (toolbar reorder)** Audit Section 3 Bug 5: swaps `filter-group-dropdowns` and `filter-group-toggles` in `public/packets.js` so time range + Group by Hash + ★ My Nodes sit next to the search input. Pure markup reorder. No CSS / no JS-handler changes. **Gap E — Bug 1 belt-and-suspenders** Audit Section 3 Bug 1 sub-bullets: - locks `.path-hops .hop / .hop-named / .arrow` to `line-height: 18px` so a chip with mixed font metrics cannot overflow the 22px host vertically and bleed into the row above - converts `.col-path { max-height: 28px }` → `height: 28px` because browsers widely ignore `max-height` on `<td>`s; the earlier rule was a no-op ### TDD discipline (red → green) ``` $ git log --oneline origin/master..HEAD |
||
|
|
364c5766fc |
feat(logo): wire new CoreScope SVG logo into navbar + home hero (#1137)
## Adds new logo and home hero Replaces the navbar mushroom emoji + "CoreScope" text spans with the new CoreScope SVG mark, and adds a hero SVG (with the MESH ANALYZER tagline) above the home page H1. ### What changed - `public/img/corescope-logo.svg` — navbar mark, no tagline (locked "aggressive low-amp chirp" variant: facing-arcs + low-amp chirp connector between the two nodes). - `public/img/corescope-hero.svg` — home hero version, includes the MESH ANALYZER tagline. - `public/index.html` — replaces `<span class="brand-icon">🍄</span><span class="brand-text">CoreScope</span>` with `<img class="brand-logo" src="img/corescope-logo.svg?__BUST__" …>`. `.nav-brand` link still routes to `#/`. `.live-dot` retained. - `public/style.css` — adds `.brand-logo { height: 36px }` (32px on tablet ≤900px). Existing 52px nav height unchanged. - `public/home.js` / `public/home.css` — adds `<img class="home-hero-logo">` above the hero `<h1>`, sized `max-width: min(720px, 90vw)` and centered. ### TDD Red→green is visible in the branch: - `3159b82` — `test(logo): add failing E2E …` (red commit). Adds `test-logo-rebrand-e2e.js` and wires it into the `e2e-test` job in `deploy.yml` with `CHROMIUM_REQUIRE=1`. On this commit `index.html` still has the emoji + text spans, `home.js` has no hero img, and the SVG asset files do not exist — the test asserts on each so CI fails on assertion. - `19434e1` — `feat(logo): wire new CoreScope SVG logo …` (green commit). Implements the fix. ### E2E asserts 1. `.nav-brand img` exists with `src` ending `corescope-logo.svg` 2. legacy `.brand-icon` / `.brand-text` are gone 3. `.live-dot` is present, visible, and to the right of the logo (no overlap) 4. `.home-hero img.home-hero-logo` exists with `src` ending `corescope-hero.svg`, positioned BEFORE the `<h1>` 5. both `/img/corescope-{logo,hero}.svg` return 200 with svg content-type ### Customizer compatibility - `customize.js` still does `querySelector('.brand-text')` / `.brand-icon` for live branding updates. Both now return `null`; existing `if (el)` guards make those branches silent no-ops. **No JS errors, but the customizer's `branding.siteName` and `branding.logoUrl` fields no longer rewrite the navbar brand** — the brand is now a fixed SVG asset. - **Theme accent does NOT recolor the SVG.** SVGs loaded via `<img src>` are isolated documents and cannot inherit document CSS variables; the SVG falls back to its embedded brand colors. This is appropriate for a brand mark; if recoloring per theme is desired later, swap to inline SVG (separate PR). ### Browser validation Local Chromium not available in this env; the E2E test soft-skips locally and hard-fails in CI (`CHROMIUM_REQUIRE=1`). Server-side checks done locally: - `curl http://localhost:13581/` → confirmed `<img class="brand-logo" src="img/corescope-logo.svg?<bust>" …>` rendered, no `.brand-icon`/`.brand-text` spans. - `curl -I /img/corescope-logo.svg` and `/img/corescope-hero.svg` → both 200. ### Performance No hot-path changes. Two new static SVG assets (~7.6KB each), served directly by the Go static handler. Cache-busted via `?__BUST__` (auto-replaced server-side). --------- Co-authored-by: OpenClaw Bot <bot@openclaw.local> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local> |
||
|
|
ac0cf5ac7d |
fix(channels): #1087 QR library + share modal + PSK persistence (#1090)
Red commit: 5def4d073c61058fc9f327a3c60ece27e21cbc69 (CI run pending — see Checks tab) Fixes #1087 ## What's broken (4 bugs) 1. **"QR library not loaded"** — `channel-qr.js` checked `root.QRCode` (capital), but the vendored library exports lowercase `qrcode` (Kazuhiko Arase API). Generate & Show QR always fell into the "library not loaded" branch. 2. **QR encodes `name=psk:hex`** — the Share button (and parts of the Generate path) passed the internal `psk:<hex8>` lookup key to `ChannelQR.generate`, ignoring the user's display label stored in `LABELS_KEY`. 3. **PSK channel doesn't persist on refresh** — the persistence path was scattered, and the read-back wasn't verified. Added channels disappeared on refresh and "reappeared" only when a later add ran the persist hook. 4. **Share button reuses the Add Channel modal** — wrong intent reuse (Add = INPUT, Share = OUTPUT). Replaced with a dedicated `#chShareModal` (separate DOM id, separate title, share-only affordances, privacy warning). ## TDD Red commit (this) lands ONLY the failing tests: - `test-channel-issue-1087.js` — source-string contract assertions for all 4 bugs - `test-channel-issue-1087-e2e.js` — Playwright E2E covering generate → QR render, QR display name, persistence across refresh, Share opens dedicated modal Green commit (follow-up) lands the production fixes. ## E2E assertion added E2E assertion added: test-channel-issue-1087-e2e.js:55 ## CI wiring - `test-channel-issue-1087.js` added to `.github/workflows/deploy.yml` (go-test JS unit step) + `test-all.sh` - `test-channel-issue-1087-e2e.js` added to `.github/workflows/deploy.yml` (e2e-test step) --------- Co-authored-by: bot <bot@corescope> Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: clawbot <clawbot@users.noreply.github.com> |
||
|
Kpa-clawbotandyou
|
a8e1cea683 |
fix: use payload type bits only in content hash (not full header byte) (#787)
## Problem The firmware computes packet content hash as: ``` SHA256(payload_type_byte + [path_len for TRACE] + payload) ``` Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type bits (2-5). CoreScope was using the **full header byte** in its hash computation, which includes route type bits (0-1) and version bits (6-7). This meant the same logical packet produced different content hashes depending on route type — breaking dedup and packet lookup. **Firmware reference:** `Packet.cpp::calculatePacketHash()` uses `getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) & PH_TYPE_MASK`. ## Fix - Extract only payload type bits: `payloadType := (headerByte >> 2) & 0x0F` - Include `path_len` byte in hash for TRACE packets (matching firmware behavior) - Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` ## Tests Added - **Route type independence:** Same payload with FLOOD vs DIRECT route types produces identical hash - **TRACE path_len inclusion:** TRACE packets with different `path_len` produce different hashes - **Firmware compatibility:** Hash output matches manual computation of firmware algorithm ## Migration Impact Existing packets in the DB have content hashes computed with the old (incorrect) formula. Options: 1. **Recompute hashes** via migration (recommended for clean state) 2. **Dual lookup** — check both old and new hash on queries (backward compat) 3. **Accept the break** — old hashes become stale, new packets get correct hashes Recommend option 1 (migration) as a follow-up. The volume of affected packets depends on how many distinct route types were seen for the same logical packet. Fixes #786 --------- Co-authored-by: you <you@example.com> |
||
|
efitenandClaude Sonnet 4.6
|
fe314be3a8 |
feat: geo_filter enforcement, DB pruning, geofilter-builder tool, HB column (#215)
## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Kpa-clawbotandyou
|
a555b68915 |
feat: expand frontend coverage collector for 60%+ target (#275)
## Summary
Expands the parallel frontend coverage collector to boost coverage from
~43% toward 60%+. This covers Phases 1 and 2 of the coverage improvement
plan.
### Phase 1 — Visit unvisited pages
- **Compare page** (`#/compare`): Navigates with query params selecting
two real observers from fixture DB, also exercises UI controls
- **Node analytics** (`#/nodes/{pubkey}/analytics`): Visits analytics
for two real nodes from fixture DB, clicks day buttons
- **Traces search** (`#/traces`): Searches for two valid packet hashes
from fixture DB
- **Personalized home**: Sets `localStorage.myNodes` with real pubkeys
before visiting `#/home`
- **Observer detail pages**: Direct navigation to
`#/observers/test-obs-1` and `#/observers/test-status-obs`
- **Real packet detail**: Navigates to `#/packets/b6b839cb61eead4a`
(real hash)
- **Rapid route transitions**: Exercises destroy/init cycles across all
pages
- **Compare in route list**: Added to the full route transition exercise
### Phase 2 — page.evaluate() for interactive code paths
| File | Functions exercised |
|------|-------------------|
| **live.js** | `vcrPause`, `vcrSpeedCycle`, `vcrReplayFromTs`,
`drawLcdText`, `vcrResumeLive`, `vcrUnpause`, `vcrRewind`,
`updateVCRClock`, `updateVCRLcd`, `updateVCRUI`, `bufferPacket`
(synthetic WS packets), `dbPacketToLive`, `renderPacketTree` |
| **packets.js** | `renderDecodedPacket` (ADVERT + GRP_TXT), `obsName`,
`renderPath`, `renderHop` |
| **packet-filter.js** | 30+ filter expressions now **evaluated against
4 synthetic packets** (previously only compiled, not run). Covers
`resolveField` for all field types including `payload.*` dot notation |
| **nodes.js** | `getStatusInfo`, `renderNodeBadges`,
`renderStatusExplanation`, `renderHashInconsistencyWarning` with varied
node types/roles |
| **roles.js** | `getHealthThresholds` (all roles), `getNodeStatus` (all
roles × active/stale), `getTileUrl`, `syncBadgeColors`, `miniMarkdown`
(bold, italic, code, links, lists), `copyToClipboard` |
| **channels.js** | `hashCode`, `getChannelColor`, `getSenderColor`,
`highlightMentions`, `formatSecondsAgo` |
| **app.js** | `escapeHtml`, `debouncedOnWS`, extended
`timeAgo`/`truncate` edge cases, extended
`routeTypeName`/`payloadTypeName`/`payloadTypeColor` ranges |
### What changed
- `scripts/collect-frontend-coverage.js` — +336 lines across existing
groups (no new groups added)
### Testing
- `npm test` passes (all 13 tests)
- No other files modified
Co-authored-by: you <you@example.com>
|
||
|
Kpa-clawbotandyou
|
5777780fc8 |
refactor: parallel coverage collector (~30-60s vs 8min) (#272)
## Summary Redesigned frontend coverage collector with 7 parallel browser contexts. Coverage collector runs on master pushes only (skipped on PRs). ### Architecture 7 groups run simultaneously via `Promise.allSettled()`: - G1: Home + Customizer - G2: Nodes + Node Detail - G3: Packets + Packet Detail - G4: Map - G5: Analytics + Channels + Observers - G6: Live + Perf + Traces + Globals - G7: Utility functions (page.evaluate) ### Speed gains - `safeClick` 500ms → 100ms - `navHash` 150ms → 50ms - Removed redundant page visits and E2E-duplicate interactions - Wall time = slowest group (~30-60s estimated) ### 821 lines → ~450 lines Each group writes its own coverage JSON, nyc merges automatically. ### CI behavior - **PRs:** Coverage collector skipped (fast CI) - **Master:** Coverage collector runs (full synthetic user validation) Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
f438411a27 |
chore: remove deprecated Node.js backend (-11,291 lines) (#265)
## Summary Removes all deprecated Node.js backend server code. The Go server (`cmd/server/`) has been the production backend — the Node.js server was kept "just in case" but is no longer needed. ### Removed (19 files, -11,291 lines) **Backend server (6 files):** `server.js`, `db.js`, `decoder.js`, `server-helpers.js`, `packet-store.js`, `iata-coords.js` **Backend tests (9 files):** `test-decoder.js`, `test-decoder-spec.js`, `test-server-helpers.js`, `test-server-routes.js`, `test-packet-store.js`, `test-db.js`, `test-db-migration.js`, `test-regional-filter.js`, `test-regional-integration.js` **Backend tooling (4 files):** `tools/e2e-test.js`, `tools/frontend-test.js`, `benchmark.js`, `benchmark-ab.sh` ### Updated - `AGENTS.md` — Rewritten architecture section for Go, explicit deprecation warnings - `test-all.sh` — Only runs frontend tests - `package.json` — Updated test:unit - `scripts/validate.sh` — Removed Node.js server syntax check - `docker/supervisord.conf` — Points to Go binary ### NOT touched - `public/` (active frontend) ✅ - `test-e2e-playwright.js` (frontend E2E tests) ✅ - Frontend test files (`test-packet-filter.js`, `test-aging.js`, `test-frontend-helpers.js`) ✅ - `package.json` / Playwright deps ✅ ### Follow-up - Server-only npm deps (express, better-sqlite3, mqtt, ws, supertest) can be cleaned from package.json separately - `Dockerfile.node` can be removed separately --------- Co-authored-by: you <you@example.com> |
||
|
|
ec7ae19bb5 |
ci: restructure pipeline — sequential fail-fast, Go server E2E, remove deprecated JS tests (#256)
## Summary Complete CI pipeline restructure. Sequential fail-fast chain, E2E tests against Go server with real staging data, all deprecated Node.js server tests removed. ### Pipeline (PR): 1. **Go unit tests** — fail-fast, coverage + badges 2. **Playwright E2E** — against Go server with fixture DB, frontend coverage, fail-fast on first failure 3. **Docker build** — verify containers build ### Pipeline (master merge): Same chain + deploy to staging + badge publishing ### Removed: - All Node.js server-side unit tests (deprecated JS server) - `npm ci` / `npm run test` steps - JS server coverage collection (`COVERAGE=1 node server.js`) - Changed-files detection logic - Docs-only CI skip logic - Cancel-workflow API hacks ### Added: - `test-fixtures/e2e-fixture.db` — real data from staging (200 nodes, 31 observers, 500 packets) - `scripts/capture-fixture.sh` — refresh fixture from staging API - Go server launches with `-port 13581 -db test-fixtures/e2e-fixture.db -public public-instrumented` --------- Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> Co-authored-by: you <you@example.com> |
||
|
you
|
13cab9bede |
perf: optimize frontend coverage collector (~2x faster)
Three optimizations to reduce wall-clock time: 1. Reduce safeClick timeout from 3000ms to 500ms - Elements either exist immediately after navigation or don't exist at all - ~75 safeClick calls; if ~30 miss, saves ~75s of dead wait time 2. Replace 18 page.goto() calls with SPA hash navigation - After initial page load, the SPA shell is already in the DOM - page.goto() reloads the entire page (network round-trip + parse) - Hash navigation via location.hash triggers the SPA router instantly - Only 3 page.goto() remain: initial load + 2 home page loads after localStorage.clear() 3. Remove redundant final route sweep - All 10 routes were already visited during the page-specific sections - The sweep just re-navigated to pages that had already been exercised - Saves ~2s of redundant navigation Also: - Reduce inter-route wait from 200ms to 50ms (SPA router is synchronous) - Merge utility function + packet filter exercises into single evaluate() call - Use navHash() helper for consistent hash navigation with 150ms settle time |
||
|
you
|
12d1174e39 |
perf: speed up frontend coverage tests (~3x faster)
Three optimizations to the CI frontend test pipeline:
1. Run E2E tests and coverage collection concurrently
- Previously sequential (E2E ~1.5min, then coverage ~5.75min)
- Now both run in parallel against the same instrumented server
- Expected savings: ~5 min (coverage runs alongside E2E instead of after)
2. Replace networkidle with domcontentloaded in coverage collector
- SPA uses hash routing — networkidle waits 500ms for network silence
on every navigation, adding ~10-15s of dead time across 23 navigations
- domcontentloaded fires immediately once HTML is parsed; JS initializes
the route handler synchronously
- For in-page hash changes, use 200ms setTimeout instead of
waitForLoadState (which would never re-fire for same-document nav)
3. Extract coverage from E2E tests too
- E2E tests already exercise the app against the instrumented server
- Now writes window.__coverage__ to .nyc_output/e2e-coverage.json
- nyc merges both coverage files for higher total coverage
Also:
- Split Playwright install into browser + deps steps (deps skip if present)
- Replace sleep 5 with health-check poll in quick E2E path
|
||
|
Kpa-clawbotandCopilot
|
72161ba8fe |
perf: replace 169 blind sleeps with Playwright waits in coverage script
Remove all 169 waitForTimeout() calls (totaling 104.1s of blind sleeping)
from scripts/collect-frontend-coverage.js:
- Helper functions (safeClick, safeFill, safeSelect, clickAll, cycleSelect):
removed 300-400ms waits after every interaction — Playwright's built-in
actionability checks handle waiting for elements automatically
- Post-navigation waits: removed redundant sleeps after page.goto() calls
that already use waitUntil: 'networkidle'
- Hash-change navigations: replaced waitForTimeout with
waitForLoadState('networkidle') for proper SPA route settling
- Toggle/button waits: removed — event handlers execute synchronously
before click() resolves
- Post-evaluate waits: removed — evaluate() is synchronous
Local benchmark (Windows, sparse test data):
Before: 744.8s
After: 484.8s (35% faster, 260s saved)
On CI runner (ARM Linux with real mesh data), savings will be
proportionally better since most elements exist and the 104s
of blind sleeping was the dominant bottleneck.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
you
|
9b0c740537 |
revert: aggressive coverage interactions dropped score from 42% to 39%
The page.evaluate() calls corrupting localStorage and firing fake events caused page error-reloads, losing accumulated coverage. Reverting to the 42% version which was the actual high water mark. |
||
|
you
|
556e3b19db |
coverage: aggressive branch coverage push — target 80%+
Add ~900 lines of deep branch-coverage interactions: - Utility functions with all edge cases (timeAgo, truncate, escapeHtml, formatHex, etc.) - roles.js: getHealthThresholds/getNodeStatus for all roles + edge inputs - PacketFilter: compile+match with mock packets, all operators, bad expressions - HopResolver/HopDisplay: init, resolve, renderPath with various inputs - RegionFilter: onChange, getSelected, isEnabled, setRegions, render - Customize: deep tab cycling, import/export, bad JSON, theme preview - WebSocket reconnection trigger - Keyboard shortcuts (Ctrl+K, Meta+K, Escape) - Window resize (mobile/tablet/desktop) for responsive branches - Error routes: nonexistent nodes/packets/observers/channels - localStorage corruption to trigger catch branches - Theme toggling (dark/light rapid switching, custom vars) - Live page: VCR modes, timeline clicks, speed cycling, all toggles - Audio Lab: play/stop/loop, BPM/volume sliders, voice selection - All analytics tabs via deep-link + sort headers - Packets: complex filter expressions, scroll-to-load, double-click - Nodes: special char search, all sort columns, fav stars - Channels: resize handle drag, theme observer, node tooltips - Observers/Observer Detail: sort, tabs, day cycling - Node Analytics: day buttons, tabs - Home: both new/experienced flows, empty search results - debouncedOnWS/onWS/offWS exercise |
||
|
you
|
86dce6f350 |
coverage: massively expand frontend interaction coverage
Exercise every major code path across all frontend files: app.js: all routes, bad routes, hashchange, theme toggle x4, hamburger menu, favorites dropdown, global search, Ctrl+K, apiPerf(), timeAgo/truncate/routeTypeName utils nodes.js: sort every column (both directions), every role tab, every status filter, cycle all Last Heard options, click rows for side pane, navigate to detail page, copy URL, show all paths, node analytics day buttons (1/7/30/365), scroll target packets.js: 12 filter expressions including bad ones, cycle all time windows, group by hash toggle, My Nodes toggle, observer menu, type filter menu, hash input, node filter, observer sort, column toggle menu, hex hash toggle, pause button, resize handle, deep-link to packet hash map.js: all role checkboxes toggle, clusters/heatmap/neighbors/ hash labels toggles, cycle Last Heard, status filter buttons, jump buttons, markers, zoom controls, dark mode tile swap analytics.js: all 9 tabs clicked, deep-link to each tab via URL, observer selector on topology, navigate rows on collisions/ subpaths, sortable headers on nodes tab, region filter customize.js: all 5 tabs, all preset themes, branding text inputs, theme color inputs, node color inputs, type color inputs, reset buttons, home tab fields (hero, journey steps, checklist, links), export tab, reset preview/user theme live.js: VCR pause/speed/missed/prompt buttons, all visualization toggles (heat/ghost/realistic/favorites/matrix/rain), audio toggle + BPM slider, timeline click, resize event channels.js: click rows, navigate to specific channel observers.js: click rows, navigate to detail, cycle days select traces.js: click rows perf.js: refresh + reset buttons home.js: both chooser paths, search + suggest, my-node cards, health/packets buttons, remove buttons, toggle level, timeline Also exercises packet-filter parser and region-filter directly. |
||
|
you
|
26d4bbc39d | ci: fix coverage collector — use Playwright bundled chromium on CI | ||
|
you
|
5ee976055b |
feat(coverage): add targeted Playwright interactions for higher frontend coverage
Add redundant selectors (data-sort, data-role, data-status, data-tab, placeholder-based search, emoji theme toggle, .cust-close, #fTimeWindow, broader preset selectors) to exercise more frontend code paths. All interactions wrapped in try/catch for resilience. |
||
|
you
|
860d5c574e |
test: expanded frontend coverage collection with page interactions
367 lines of Playwright interactions covering nodes, packets, map, analytics, customizer, channels, live, home pages. Fixed e2e channels assertion (chList vs chResp.channels). |
||
|
you
|
d7faa4d978 |
Add frontend code coverage via Istanbul instrumentation + Playwright
- Install nyc for Istanbul instrumentation - Add scripts/instrument-frontend.sh to instrument public/*.js - Add scripts/collect-frontend-coverage.js to extract window.__coverage__ - Add scripts/combined-coverage.sh for combined server+frontend coverage - Make server.js serve public-instrumented/ when COVERAGE=1 is set - Add test:full-coverage npm script - Add public-instrumented/ and .nyc_output/ to .gitignore |
||
|
you
|
91a6a2c525 | fix: migration handles concurrent dual-write with INSERT OR IGNORE | ||
|
you
|
d7e415daa7 | fix: raw_hex NOT NULL in transmissions schema — deleted 4 junk test rows | ||
|
you
|
2c6148fd2d |
Add dedup migration script (Milestone 1)
Creates transmissions and observations tables from existing packets table. - Groups packets by hash → 1 transmission per unique hash - Creates 1 observation per original packet row with FK to transmission - Idempotent: drops and recreates new tables on each run - Does NOT modify the original packets table - Prints stats and verifies counts match Tested on test DB: 33813 packets → 11530 transmissions (2.93x dedup ratio) |
||
|
you
|
10b11106f6 |
ci: add pre-deploy JS validation — syntax check + undefined reference detection
Validation runs BEFORE docker build. If it fails, deployment is blocked. No more broken code reaching production. |