mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-25 23:13:49 +00:00
aabeda0f2ce173f51b3b42ea5e2bfa55817e18ea
34
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex B
|
893773338e |
chore(tests): move root test-*.js into tests/unit and tests/e2e (#2036)
Moves 290 root test-*.js into tests/unit (177, listed in test-all.sh) and tests/e2e (113, classified in scripts/non-unit-tests.json), per #1981 and PR-D of #1385. Root goes from 348 entries to 48. test-all.sh and test-fixtures/ stay put. The inventory guard now fails if a test reappears in the root or sits in the wrong folder.
Verified independently of the diff: the invoked sets are unchanged (test-all.sh 177 before and after, deploy.yml 96 before and after, both identical as sets), and a full local run of test-all.sh on master and on the branch produced 4702 output lines each whose only differences are absolute paths, stack-trace line numbers shifted by the REPO_ROOT line, the inventory wording and two perf ratios. The guard was mutation-checked: a test back in the root, a unit suite in tests/e2e, and a suite dropped from test-all.sh each make it exit 1. CI run 35246304316 ran 97 suites from tests/e2e and is green.
Follow-up
|
||
|
Sylvain Rabot
|
a2ea18f778 |
perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992)
Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite
3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4),
and pays the resulting cross-compilation cost with `zig cc`.
Draft because the riskiest part of this deletes rows — see [Please
review this part first](#please-review-this-part-first) — and because
three things remain unverified at the bottom.
`modernc.org/sqlite` is a transpilation of the C amalgamation. This repo
is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out
neighbour/topology/analytics queries per request, and it pays for that
transpilation on exactly those paths. Head-to-head on the same
120k-transmission / 240k-observation database, running our own hot-path
SQL under both drivers (Apple M4, `-count=5`, medians):
| workload | modernc | mattn | |
|---|---:|---:|---|
| chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms |
**2.3×** |
| aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×**
|
| 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** |
Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the
chunk load.
**Superseded by a production run.** @efiten measured both drivers on a
real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 —
as server-only containers against the same live volume, one at a time,
with round 2 reversing the order so the page cache favours the old
driver:
| | audit 7d | audit 24h | background fill (13 chunks) | start →
/api/health |
|---|---:|---:|---:|---:|
| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |
Warm, the old driver improves to 13.46s on the 7d audit and still loses
by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against
0.037s — nothing.
**So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.**
The shape the harness predicted holds — scans and joins gain, small
lookups do not — which is more reassuring than the magnitude would have
been. Quote these numbers.
**The counterweight**, cold and native on that machine: a build goes
from **52s to 163s**. An instance that builds its own image pays that
per deploy.
## The build is cgo now, and one thing about that is a trap
**`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary
dies on its first query with `go-sqlite3 requires cgo to work. This is a
stub`. A green build is not evidence of anything here, which is why
`AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely
cannot cross-compile any more.
A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc
-target {x86_64,aarch64}-linux-musl` and links static, so each artifact
stays a single self-contained file and the `alpine:3.20` runtime no
longer depends on the base image's libc at all.
`-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl
objects zig links in, and without it the server binary is 19.8MB instead
of 12.1MB.
The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU
for compilation — and gains a checksum-pinned zig plus BuildKit cache
mounts. The mounts are not a nicety: without them an image build
recompiles the amalgamation from cold and takes over half an hour.
## Please review this part first
`internal/dbschema/dedup_index.go` **deletes observation rows**. It is
the one part of this change that can lose data, and it exists because
the migration exposed a real bug rather than causing one.
`stmtInsertObservation` resolves its `ON CONFLICT` against
`idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created
inside the branch that creates the `observations` table for the first
time. Any database whose table predates that branch never got one, so
the UPSERT had no conflict target. modernc failed on the first insert;
mattn fails at `OpenStore`. Same bug, found earlier.
Creating the index unconditionally repairs it — but the index is what
was supposed to prevent duplicates, so a database that never had it can
already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this
repo holds one.** So duplicates are collapsed first. Refusing is not the
safer option: without the index the ingestor cannot prepare its UPSERT,
so it cannot start at all.
Replaying that UPSERT faithfully is subtler than it looks, and a first
cut of this got it wrong twice:
- `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a
group in id order the survivor keeps the **last** non-NULL value. Taking
the first silently discarded newer readings.
- The UPSERT names exactly five columns (`snr`, `rssi`, `score`,
`raw_hex`, `resolved_path`). Every other column must keep the surviving
row's own value; merging those too invents history the ingestor would
never have written.
Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split
apart, a writer inserting a duplicate in the gap fails the index
creation while leaving the deletions committed — rows destroyed and no
index to show for it.
Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**,
holding the write lock throughout, once, at ingestor startup before MQTT
subscribe. Materialising the duplicate-group scan once rather than per
column took that from 9.7s; the pathological case (400k of 600k rows
duplicated) is 5.7s, slightly worse than the 4.2s it was before that
change.
## Four more behavioural differences
Full detail in `docs/sqlite-driver-migration.md`. Briefly:
**Statement preparation is eager.** modernc's `newStmt` stored the SQL
and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`,
so SQL naming a missing table fails at *open*. 59 server tests failed on
this alone, all fixtures with partial schemas. `OpenDB` keeps failing
loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the
fixtures now declare what they are prepared against via
`ensurePreparable`. This also exposed nine `nodes(pubkey …)`
declarations across seven files, where production has only ever had
`public_key` — lazy compilation had hidden the mismatch for as long as
it existed.
**`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to
NORMAL and executes the pragma unconditionally, where SQLite's own
default (what modernc left alone) is FULL. In WAL mode that weakens
durability under power loss. Pinned in `dbschema.WriterDSN`, which both
writers now share — `cmd/migrate` kept a bare path at first and so
quietly wrote at NORMAL, which is what a second copy of a DSN buys you.
**The DSN dialects are mutually invisible.** modernc understood only
`_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither
errors on the other's form — a driver-only rename would have dropped
every pragma in silence. `_journal_mode=WAL` is also gone from the
server's read handle: modernc ignored it, mattn honours it, and setting
`journal_mode` on a read-only connection is a write. Dropping
`_busy_timeout` with it costs nothing, since mattn already defaults to
5000ms — which means the read handle finally *gets* the busy timeout it
had silently lacked.
**`mode=ro` survives for a non-obvious reason.** mattn always passes
`READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what
makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the
#1283/#1289 invariant holds with no build flags — but it depends on the
`file:` prefix. `cmd/decrypt` had been building its DSN without one, so
its `mode=ro` had never applied and a missing path was created
read-write. Fixed in passing; never a migration regression.
## What did not change
No modernc-specific API was in use: no `RegisterFunction`, no
`*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No
`time.Time` is ever bound as a query argument, so driver time handling
is not in play. Both drivers convert declared
`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so
`/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an
earlier draft "fixed" that with a `CAST` and would have been the
regression.
## Tests and CI
New regression tests, each written because something got through without
it:
- `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge
ordering. The original test used complementary NULLs, which passes
whichever direction you pick, which is why the bug survived it.
- `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation
must roll the deletions back.
- `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma,
read back through the store's own connection. A separate `sqlite3`
session or the startup log line would prove nothing.
- `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which
now rests on a detail of the driver's C wrapper.
- `TestEnsurePreparableMatchesPrepareStatements` — fails when a new
prepared statement outgrows the fixture helper.
CI gains test execution for `cmd/migrate` and `internal/dbschema`, which
had none and both open the database. A PR-time two-arch build plus an
arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without
it nothing on a PR would exercise zig, static musl linking or arm64, and
the first signal would arrive on master. `cache-dependency-path` widens
from 2 of the 5 tracked `go.sum` files to all of them.
`make test` passes across all 14 modules, `cmd/server` also under `-race
-count=2` with no failures and no races. `gofmt` and `go vet` clean.
Release-routing and Dockerfile COPY-invariant gates pass.
## Verified by running
- All 8 cross-builds static and correct-architecture; both arches of the
container image built, exported and run under QEMU, serving
`/api/health` and `/api/nodes` against a 2.9M-observation production
snapshot.
- The `migrate` binary repairing that snapshot's duplicate on bare
Alpine.
- `CGO_ENABLED=0` producing a binary that builds and then fails on first
query.
## Not verified
- ~~The 2–2.3× figures come from a standalone harness, not this load
under the old driver.~~ **Closed** by @efiten's production run above,
which also corrected the multiplier.
- SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no
total `ORDER BY`.
- Sustained live ingest through the new writer DSN, and the duplicate
collapse against a database an ingestor is actively writing to. Verified
against a static snapshot only, and the collapse is measured at 4.1s on
2.4M synthetic rows with 5 duplicates — well short of an 11M-row
instance. @efiten has offered a staging instance taking real MQTT
traffic; **this is the item to close before the PR leaves draft.**
An earlier revision of this branch shipped the dedup merge in the wrong
direction with a green test suite, and review then found three more
things in the same file: the repair gated on an error string, a
non-atomic TEMP table drop aimed at the wrong connection, and a deletion
whose only record was a row count. All fixed in
|
||
|
n30nex
|
cd9b4c04d0 |
test: unify frontend test runs and prevent inventory drift (#1965)
Make `test-all.sh` the authoritative standalone frontend runner for npm and CI. Restore stale assertions and reject missing, duplicate, removed or undocumented inventory entries. Fixes #1858. Rebased onto `a2f039d4`. Retains release-routing, map scope-state and Scope Audit stylesheet tests, adds `test-packets-local-channels.js` to the sorted runner, and classifies `test-neighbor-map-btn-clip-e2e.js` under browser. Its separate CI browser step is preserved. Inventory: 280 root suites, 167 standalone, 113 requiring separate setup. The icon repair fixes two suites red on master: `test-issue-1648-m2-emoji-scan.js` and `test-issue-1648-m6-final-sweep.js`. Node/live configured-scope confirmations now use the existing accessible Phosphor check sprite. Values, visibility conditions and scanner assertions are preserved. - Red evidence: `89e45a9` inventory assertions; `4e255df` accessible confirmation assertion. The latest rebase also reproduced both unclassified-file failures before adding their entries. This follow-up only changes runner/classification configuration and counts; no test files modified. - Local validation: all 167 standalone suites; 27 M2 browser checks and 16 neighbor geometry checks in Chromium. Syntax, whitespace, PII, CSS-variable and XSS checks passed. - Browser coverage includes populated/empty/null configured scopes in node and live views. - No new dependencies, requests, application settings or Go changes. Workflow outside the unit step matches master. - Windows validation uses process-local UTF-8 settings. Encoding and node-reach confirmation follow-ups remain separate, as requested. ## Preflight override External `run-all.sh` is unavailable; applicable repository checks were run directly. |
||
|
|
a2004351d3 |
fix(#1684): staging disk monitor + cleanup cron (#1686)
## Summary Adds a staging VM disk-usage monitor + daily cleanup cron, fixing the gap surfaced by #1684 (staging hit 100% disk during a hot-patch, no alert, no cleanup). ## What landed - **`scripts/staging/disk-monitor.sh`** — parses `df -P <mount>`, classifies usage `<80 ok / >=80 warn / >=90 error / >=95 alert`, emits to stderr + journald via `logger -p`, exits non-zero on `error|alert` so the systemd unit surfaces as failed. - **`scripts/staging/disk-cleanup.sh`** — daily prune of `/tmp` snapshot patterns (`*.db`, `staging-snap.*`, `cs-*`, `node-compile-cache`) older than 7d + `docker builder/image prune --filter until=72h --filter label!=keep`. Honors `CORESCOPE_CLEANUP_DRY_RUN=1`. - **`scripts/staging/test-disk-monitor.sh`** — pure-bash unit tests for the testable helpers (22 cases covering threshold boundaries, df parsing, invalid input, severity→priority mapping). - **`DEPLOY.md`** — install one-liner with full inline systemd unit + timer content (15-min monitor, daily 03:30 cleanup). Uses `<STAGING_HOST>` placeholder. - **`.github/workflows/deploy.yml`** — wires `test-disk-monitor.sh` into the Go build & test job. ## TDD - Commit `26185967` (RED): tests against stub helpers — `PASS=5 FAIL=17` on assertions. - Commit `d31a1082` (GREEN): real helpers — `PASS=22 FAIL=0`. ## Phase 3 — `staging-snap.db` root cause `grep -rn staging-snap.db cmd/ public/ scripts/` → **zero hits**. The 4.4 GB orphan was a manual debug artifact, not committed code. The cleanup retention rule prevents recurrence. Partial fix for #1684 — leaves issue open for operator to verify install on staging and confirm alert fires at 85%. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: clawbot <bot@openclaw.dev> |
||
|
Kpa-clawbotandopenclaw-bot
|
55e4d957b1 |
M1: emoji → Phosphor Icons — top-nav, mobile nav, Compare (#1648) (#1649)
Red commit:
|
||
|
|
63bfa3d910 |
feat(security): detect CDN-fronted deployment + document bypass requirement (closes #1561) (#1564)
Closes #1561. Follow-up to #1551. ## Why #1551 added `Cache-Control: no-store` to all `/api/*` responses. That's sufficient for CDNs that honour origin headers (Varnish, nginx). It is **not** sufficient for Cloudflare zones where Cache Rules / Page Rules override origin Cache-Control. Field evidence from the meshat.se diagnosis (2026-06-04): observers behind Cloudflare were returning `cf-cache-status: HIT` with `age` up to ~6 hours despite the origin emitting `no-store`. The CDN was caching per zone policy and ignoring the upstream directive — exactly the failure mode #1551 cannot reach. The application has no way to inject CDN rules; the only durable fix is operator-side. This PR makes that operator step discoverable and verifiable. ## What ### Server-side detection (log-only) `cmd/server/cdn_detection.go` adds a middleware wired into the `/api/*` chain after `noStoreAPIMiddleware`. On the **first** request bearing any CDN-typical header (`CF-Connecting-IP`, `CF-Ray`, `X-Forwarded-For`, `X-Real-IP`, `Fastly-Client-IP`, `True-Client-IP`) it logs: ``` [security] WARNING: detected request via CDN (CF-Ray header present). Ensure /api/* is bypassed in your CDN config — see docs/deployment-behind-cdn.md. Cached API responses cause observer-flap and incorrect dashboards. ``` `sync.Once` guarantees the warning fires at most once per process boot. The middleware never blocks, never modifies the response, never adds headers. Detection is observational only — operators who run behind a CDN without bypass have a real bug; the warning is appropriate. ### Operator documentation `docs/deployment.md` gains a new **"Behind a CDN (Cloudflare, Fastly)"** section covering: 1. Curl verification command + healthy vs unhealthy output examples 2. Cloudflare Cache Rule creation (URI Path starts-with `/api/` → Bypass cache) 3. Legacy Page Rules equivalent 4. Fastly note 5. Re-verification 6. Meaning of the startup log warning 7. Why we can't fix this server-side `docs/deployment-behind-cdn.md` is the canonical path the log message references — it's a short TL;DR that links back to the full section. ### Healthcheck script `scripts/check-cdn-bypass.sh` — POSIX sh, no dependencies beyond curl + grep + awk. Operators run: ```sh scripts/check-cdn-bypass.sh https://your-domain.example.com ``` Exits `0` with `OK: no CDN caching detected ...` or `1` with a precise diagnostic naming the offending header (`cf-cache-status: HIT` or stale `age`). ## TDD - **Red commit `e90ccaba`** (`test(security): RED ...`) — `cmd/server/cdn_detection_test.go` (4 Go tests + 6 subtests for each header) and `scripts/test-check-cdn-bypass.sh` (3 shell harness cases). Middleware stub returns `next` unchanged so tests compile and fail on assertions, not build errors. - **Green commit `5e6a60b5`** (`feat(security): GREEN ...`) — real middleware, wiring in `routes.go`, healthcheck script, doc. ## Deliverables | File | Status | Purpose | |------|--------|---------| | `cmd/server/cdn_detection.go` | new | middleware + sync.Once warning | | `cmd/server/cdn_detection_test.go` | new | 4 Go tests (1 stand-alone + 1 silence + 1 once + 1 table-driven over 6 headers) | | `cmd/server/routes.go` | modified | `r.Use(cdnDetectionMiddleware)` after no-store | | `docs/deployment.md` | modified | TOC entry + "Behind a CDN" section | | `docs/deployment-behind-cdn.md` | new | canonical path referenced by log message + script output | | `scripts/check-cdn-bypass.sh` | new | operator-runnable healthcheck | | `scripts/test-check-cdn-bypass.sh` | new | shell harness with fake curl | ## What this PR explicitly does NOT do - Does not block requests based on CDN detection (log-only). - Does not enforce CDN bypass (impossible — operator-controlled). - Does not spoof, strip or modify CDN headers. - Does not add CSP / HSTS / other security headers (out of scope). - Warning is not configurable — operators behind a CDN without bypass have a real bug, surfacing it is correct. ## Verification - `go test ./...` in `cmd/server/` — full suite green. - `sh scripts/test-check-cdn-bypass.sh` — 3/3 pass. - Preflight checklist — all 11 gates clean (PII, branch scope, red commit, CSS vars, CSS self-fallback, LIKE-on-JSON, sync migration, async-migration annotation, XSS sinks, img/SVG ratio, themed-img/SVG, fixture coverage). --------- Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: clawbot <bot@clawbot.invalid> |
||
|
|
e4a21fc9ab |
feat(preflight): hard-fail gate on unescaped node-controlled HTML sinks (#1543)
## Summary Closes the "XSS regression in newly-added sink" class. Follow-up to #1537 (10 stored-XSS sinks in node names) and the post-#1537 audit (TRACE-1, OBS-1, ANL-1 — 3 additional HIGH XSS in files #1537 didn't touch). After those fixes land, the project still has **zero automated catch for the next one**. Every future PR can re-introduce the same class freely. This PR closes that gap with a hard-fail pr-preflight gate that runs at PR-creation time and in CI. ## What the gate does A NEW or MODIFIED line in the PR diff under `public/**/*.{js,html}` is flagged when it matches any of these sink patterns: | Pattern | What it catches | |---|---| | `.innerHTML = \`…\`` / `'…'` | template-literal or string-concat HTML injection | | `insertAdjacentHTML(…, \`…\`)` | DOM-adjacent injection | | `.bindPopup(\`…\`)` / `.bindTooltip(\`…\`)` | Leaflet popup/tooltip injection (the OBS-1 class) | | `.setAttribute('on<event>', …)` | inline event-handler injection | | `.setAttribute('href'\|'src'\|'action'\|'formaction', <interp>)` | `javascript:` URI class | For each flagged line, the gate then walks the dynamic substring (`${…}`, post-`+`, or `setAttribute` value arg) and only fires if it interpolates an identifier from the node-controlled allowlist (`name`, `observer`, `sender`, `pubkey`, `body`, `hash`, …). This keeps the regex off static CSS classes like `text-center`. A flagged line is accepted (no fail) when ANY of: - **(a)** wrapped in `escapeHtml(` / `escapeAttr(` / `safeEsc(` / local `esc(` — the audited helpers - **(b)** a same-PR `test*.js` file DOM-greps the audit payload (`' onfocus=` or `onerror=alert`) AND references the sink file's basename - **(c)** the PR body carries `PREFLIGHT-XSS-OPTOUT: <file>:<line> reason="…"` — explicit author opt-out logged for reviewer attention Otherwise: **HARD FAIL** with `file:line: flagged: <token>` plus a suggested fix. ## Split - **Skill directory** (local, no PR): - `~/.openclaw/skills/pr-preflight/scripts/check-xss-sinks.sh` — canonical gate - `~/.openclaw/skills/pr-preflight/data/xss-node-controlled-fields.txt` — allowlist (27 identifiers, easy to extend without a repo PR) - wired into `~/.openclaw/skills/pr-preflight/scripts/run-all.sh` - **This PR** (in repo): - `testdata/preflight-xss/` — fixtures (`bad-1..bad-3`, `good-1..good-2`, `test-good-2.js`) - `scripts/check-xss-sinks.sh` — local mirror of the canonical gate, so CI can exercise the gate without depending on the skill dir - `test-preflight-xss-gate.js` — Node test wrapper that asserts bad fixtures fail (exit 1) and good fixtures pass (exit 0) - `public/app.js` — `escapeHtml` docstring marked CANONICAL with links to the enforcing gate - `.github/workflows/deploy.yml` — invoke `node test-preflight-xss-gate.js` alongside the existing `test-xss-escape-sinks.js` ## TDD red → green | | Commit | Test result | |---|---|---| | **Red** | `test(preflight-xss): RED — fixtures + assertion wrapper for XSS sink gate` | `test-preflight-xss-gate.js` exits 1 — bad fixtures unexpectedly pass because `scripts/check-xss-sinks.sh` is a no-op stub. Genuine assertion failure (not a build error). | | **Green** | `feat(preflight): GREEN — implement XSS-sink check + escapeHtml docstring` | stub replaced with real check; all 5 fixtures behave as expected. | The red commit ships a working stub script so the test runs to completion and fails on an **assertion**, not on a missing-file error. ## Coverage proof — would the gate have caught the originals? - **PR #1537 (10 sinks):** synthetic file from the deleted lines of #1537 → gate flags `n.name` in `innerHTML \`tpl\`` and two `bindPopup(\`…${n.name}\`)` lines. Yes, the gate would have caught these the moment they hit a PR diff. - **Post-#1537 audit:** - **TRACE-1** (`traces.js` `${e.message}` / `${urlHash}` in innerHTML): yes — the `hash`/`urlHash` tokens are allowlisted and the innerHTML template-literal pattern matches. - **OBS-1** (`observer-detail.js` URL fragment + MQTT fields into innerHTML / bindPopup): yes — the `observer`, `text`, `hash` tokens are allowlisted and both sink patterns match. - **ANL-1** (`analytics.js` attribute-mutation roundtrip): yes for `setAttribute('on*', …)` and `setAttribute('href', \`…${interp}…\`)` patterns. (Note: pure innerHTML lines with only `${e.message}` are not node-controlled and are intentionally not flagged.) ## Allowlist (initial 27 identifiers) ``` adv_name name observer observer_name sender from_node channel channel_name model firmware client_version radio iata hopNames nodeLabel obsName n.name o.name obs.name public_key pubkey area_key region_name text body message preview hash urlHash ``` Extend in `~/.openclaw/skills/pr-preflight/data/xss-node-controlled-fields.txt` whenever a new node-controlled field surfaces in an audit — no repo PR required. ## Hard rules respected - No build step, no ESLint plugin, no AST analysis — grep + heuristics + opt-out escape valves - Hard fail (exit 1), not warning-only (exit 2) - PII preflight grep on every commit + this PR body - Same split as the sibling migration-gate PR ## Three-axis merge-readiness - **Mergeable:** yes — branch is clean off `origin/master`, no conflicts - **CI:** will report on push; red commit expected to fail, green commit expected to pass - **Threads:** none open yet (new PR) --------- Co-authored-by: meshcore-bot <bot@local> Co-authored-by: mc-bot <bot@meshcore.local> Co-authored-by: corescope-bot <bot@corescope> |
||
|
efitenandClaude Sonnet 4.6
|
8151185ede |
fix(ci): Dockerfile COPY invariant check — prevent missing internal/<pkg> Docker failures (#1316) (#1432)
## Summary - Adds `scripts/check-dockerfile-internal-pkgs.sh`: reads `replace => ../../internal/<pkg>` directives from `cmd/server/go.mod` and `cmd/ingestor/go.mod`, then verifies each referenced package has the correct number of `COPY internal/<pkg>/` lines in `Dockerfile` (one per builder section that needs it) - Wired into CI as a step in the `go-test` job, before CSS lint — runs on every PR, adds ~0.1s - Prevents the recurring failure pattern (#1316): new `internal/<pkg>` added to go.mod but COPY line forgotten in Dockerfile; non-Docker CI passes, Docker build fails after merge with a cryptic module error Key details: - Counts COPY occurrences per package: if a pkg is referenced in both go.mods (both binaries need it), it must appear in at least 2 builder sections - Anchored regex: only matches actual `replace` directives (not comments) - Anchored grep: skips commented-out `COPY internal/...` lines Closes #1316. ## Test plan - [ ] Run `bash scripts/check-dockerfile-internal-pkgs.sh` locally — exits 0 on current Dockerfile - [ ] Manually remove a `COPY internal/perfio/` line from Dockerfile → script exits 1 with a clear error - [ ] CI step visible in the `go-test` job on this PR 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Kpa-clawbotandopenclaw-bot
|
e267fb754d |
fix(ci): aggregate e2e pass/fail across all suites instead of broken digits-before-slash regex (#1298)
RED |
||
|
Kpa-clawbotandOpenClaw Bot
|
43203b09b7 |
fix(#1249): IATA badge missing on fixture + mobile clipping (#1252)
Failing test commit: `bdb4eefb` (added in #1189 R1) — original CI failure: https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995819598 Fixes #1249. ## Root cause Two independent bugs surfaced by the same E2E test: 1. **Fixture join broken.** `scripts/capture-fixture.sh` wrote the text observer hash into `observations.observer_idx`, but the v3 join in `cmd/server` is `observers.rowid = observations.observer_idx`. The join silently nulled out `observer_id` / `observer_iata` for every packet. 2. **Mobile clipping.** `.col-observer` had `data-priority=3` (hides at ≤1024px) and was in the narrow-viewport `defaultHidden` list, so at 375px the cell collapsed to `display:none` and `.badge-iata` had a 0×0 box. ## Changes - `test-fixtures/e2e-fixture.db`: remap `observer_idx` text hash → integer rowid (500/500 rows resolved). - `scripts/capture-fixture.sh`: build an `observer_id → rowid` map before insert; skip rows whose observer isn't in the fixture. Comment explains the trap. - `public/packets.js`: bump `.col-observer` priority `3 → 1` and drop `observer` from narrow-viewport `defaultHidden`. ## Verification All three sub-tests in `test-observer-iata-1188-e2e.js` pass locally against the freshened fixture. `curl /api/packets?limit=5` returns real IATA codes (OAK / MRY / SFO) instead of empty strings. Co-authored-by: OpenClaw Bot <bot@openclaw.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
eae1b915ca |
feat: load Aldrich webfont for #1137 logo SVG (#1138)
Adds Aldrich webfont so the merged #1137 logo renders in the intended typeface. ## Problem The inline SVG logo merged in #1137 declares `font-family="Aldrich, monospace"` in `public/index.html` and `public/home.js`, but the page never loaded the Aldrich font face. Browsers silently fell back to monospace. ## Fix Self-hosted webfont: - `public/fonts/aldrich-regular.woff2` — Regular 400, ~16KB, downloaded from Google Fonts (latin subset). Self-hosted to avoid third-party CDN dependency, privacy concern, and FOUT delay. - `@font-face` declaration added at the top of `public/style.css` with `font-display: swap`. Aldrich only ships in 400; the SVG `font-weight="700"` on the wordmark synthesizes bold (matches the design intent of #1137). ## TDD - Red commit: E2E test asserting `document.fonts.check('1em Aldrich')` is true and the navbar SVG `<text>` `font-family` contains "Aldrich". Without the font face declaration, both assertions fail on an assertion (not a build error). - Green commit: adds the woff2 + `@font-face` rule, both assertions pass. ## Files - `public/fonts/aldrich-regular.woff2` (new, 16460 bytes) - `public/style.css` — `@font-face` rule - `test-e2e-playwright.js` — new test --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
494d3022f9 |
Partial fix for #1128: close audit gaps (z-scale, css-var lint, multi-viewport E2E, Bug 1+5 polish) (#1133)
## Partial fix for #1128 — closes the gaps PR #1131 left behind PR #1131 was a partial fix for the packets-page layout chaos (merged 2026-05-06 ~01:55 UTC, then the issue was reopened by the maintainer). #1131 shipped Bug 4 (`--surface` definition), the `.path-popover` flip + lower z-index, the debounced re-measure for Bug 1, the `.filter-bar` row-gap + `.multi-select-trigger` truncation for Bug 3, the new z-index TOKENS, and a single-viewport E2E with five individual-component assertions. This PR closes everything else the issue body and the `specs/packets-layout-audit.md` audit asked for. ### What changed (per gap) **Gap A — apply the z-index scale (audit Section 2)** #1131 added `--z-dropdown` / `--z-popover` / `--z-modal` / `--z-tooltip` but explicitly left existing literal values in place. This PR renumbers the 7 dropdowns/popovers the audit named: | Selector | Before | After | |---|---:|---:| | `.col-toggle-menu` | 50 | `var(--z-dropdown)` (100) | | `.multi-select-menu` | 90 | `var(--z-dropdown)` | | `.region-dropdown-menu` | 90 | `var(--z-dropdown)` | | `.node-filter-dropdown` | 100 | `var(--z-dropdown)` | | `.fux-saved-menu` | `var(--z-tooltip)` (9200) | `var(--z-dropdown)` | | `.fux-ac-dropdown` | `var(--z-tooltip)` | `var(--z-dropdown)` | | `.hop-conflict-popover` | `var(--z-tooltip)` | `var(--z-popover)` (300) | `.fux-ctx-menu` deliberately retains the tooltip band — context menus must float above all toolbar UI. `.region-filter-options-menu` no longer exists in the source (was renamed `.region-dropdown-menu`). The `style.css` doc-block at the top is rewritten to record the applied scale and to point operators at the new lint. **Gap B — CSS-var lint (audit Section 5 #1, "single highest-value addition")** Adds `scripts/check-css-vars.js` (~70 lines). Walks `public/*.css`, extracts every `var(--name)` reference WITHOUT a fallback, asserts the name is defined in some `public/*.css`. References WITH a fallback are tolerated. Wired into CI in the `go-test` job before the JS unit tests. The red commit (`608d81f`) shipped this lint exiting 1 against the master tree — three undefined vars that bypassed earlier review: ``` public/style.css:2628 var(--text-primary) public/style.css:2675 var(--bg-hover) public/style.css:2924 var(--primary) ``` The green commit (`1369d1e`) defines those three as aliases in the :root block (`--text-primary` → `--text`, `--bg-hover` → `--hover-bg`, `--primary` → `--accent`). Light + dark themes inherit through the existing tokens. **Gap C — multi-viewport E2E (issue acceptance criterion)** Adds `test-issue-1128-multi-viewport-e2e.js` — sister of the existing single-viewport test. At each of three viewports (1280×900, 1080×800, 768×1024): - takes a screenshot to `e2e-screenshots/issue-1128-<viewport>.png` - asserts no two `.filter-group` siblings vertically overlap - on desktop+laptop, opens the Saved menu and the Types multi-select and asserts the dropdown does not vertically overlap any `.filter-group` below it Plus three viewport-agnostic assertions: - dropdown selectors compute z-index in `[100,199]` (`.col-toggle-menu`, `.multi-select-menu`, `.region-filter-options-menu`, `.fux-saved-menu`, `.fux-ac-dropdown`) - `.path-hops .hop / .hop-named / .arrow` compute `line-height ≤ 18px` - `.col-path` computes `height ≤ 28px` Wired into the e2e-test job after the existing #1128 test. **Gap D — Bug 5 polish (toolbar reorder)** Audit Section 3 Bug 5: swaps `filter-group-dropdowns` and `filter-group-toggles` in `public/packets.js` so time range + Group by Hash + ★ My Nodes sit next to the search input. Pure markup reorder. No CSS / no JS-handler changes. **Gap E — Bug 1 belt-and-suspenders** Audit Section 3 Bug 1 sub-bullets: - locks `.path-hops .hop / .hop-named / .arrow` to `line-height: 18px` so a chip with mixed font metrics cannot overflow the 22px host vertically and bleed into the row above - converts `.col-path { max-height: 28px }` → `height: 28px` because browsers widely ignore `max-height` on `<td>`s; the earlier rule was a no-op ### TDD discipline (red → green) ``` $ git log --oneline origin/master..HEAD |
||
|
|
364c5766fc |
feat(logo): wire new CoreScope SVG logo into navbar + home hero (#1137)
## Adds new logo and home hero Replaces the navbar mushroom emoji + "CoreScope" text spans with the new CoreScope SVG mark, and adds a hero SVG (with the MESH ANALYZER tagline) above the home page H1. ### What changed - `public/img/corescope-logo.svg` — navbar mark, no tagline (locked "aggressive low-amp chirp" variant: facing-arcs + low-amp chirp connector between the two nodes). - `public/img/corescope-hero.svg` — home hero version, includes the MESH ANALYZER tagline. - `public/index.html` — replaces `<span class="brand-icon">🍄</span><span class="brand-text">CoreScope</span>` with `<img class="brand-logo" src="img/corescope-logo.svg?__BUST__" …>`. `.nav-brand` link still routes to `#/`. `.live-dot` retained. - `public/style.css` — adds `.brand-logo { height: 36px }` (32px on tablet ≤900px). Existing 52px nav height unchanged. - `public/home.js` / `public/home.css` — adds `<img class="home-hero-logo">` above the hero `<h1>`, sized `max-width: min(720px, 90vw)` and centered. ### TDD Red→green is visible in the branch: - `3159b82` — `test(logo): add failing E2E …` (red commit). Adds `test-logo-rebrand-e2e.js` and wires it into the `e2e-test` job in `deploy.yml` with `CHROMIUM_REQUIRE=1`. On this commit `index.html` still has the emoji + text spans, `home.js` has no hero img, and the SVG asset files do not exist — the test asserts on each so CI fails on assertion. - `19434e1` — `feat(logo): wire new CoreScope SVG logo …` (green commit). Implements the fix. ### E2E asserts 1. `.nav-brand img` exists with `src` ending `corescope-logo.svg` 2. legacy `.brand-icon` / `.brand-text` are gone 3. `.live-dot` is present, visible, and to the right of the logo (no overlap) 4. `.home-hero img.home-hero-logo` exists with `src` ending `corescope-hero.svg`, positioned BEFORE the `<h1>` 5. both `/img/corescope-{logo,hero}.svg` return 200 with svg content-type ### Customizer compatibility - `customize.js` still does `querySelector('.brand-text')` / `.brand-icon` for live branding updates. Both now return `null`; existing `if (el)` guards make those branches silent no-ops. **No JS errors, but the customizer's `branding.siteName` and `branding.logoUrl` fields no longer rewrite the navbar brand** — the brand is now a fixed SVG asset. - **Theme accent does NOT recolor the SVG.** SVGs loaded via `<img src>` are isolated documents and cannot inherit document CSS variables; the SVG falls back to its embedded brand colors. This is appropriate for a brand mark; if recoloring per theme is desired later, swap to inline SVG (separate PR). ### Browser validation Local Chromium not available in this env; the E2E test soft-skips locally and hard-fails in CI (`CHROMIUM_REQUIRE=1`). Server-side checks done locally: - `curl http://localhost:13581/` → confirmed `<img class="brand-logo" src="img/corescope-logo.svg?<bust>" …>` rendered, no `.brand-icon`/`.brand-text` spans. - `curl -I /img/corescope-logo.svg` and `/img/corescope-hero.svg` → both 200. ### Performance No hot-path changes. Two new static SVG assets (~7.6KB each), served directly by the Go static handler. Cache-busted via `?__BUST__` (auto-replaced server-side). --------- Co-authored-by: OpenClaw Bot <bot@openclaw.local> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local> |
||
|
|
ac0cf5ac7d |
fix(channels): #1087 QR library + share modal + PSK persistence (#1090)
Red commit: 5def4d073c61058fc9f327a3c60ece27e21cbc69 (CI run pending — see Checks tab) Fixes #1087 ## What's broken (4 bugs) 1. **"QR library not loaded"** — `channel-qr.js` checked `root.QRCode` (capital), but the vendored library exports lowercase `qrcode` (Kazuhiko Arase API). Generate & Show QR always fell into the "library not loaded" branch. 2. **QR encodes `name=psk:hex`** — the Share button (and parts of the Generate path) passed the internal `psk:<hex8>` lookup key to `ChannelQR.generate`, ignoring the user's display label stored in `LABELS_KEY`. 3. **PSK channel doesn't persist on refresh** — the persistence path was scattered, and the read-back wasn't verified. Added channels disappeared on refresh and "reappeared" only when a later add ran the persist hook. 4. **Share button reuses the Add Channel modal** — wrong intent reuse (Add = INPUT, Share = OUTPUT). Replaced with a dedicated `#chShareModal` (separate DOM id, separate title, share-only affordances, privacy warning). ## TDD Red commit (this) lands ONLY the failing tests: - `test-channel-issue-1087.js` — source-string contract assertions for all 4 bugs - `test-channel-issue-1087-e2e.js` — Playwright E2E covering generate → QR render, QR display name, persistence across refresh, Share opens dedicated modal Green commit (follow-up) lands the production fixes. ## E2E assertion added E2E assertion added: test-channel-issue-1087-e2e.js:55 ## CI wiring - `test-channel-issue-1087.js` added to `.github/workflows/deploy.yml` (go-test JS unit step) + `test-all.sh` - `test-channel-issue-1087-e2e.js` added to `.github/workflows/deploy.yml` (e2e-test step) --------- Co-authored-by: bot <bot@corescope> Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: clawbot <clawbot@users.noreply.github.com> |
||
|
Kpa-clawbotandyou
|
a8e1cea683 |
fix: use payload type bits only in content hash (not full header byte) (#787)
## Problem The firmware computes packet content hash as: ``` SHA256(payload_type_byte + [path_len for TRACE] + payload) ``` Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type bits (2-5). CoreScope was using the **full header byte** in its hash computation, which includes route type bits (0-1) and version bits (6-7). This meant the same logical packet produced different content hashes depending on route type — breaking dedup and packet lookup. **Firmware reference:** `Packet.cpp::calculatePacketHash()` uses `getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) & PH_TYPE_MASK`. ## Fix - Extract only payload type bits: `payloadType := (headerByte >> 2) & 0x0F` - Include `path_len` byte in hash for TRACE packets (matching firmware behavior) - Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` ## Tests Added - **Route type independence:** Same payload with FLOOD vs DIRECT route types produces identical hash - **TRACE path_len inclusion:** TRACE packets with different `path_len` produce different hashes - **Firmware compatibility:** Hash output matches manual computation of firmware algorithm ## Migration Impact Existing packets in the DB have content hashes computed with the old (incorrect) formula. Options: 1. **Recompute hashes** via migration (recommended for clean state) 2. **Dual lookup** — check both old and new hash on queries (backward compat) 3. **Accept the break** — old hashes become stale, new packets get correct hashes Recommend option 1 (migration) as a follow-up. The volume of affected packets depends on how many distinct route types were seen for the same logical packet. Fixes #786 --------- Co-authored-by: you <you@example.com> |
||
|
efitenandClaude Sonnet 4.6
|
fe314be3a8 |
feat: geo_filter enforcement, DB pruning, geofilter-builder tool, HB column (#215)
## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Kpa-clawbotandyou
|
a555b68915 |
feat: expand frontend coverage collector for 60%+ target (#275)
## Summary
Expands the parallel frontend coverage collector to boost coverage from
~43% toward 60%+. This covers Phases 1 and 2 of the coverage improvement
plan.
### Phase 1 — Visit unvisited pages
- **Compare page** (`#/compare`): Navigates with query params selecting
two real observers from fixture DB, also exercises UI controls
- **Node analytics** (`#/nodes/{pubkey}/analytics`): Visits analytics
for two real nodes from fixture DB, clicks day buttons
- **Traces search** (`#/traces`): Searches for two valid packet hashes
from fixture DB
- **Personalized home**: Sets `localStorage.myNodes` with real pubkeys
before visiting `#/home`
- **Observer detail pages**: Direct navigation to
`#/observers/test-obs-1` and `#/observers/test-status-obs`
- **Real packet detail**: Navigates to `#/packets/b6b839cb61eead4a`
(real hash)
- **Rapid route transitions**: Exercises destroy/init cycles across all
pages
- **Compare in route list**: Added to the full route transition exercise
### Phase 2 — page.evaluate() for interactive code paths
| File | Functions exercised |
|------|-------------------|
| **live.js** | `vcrPause`, `vcrSpeedCycle`, `vcrReplayFromTs`,
`drawLcdText`, `vcrResumeLive`, `vcrUnpause`, `vcrRewind`,
`updateVCRClock`, `updateVCRLcd`, `updateVCRUI`, `bufferPacket`
(synthetic WS packets), `dbPacketToLive`, `renderPacketTree` |
| **packets.js** | `renderDecodedPacket` (ADVERT + GRP_TXT), `obsName`,
`renderPath`, `renderHop` |
| **packet-filter.js** | 30+ filter expressions now **evaluated against
4 synthetic packets** (previously only compiled, not run). Covers
`resolveField` for all field types including `payload.*` dot notation |
| **nodes.js** | `getStatusInfo`, `renderNodeBadges`,
`renderStatusExplanation`, `renderHashInconsistencyWarning` with varied
node types/roles |
| **roles.js** | `getHealthThresholds` (all roles), `getNodeStatus` (all
roles × active/stale), `getTileUrl`, `syncBadgeColors`, `miniMarkdown`
(bold, italic, code, links, lists), `copyToClipboard` |
| **channels.js** | `hashCode`, `getChannelColor`, `getSenderColor`,
`highlightMentions`, `formatSecondsAgo` |
| **app.js** | `escapeHtml`, `debouncedOnWS`, extended
`timeAgo`/`truncate` edge cases, extended
`routeTypeName`/`payloadTypeName`/`payloadTypeColor` ranges |
### What changed
- `scripts/collect-frontend-coverage.js` — +336 lines across existing
groups (no new groups added)
### Testing
- `npm test` passes (all 13 tests)
- No other files modified
Co-authored-by: you <you@example.com>
|
||
|
Kpa-clawbotandyou
|
5777780fc8 |
refactor: parallel coverage collector (~30-60s vs 8min) (#272)
## Summary Redesigned frontend coverage collector with 7 parallel browser contexts. Coverage collector runs on master pushes only (skipped on PRs). ### Architecture 7 groups run simultaneously via `Promise.allSettled()`: - G1: Home + Customizer - G2: Nodes + Node Detail - G3: Packets + Packet Detail - G4: Map - G5: Analytics + Channels + Observers - G6: Live + Perf + Traces + Globals - G7: Utility functions (page.evaluate) ### Speed gains - `safeClick` 500ms → 100ms - `navHash` 150ms → 50ms - Removed redundant page visits and E2E-duplicate interactions - Wall time = slowest group (~30-60s estimated) ### 821 lines → ~450 lines Each group writes its own coverage JSON, nyc merges automatically. ### CI behavior - **PRs:** Coverage collector skipped (fast CI) - **Master:** Coverage collector runs (full synthetic user validation) Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
f438411a27 |
chore: remove deprecated Node.js backend (-11,291 lines) (#265)
## Summary Removes all deprecated Node.js backend server code. The Go server (`cmd/server/`) has been the production backend — the Node.js server was kept "just in case" but is no longer needed. ### Removed (19 files, -11,291 lines) **Backend server (6 files):** `server.js`, `db.js`, `decoder.js`, `server-helpers.js`, `packet-store.js`, `iata-coords.js` **Backend tests (9 files):** `test-decoder.js`, `test-decoder-spec.js`, `test-server-helpers.js`, `test-server-routes.js`, `test-packet-store.js`, `test-db.js`, `test-db-migration.js`, `test-regional-filter.js`, `test-regional-integration.js` **Backend tooling (4 files):** `tools/e2e-test.js`, `tools/frontend-test.js`, `benchmark.js`, `benchmark-ab.sh` ### Updated - `AGENTS.md` — Rewritten architecture section for Go, explicit deprecation warnings - `test-all.sh` — Only runs frontend tests - `package.json` — Updated test:unit - `scripts/validate.sh` — Removed Node.js server syntax check - `docker/supervisord.conf` — Points to Go binary ### NOT touched - `public/` (active frontend) ✅ - `test-e2e-playwright.js` (frontend E2E tests) ✅ - Frontend test files (`test-packet-filter.js`, `test-aging.js`, `test-frontend-helpers.js`) ✅ - `package.json` / Playwright deps ✅ ### Follow-up - Server-only npm deps (express, better-sqlite3, mqtt, ws, supertest) can be cleaned from package.json separately - `Dockerfile.node` can be removed separately --------- Co-authored-by: you <you@example.com> |
||
|
|
ec7ae19bb5 |
ci: restructure pipeline — sequential fail-fast, Go server E2E, remove deprecated JS tests (#256)
## Summary Complete CI pipeline restructure. Sequential fail-fast chain, E2E tests against Go server with real staging data, all deprecated Node.js server tests removed. ### Pipeline (PR): 1. **Go unit tests** — fail-fast, coverage + badges 2. **Playwright E2E** — against Go server with fixture DB, frontend coverage, fail-fast on first failure 3. **Docker build** — verify containers build ### Pipeline (master merge): Same chain + deploy to staging + badge publishing ### Removed: - All Node.js server-side unit tests (deprecated JS server) - `npm ci` / `npm run test` steps - JS server coverage collection (`COVERAGE=1 node server.js`) - Changed-files detection logic - Docs-only CI skip logic - Cancel-workflow API hacks ### Added: - `test-fixtures/e2e-fixture.db` — real data from staging (200 nodes, 31 observers, 500 packets) - `scripts/capture-fixture.sh` — refresh fixture from staging API - Go server launches with `-port 13581 -db test-fixtures/e2e-fixture.db -public public-instrumented` --------- Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> Co-authored-by: you <you@example.com> |
||
|
you
|
13cab9bede |
perf: optimize frontend coverage collector (~2x faster)
Three optimizations to reduce wall-clock time: 1. Reduce safeClick timeout from 3000ms to 500ms - Elements either exist immediately after navigation or don't exist at all - ~75 safeClick calls; if ~30 miss, saves ~75s of dead wait time 2. Replace 18 page.goto() calls with SPA hash navigation - After initial page load, the SPA shell is already in the DOM - page.goto() reloads the entire page (network round-trip + parse) - Hash navigation via location.hash triggers the SPA router instantly - Only 3 page.goto() remain: initial load + 2 home page loads after localStorage.clear() 3. Remove redundant final route sweep - All 10 routes were already visited during the page-specific sections - The sweep just re-navigated to pages that had already been exercised - Saves ~2s of redundant navigation Also: - Reduce inter-route wait from 200ms to 50ms (SPA router is synchronous) - Merge utility function + packet filter exercises into single evaluate() call - Use navHash() helper for consistent hash navigation with 150ms settle time |
||
|
you
|
12d1174e39 |
perf: speed up frontend coverage tests (~3x faster)
Three optimizations to the CI frontend test pipeline:
1. Run E2E tests and coverage collection concurrently
- Previously sequential (E2E ~1.5min, then coverage ~5.75min)
- Now both run in parallel against the same instrumented server
- Expected savings: ~5 min (coverage runs alongside E2E instead of after)
2. Replace networkidle with domcontentloaded in coverage collector
- SPA uses hash routing — networkidle waits 500ms for network silence
on every navigation, adding ~10-15s of dead time across 23 navigations
- domcontentloaded fires immediately once HTML is parsed; JS initializes
the route handler synchronously
- For in-page hash changes, use 200ms setTimeout instead of
waitForLoadState (which would never re-fire for same-document nav)
3. Extract coverage from E2E tests too
- E2E tests already exercise the app against the instrumented server
- Now writes window.__coverage__ to .nyc_output/e2e-coverage.json
- nyc merges both coverage files for higher total coverage
Also:
- Split Playwright install into browser + deps steps (deps skip if present)
- Replace sleep 5 with health-check poll in quick E2E path
|
||
|
Kpa-clawbotandCopilot
|
72161ba8fe |
perf: replace 169 blind sleeps with Playwright waits in coverage script
Remove all 169 waitForTimeout() calls (totaling 104.1s of blind sleeping)
from scripts/collect-frontend-coverage.js:
- Helper functions (safeClick, safeFill, safeSelect, clickAll, cycleSelect):
removed 300-400ms waits after every interaction — Playwright's built-in
actionability checks handle waiting for elements automatically
- Post-navigation waits: removed redundant sleeps after page.goto() calls
that already use waitUntil: 'networkidle'
- Hash-change navigations: replaced waitForTimeout with
waitForLoadState('networkidle') for proper SPA route settling
- Toggle/button waits: removed — event handlers execute synchronously
before click() resolves
- Post-evaluate waits: removed — evaluate() is synchronous
Local benchmark (Windows, sparse test data):
Before: 744.8s
After: 484.8s (35% faster, 260s saved)
On CI runner (ARM Linux with real mesh data), savings will be
proportionally better since most elements exist and the 104s
of blind sleeping was the dominant bottleneck.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
you
|
9b0c740537 |
revert: aggressive coverage interactions dropped score from 42% to 39%
The page.evaluate() calls corrupting localStorage and firing fake events caused page error-reloads, losing accumulated coverage. Reverting to the 42% version which was the actual high water mark. |
||
|
you
|
556e3b19db |
coverage: aggressive branch coverage push — target 80%+
Add ~900 lines of deep branch-coverage interactions: - Utility functions with all edge cases (timeAgo, truncate, escapeHtml, formatHex, etc.) - roles.js: getHealthThresholds/getNodeStatus for all roles + edge inputs - PacketFilter: compile+match with mock packets, all operators, bad expressions - HopResolver/HopDisplay: init, resolve, renderPath with various inputs - RegionFilter: onChange, getSelected, isEnabled, setRegions, render - Customize: deep tab cycling, import/export, bad JSON, theme preview - WebSocket reconnection trigger - Keyboard shortcuts (Ctrl+K, Meta+K, Escape) - Window resize (mobile/tablet/desktop) for responsive branches - Error routes: nonexistent nodes/packets/observers/channels - localStorage corruption to trigger catch branches - Theme toggling (dark/light rapid switching, custom vars) - Live page: VCR modes, timeline clicks, speed cycling, all toggles - Audio Lab: play/stop/loop, BPM/volume sliders, voice selection - All analytics tabs via deep-link + sort headers - Packets: complex filter expressions, scroll-to-load, double-click - Nodes: special char search, all sort columns, fav stars - Channels: resize handle drag, theme observer, node tooltips - Observers/Observer Detail: sort, tabs, day cycling - Node Analytics: day buttons, tabs - Home: both new/experienced flows, empty search results - debouncedOnWS/onWS/offWS exercise |
||
|
you
|
86dce6f350 |
coverage: massively expand frontend interaction coverage
Exercise every major code path across all frontend files: app.js: all routes, bad routes, hashchange, theme toggle x4, hamburger menu, favorites dropdown, global search, Ctrl+K, apiPerf(), timeAgo/truncate/routeTypeName utils nodes.js: sort every column (both directions), every role tab, every status filter, cycle all Last Heard options, click rows for side pane, navigate to detail page, copy URL, show all paths, node analytics day buttons (1/7/30/365), scroll target packets.js: 12 filter expressions including bad ones, cycle all time windows, group by hash toggle, My Nodes toggle, observer menu, type filter menu, hash input, node filter, observer sort, column toggle menu, hex hash toggle, pause button, resize handle, deep-link to packet hash map.js: all role checkboxes toggle, clusters/heatmap/neighbors/ hash labels toggles, cycle Last Heard, status filter buttons, jump buttons, markers, zoom controls, dark mode tile swap analytics.js: all 9 tabs clicked, deep-link to each tab via URL, observer selector on topology, navigate rows on collisions/ subpaths, sortable headers on nodes tab, region filter customize.js: all 5 tabs, all preset themes, branding text inputs, theme color inputs, node color inputs, type color inputs, reset buttons, home tab fields (hero, journey steps, checklist, links), export tab, reset preview/user theme live.js: VCR pause/speed/missed/prompt buttons, all visualization toggles (heat/ghost/realistic/favorites/matrix/rain), audio toggle + BPM slider, timeline click, resize event channels.js: click rows, navigate to specific channel observers.js: click rows, navigate to detail, cycle days select traces.js: click rows perf.js: refresh + reset buttons home.js: both chooser paths, search + suggest, my-node cards, health/packets buttons, remove buttons, toggle level, timeline Also exercises packet-filter parser and region-filter directly. |
||
|
you
|
26d4bbc39d | ci: fix coverage collector — use Playwright bundled chromium on CI | ||
|
you
|
5ee976055b |
feat(coverage): add targeted Playwright interactions for higher frontend coverage
Add redundant selectors (data-sort, data-role, data-status, data-tab, placeholder-based search, emoji theme toggle, .cust-close, #fTimeWindow, broader preset selectors) to exercise more frontend code paths. All interactions wrapped in try/catch for resilience. |
||
|
you
|
860d5c574e |
test: expanded frontend coverage collection with page interactions
367 lines of Playwright interactions covering nodes, packets, map, analytics, customizer, channels, live, home pages. Fixed e2e channels assertion (chList vs chResp.channels). |
||
|
you
|
d7faa4d978 |
Add frontend code coverage via Istanbul instrumentation + Playwright
- Install nyc for Istanbul instrumentation - Add scripts/instrument-frontend.sh to instrument public/*.js - Add scripts/collect-frontend-coverage.js to extract window.__coverage__ - Add scripts/combined-coverage.sh for combined server+frontend coverage - Make server.js serve public-instrumented/ when COVERAGE=1 is set - Add test:full-coverage npm script - Add public-instrumented/ and .nyc_output/ to .gitignore |
||
|
you
|
91a6a2c525 | fix: migration handles concurrent dual-write with INSERT OR IGNORE | ||
|
you
|
d7e415daa7 | fix: raw_hex NOT NULL in transmissions schema — deleted 4 junk test rows | ||
|
you
|
2c6148fd2d |
Add dedup migration script (Milestone 1)
Creates transmissions and observations tables from existing packets table. - Groups packets by hash → 1 transmission per unique hash - Creates 1 observation per original packet row with FK to transmission - Idempotent: drops and recreates new tables on each run - Does NOT modify the original packets table - Prints stats and verifies counts match Tested on test DB: 33813 packets → 11530 transmissions (2.93x dedup ratio) |
||
|
you
|
10b11106f6 |
ci: add pre-deploy JS validation — syntax check + undefined reference detection
Validation runs BEFORE docker build. If it fails, deployment is blocked. No more broken code reaching production. |