mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-07 18:14:03 +00:00
6dbfd331a6919ce7525c3fa006d308db2fbbbcc2
1981
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kpa-clawbot
|
6dbfd331a6 | ci: update go-ingestor-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
a00e1c0e18 | ci: update frontend-tests.json [skip ci] | ||
|
Kpa-clawbot
|
763d4f707c | ci: update frontend-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
ad467daeeb | ci: update e2e-tests.json [skip ci] | ||
|
Kpa-clawbot
|
46ce9590f1 |
fix(#1270): Prefix Tool Network Overview shows configured-hash-size counts, not math-only slices (#1271)
Red commit: `6b68080c24106301b6bfc25f8a05484f07d0612d` (test added that fails on master). CI: see Checks tab on this PR. Fixes #1270. ## Problem Two analytics surfaces told contradictory stories about prefix usage: - **Prefix Tool → Network Overview** showed e.g. `168 / 65,536` for the 2-byte tier — a pure math fact: every repeater pubkey sliced to 2 bytes yields N distinct values. Because collisions are rare, this number always equals (or nearly equals) the repeater count, making it look like the whole network uses 2-byte hashing. - **Hash Stats → By Repeaters** showed configured-hash-size counts straight from `/api/analytics/hash-sizes` `distributionByRepeaters` — usually a minority on 2-byte and near-zero on 3-byte. The Prefix Tool was presenting a math fact as if it were operational truth. ## Fix `renderPrefixTool` now also fetches `/api/analytics/hash-sizes` and restructures each tier card into three labeled stats with explicit hierarchy: 1. **Primary** — `X of Y repeaters configured` (from `distributionByRepeaters`). Same source the Hash Stats tab uses, so the two pages agree exactly. 2. **Operational collisions** — colliding slices among repeaters configured for *this* hash size only (matches Hash Issues semantics). 3. **Theoretical** (secondary, smaller, dashed-rule footnote) — `X unique N-byte slices across all repeater pubkeys (of Y possible)`. The math fact is preserved as educational info, no longer impersonating operational truth. The "Total repeaters" card now also notes how many have a known configured hash size. The "About these numbers" footer was rewritten to explain the three numbers and link to both Hash Stats and Hash Issues. The prefix collision detector (Check / Generate panels) is unchanged — it still scans every repeater pubkey because that is its job. ## Test Added `#1270 Prefix Tool primary counts match Hash Stats By Repeaters` to `test-e2e-playwright.js`. It fetches `/api/analytics/hash-sizes` for the ground-truth `distributionByRepeaters`, then visits `#/analytics?tab=prefix-tool`, opens Network Overview, and scrapes the primary count via a new `data-pt-configured="<bytes>"` `data-value="<count>"` marker on each tier card, asserting exact equality for 1/2/3-byte. - Red commit `6b68080c` (test only): fails on master with `NO data-pt-configured marker`. - Green commit `12ed2789` (fix): test passes; full E2E suite `123/126 passed, 3 skipped`. ## Acceptance - [x] Prefix Tool Network Overview shows configured-hash-size repeater counts as the primary number - [x] "Unique slices" math is shown as secondary/educational - [x] Two pages tell the same story (E2E asserts byte-equal match) - [x] E2E asserts the configured-count matches what Hash-Sizes tab shows at the same point in time |
||
|
Kpa-clawbot
|
0022c8fd1f | ci: update go-server-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
7827c8e778 | ci: update go-ingestor-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
cb0218fc4d | ci: update frontend-tests.json [skip ci] | ||
|
Kpa-clawbot
|
385f49b3d8 | ci: update frontend-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
f4ecc96ccc | ci: update e2e-tests.json [skip ci] | ||
|
Kpa-clawbotandopenclaw-bot
|
78b666c248 |
fix(#1267): mobile VCR bar invisible — JS height clobbered bottom-nav reserve (#1269)
## Summary Mobile-only regression: on the Live page at ≤768px viewports the VCR bar was rendered behind the fixed bottom-nav and never visible to the user. iOS Safari screenshot at 375x812 showed: top header strip, full-height map, bottom-nav — **no VCR row at all**. Fixes #1267. ## Root cause `public/live.js` `initResizeHandler` (the existing JS height override) was setting `page.style.height = window.innerHeight + 'px'`, which clobbered the CSS rule that already subtracts `--bottom-nav-reserve` from the live-page height. Because `.live-page` then spanned the full viewport, the VCR bar (`position:absolute; bottom:0; z-index:1000`) was painted underneath `.bottom-nav` (`position:fixed; z-index:1200`). The VCR bar element WAS in the DOM, WAS `display: flex`, and HAD `height: 53px` — it just sat at y=758..812 underneath the bottom-nav at y=754..812. CSS-only checks for `display:none` would never catch this; the test asserts the bar's bottom edge is at or above the bottom-nav's top edge. ## Fix One-liner in spirit: subtract the bottom-nav height before applying `page.style.height`. The implementation measures the rendered `.bottom-nav` (with a fallback to a hidden probe that resolves the `--bottom-nav-reserve` token), so it survives safe-area inset and the bottom-nav's 1px border. ```js const reserve = /* measure .bottom-nav, fall back to --bottom-nav-reserve token */; const h = Math.max(0, window.innerHeight - reserve); ``` Desktop is unchanged: `.bottom-nav` is `display: none`, the probe resolves to 0, and `h === window.innerHeight` exactly as before. ## TDD - **RED** (commit 1): `test-e2e-1267-mobile-vcr.js` — Playwright at iPhone 375x812 asserts `.vcr-bar` has `display !== 'none'`, `visibility !== 'hidden'`, `height > 0`, `top < viewport.height`, and (the key check) `bottom <= bottom-nav.top`. Fails on `master` with: *"VCR bar bottom 812 overlaps bottom-nav top 754"*. - **GREEN** (commit 2): the fix above. Test passes: *"VCR bar bottom 754 ≤ bottom-nav top 754"*. ## Verification - ✅ Mobile (375x812) repro reproduced against `master` (bar at y=758..812, behind bottom-nav) - ✅ Mobile (375x812) E2E green after fix (bar at y=700..754, flush above bottom-nav) - ✅ Desktop (1440x900) unaffected — bottom-nav hidden, page height = viewport height as before, VCR bar at viewport bottom - ✅ #1234 (top-nav hidden on /live), #1246 (single-row VCR), #1206/#1213 (VCR/feed clearance) unchanged — none touched ## Files - `public/live.js` — single function (`initResizeHandler`) modified - `test-e2e-1267-mobile-vcr.js` — new mobile-viewport Playwright regression test Run: `BASE_URL=http://localhost:13581 node test-e2e-1267-mobile-vcr.js` --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbot
|
d4d569278d | ci: update go-server-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
ee21eafa66 | ci: update go-ingestor-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
c6a90e9896 | ci: update frontend-tests.json [skip ci] | ||
|
Kpa-clawbot
|
92518ab234 | ci: update frontend-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
cd84f51f8a | ci: update e2e-tests.json [skip ci] | ||
|
Kpa-clawbotandCoreScope Bot
|
4cd8445233 |
perf(#1265): wire /api/observers/clock-skew + /api/nodes/clock-skew into analytics recomputer (#1266)
RED:
|
||
|
Kpa-clawbotandcorescope-bot
|
ae17a2be12 |
perf(#1262): /api/nodes?limit=2000 cold-miss 15.7s → <100ms — prewarm repeater enrichment cache (#1263)
RED commit: `22ce5736066142583017cad7303fa48d9e00ccf0` — CI on red: https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1262 ## Problem After #1260 added a 15s-TTL bulk cache for repeater enrichment in `handleNodes`, `/api/nodes` (default limit) dropped to ~500ms. But `/api/nodes?limit=2000` — called by `public/live.js` at SPA startup for hop resolution — still took **15.7s cold** on staging (75k tx, 600 nodes). Warm hits were ~40ms. Root cause: the bulk cache was lazily populated on the first request after TTL expiry. The rebuild ran on the request-serving goroutine. Every cold SPA load triggered the rebuild and ate 15s. ## Fix Add `StartRepeaterEnrichmentRecomputer` — a steady-state background recomputer that mirrors the `analytics_recomputer.go` pattern from #1240: - **Prewarm**: initial synchronous compute on Start so the first request hits a populated cache. - **Steady-state**: ticker refreshes the snapshot every 5min (configurable via the existing analytics recompute interval knob). - **Panic-safe** + idempotent Start. Wired into `main.go` right after `StartAnalyticsRecomputers`, using `cfg.GetHealthThresholds().RelayActiveHours` as the window. ## Test `TestHandleNodesLimit2000ColdMiss` — seeds 600 nodes + 150k non-advert tx with repeaters indexed under a shared 1-byte hop prefix (matches production hop-prefix collisions), starts the recomputer, then issues `/api/nodes?limit=2000` with **no HTTP warmup**. | State | Latency | |---|---| | Before (master, on-thread rebuild) | 3.37s | | After (prewarm + steady-state) | 56ms | | Budget | 2s | Staging end-to-end: 15.7s → expected sub-100ms on the same call path. Red commit (`22ce5736066142583017cad7303fa48d9e00ccf0`) compiles with a no-op stub of the new method so the test fails on the latency **assertion**, not a missing symbol. Fixes #1262 --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandcorescope-bot
|
094a96bd6c |
perf(#1258): /#/perf — parallel health fetch, sort endpoints, pause refresh while hidden (#1261)
Fixes #1258 — Perf dashboard (/#/perf) was slow because of three frontend issues; backend APIs were never the problem. ## Findings 1. **`/api/health` fetched sequentially after `Promise.all`** in `refresh()` — added a full RTT (~50-200ms) on every 5s tick on top of the parallel batch. 2. **Endpoints table not actually sorted** despite the heading "sorted by total time". JSON shape is `map[string]EndpointStatsResp` (no defined order); frontend rendered map iteration order. Visible correctness bug surfaced during investigation. 3. **`setInterval(refresh, 5000)` kept firing while tab was hidden**, rebuilding the entire ~10-section `innerHTML` (cards + 3 tables) in the background. On tab return the user saw a backlog thrash + felt the page was "slow to render". ## Fix (`public/perf.js`) - Move `/api/health` into the same `Promise.all` as the other 4 endpoints — saves one RTT per refresh. - Sort `Object.entries(server.endpoints)` by `count * avgMs` DESC client-side. - Add `document.hidden` guard in the interval tick + `visibilitychange` listener that refreshes once on return; `destroy()` removes the listener. ## Tests `test-perf-render-1258.js` (new): - All 5 initial fetches issued in parallel (including `/api/health`) - Refresh suppressed while `document.hidden` - Endpoints table sorted by total time DESC, regardless of input map order RED commit first (`6b54f9e8`, 0/3 pass) → GREEN commit (`be81303b`, 3/3 pass). Existing `test-perf-go-runtime.js` (13/13) and `test-perf-disk-io-1120.js` (15/15) still green. ## Investigation exemption No Playwright timing test — sandbox can't run a real browser. Static analysis + render-shape unit tests cover the three identified bottlenecks. Documented per AGENTS "investigation surfaces" exemption. ## Measurement Before: refresh = parallel batch (~max(server-side)) + sequential `/api/health` (~50ms) + full innerHTML rebuild every 5s including hidden tabs. After: refresh = single parallel batch, runs only while visible. Expected improvement on tab-return ≈ -1 RTT per refresh + zero background work. --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandcorescope-bot
|
1efe93d7f6 |
perf(#1257): bulk-cache repeater enrichment in /api/nodes — 32s → <500ms (#1260)
RED commit `a2879e12` — perf regression test; CI run: see Actions tab. Fixes #1257. ## Root cause `handleNodes` looped over the response page and called `store.GetRepeaterRelayInfo(pk, win)` + `store.GetRepeaterUsefulnessScore(pk)` for every repeater/room. Each call: - grabbed its own `s.mu.RLock`, - walked `byPathHop[pk]` (+ the matching 1-byte raw-prefix bucket, which on busy networks fans out to nearly the entire non-advert tx set), - and re-parsed every `tx.FirstSeen` with `parseRelayTS`. Default page is the 50 most-recently-seen nodes — almost all hot repeaters — so the request did O(50) lock acquisitions and hundreds of thousands of timestamp parses on the same set of txs. That's the classic load-then-paginate / per-row N+1 shape called out in the issue (same family as #1226). The `?limit=2000` variant looks faster relatively only because per-node enrichment dwarfs serialization; on staging both still bottleneck on the same loop. ## Fix Two new bulk methods on `PacketStore`: - `GetRepeaterRelayInfoMap(windowHours)` → `pubkey → RepeaterRelayInfo` - `GetRepeaterUsefulnessScoreMap()` → `pubkey → 0..1` Both snapshot `byPathHop` under a single `RLock`, pre-parse each `FirstSeen` exactly once (a tx that appears in N hop buckets used to be parsed N times), and emit one entry per hop key. Cached 15s — same TTL as `GetNodeHashSizeInfo` / `GetMultiByteCapMap`, same status-column freshness budget. `handleNodes` is one map-lookup per node; behavior, output schema, and `RelayActive` / `RelayCount{1h,24h}` / `LastRelayed` / `usefulness_score` semantics are preserved. ## Why no `limit` default change The issue mentioned a default-limit knob. Investigated: `queryInt(r, "limit", 50)` already defaults to 50 — frontends calling `/api/nodes` (no limit) get a 50-row page today. Capping further would change behavior (live.js already passes `?limit=2000` when it wants more); the cost was per-repeater enrichment, not page size. Fixing the N+1 is the correct lever and preserves backward compat. ## Perf Regression test `TestHandleNodesPerfLargeFleet` (600 nodes, 150k non-advert tx, repeaters indexed under `byPathHop`): | | elapsed | vs 2s budget | |---|---|---| | before (master) | 4.72s | ✗ | | after | ~4ms | ✓ (~1000×) | ## TDD - RED: `a2879e12` — test fails at 4.72s on master. - GREEN: `c529d29a` — fix; full `cmd/server` + `cmd/ingestor` suites green. --------- Co-authored-by: corescope-bot <bot@corescope> |
||
|
Kpa-clawbotandopenclaw-bot
|
f81ed5b3cf |
perf(#1256): wire /api/analytics/roles into steady-state recomputer (#1259)
RED commit: `0190466d` — failing CI: https://github.com/Kpa-clawbot/CoreScope/actions (will populate after PR creation) ## Problem On staging (commit `d69d9fb`, 78k tx, 2.3M obs), `curl http://localhost/api/analytics/roles` times out at 60s with 0 bytes — the Roles tab is unusable. Issue #1256. PR #1248's steady-state recomputer fan-out (topology / rf / distance / channels / hash-collisions / hash-sizes) **didn't include roles**. The legacy handler: 1. Holds `s.mu.RLock` for the entire compute. 2. Calls `GetFleetClockSkew()`, which drives `clockSkew.Recompute(s)` over all ADVERT transmissions — O(78k) per request. 3. Concurrent ingest writers compound the latency through writer-starvation. Result: every request hits the cold path; the response never comes back inside the 60 s HTTP budget. ## Fix Add `roles` as the 7th endpoint in the recomputer fan-out — same pattern as #1248: - `PacketStore.recompRoles` slot, registered in `StartAnalyticsRecomputers` with default 5-min interval. - `PacketStore.GetAnalyticsRoles()` → atomic-pointer load from the snapshot (sub-ms), with a `computeAnalyticsRoles()` fallback only for the brief startup window before the initial sync compute completes. - Handler is now a thin wrapper — no lock-held work on the request path. - New optional `roles` key under `analytics.recomputeIntervalSeconds` in config; `config.example.json` and `_comment_analytics` updated. ## Latency (unit-scope benchmark) - Worst-of-50 handler latency: **<100 ms** (test budget; well under the 2 s p99 acceptance). - Compute itself is bounded by the existing 5-min recompute window — it runs once in the background, never on the request path. ## Tests - RED `0190466d`: asserts `recompRoles` is registered and the handler returns under the latency budget. Fails on master with `recompRoles not registered`. - GREEN `d7784f76`: registers the recomputer + snapshot accessor — both tests pass. Fixes #1256 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
d69d9fbf8e |
perf(#1247): surgical fix for resolveWithContext tier-1 hot path (4.6× speedup) (#1253)
## Summary Surgical fix for #1247: analytics endpoints regressed 3-9× between prod `d818527` and master. pprof against staging traced the regression to `resolveWithContext` tier-1 affinity loop running on every analytics `resolveHop` call (post-#1198 plumbing) with redundant per-(cand, ctx) work. **Result: 4.6× speedup on the synthetic hot-shape benchmark (202µs → 44µs / op).** ## Root cause - PR #1198 (`353c5264`) lit up `resolveWithContext` tier 1 from every analytics resolveHop closure (previously they passed `contextPubkeys=nil` and short-circuited the entire tier-1 block). - The inner loop did `N_cand × N_ctx` iterations where each one did: - `graph.Neighbors(strings.ToLower(ctxPK))` — graph RLock + ToLower allocation **per candidate**, redundantly - `strings.ToLower(cand.PublicKey)` per `ctxPK` - `strings.EqualFold(otherPK, ctxPK)` + `EqualFold(otherPK, candPK)` — both sides were already lowercased (`NeighborEdge.NodeA/B` via `makeEdgeKey`; `contextPubkeys` via `buildHopContextPubkeys`) - At staging scale (5k+ contextPubkeys × 30k+ resolveHop calls) this dominated `computeAnalyticsTopology` (37% of its CPU) and `computeAnalyticsRF` (55%). ## pprof attribution (staging, region-keyed queries bypassing #1240 cache) ``` computeAnalyticsTopology cum: 19.24% (5.45s / 28.32s sampled) └─ resolveWithContext 37% ├─ strings.ToLower 41% ├─ strings.EqualFold 28% └─ graph.Neighbors 24% computeAnalyticsRF cum: 10.38% ``` ## Fix (~80 LoC in `cmd/server/store.go`) 1. Lowercase `contextPubkeys` **once per call**, skipped entirely when already lowercased (the analytics fast path). 2. Lowercase candidate pubkeys **once per call**. 3. Invert the loop nesting: outer-ctx / inner-edge / candidate-map lookup. `graph.Neighbors` is called once per context pubkey instead of `N_cand` times. 4. Raw `==` instead of `strings.EqualFold` for pubkey comparisons (both sides lowercased by step 1/2). 5. Added a tiny `hasUpperASCII` byte-loop helper next to `isHexLower` for the fast-path check. Behavior preserved: same `Score × Confidence` formula, same tier-1 ratio + min-observations gate, same per-candidate "best edge wins" semantics. No change to tiers 2/3/4. ## TDD evidence - Red commit (`5f8d1564`): `TestResolveWithContextTier1Floor` asserts `<100 µs/call` on the hot shape. **199 µs/call on regressed master → FAIL.** - Green commit (`e3bdbc65`): surgical fix lands. **44 µs/call → PASS.** - Reverification: locally stashed the fix, ran the test → 199.5 µs FAIL; popped fix → 44 µs PASS. `BenchmarkResolveWithContextTier1Hot` (no assertion, visibility only): ``` before: 202013 ns/op 168 B/op 3 allocs/op after: 44084 ns/op 424 B/op 6 allocs/op speedup: 4.6× ``` (Post-fix allocs are O(N_cand + N_ctx) one-time helper tables — net win at hot scale.) ## Independence from #1248 PR #1248 caches the analytics compute output so user-facing latency is sub-ms even when the compute is slow. That's correct for UX but it masks the regression. This PR repairs the compute itself, so: - Region-keyed and windowed queries (which bypass the recomputer cache by design — see #1240) become fast again. - Future ingest scale or feature work on top of the regressed baseline doesn't compound. ## Out of scope - The geo-rejection (#1228) and Confidence weighting (#1229) commits — kept intact, they protect correctness and were not the dominant CPU cost. - Reverting any suspect commit — surgical only. ## Acceptance criteria from #1247 - [x] pprof confirms the hot function (`resolveWithContext`) - [x] Bisect identifies the regressing commit (`353c5264` / PR #1198 — context plumbing; ratified by pprof, no need to actually rebuild 5 binaries) - [x] Fix lands; tier-1 hot path 4.6× faster - [x] No regression in disambiguator correctness — full `go test ./...` green, all existing `ResolveWithContext` / `HopDisambig` / `NeighborGraph` / `Affinity` tests pass Fixes #1247 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandOpenClaw Bot
|
1d33ac53b0 |
fix(#1254): trim .badge-iata h-padding on mobile to clear 1.25px clip (#1255)
Fixes #1254. Master CI Playwright fail-fast on every push since #1252: ``` ❌ Mobile viewport (375px): observer IATA badge stays visible — not clipped: .badge-iata right edge 376.25 exceeds 375px viewport ``` ## Root cause After #1252 unhid `.col-observer` at narrow widths so the IATA pill from #1188 renders on mobile, at 375px the cell padding + truncated observer name (10 chars in grouped rows) + `.badge-iata` pill (`padding: 1px 5px` + `margin-left: 4px`) sums to ~376.25px — overflowing the viewport by 1.25px. Same class of failure as #1250/#1251 (VCR LCD-clip). ## Fix `public/style.css` — inside the existing `@media (max-width: 640px)` block, shrink `.badge-iata` `padding: 1px 5px → 1px 3px` and `margin-left: 4px → 2px`. Reclaims ~6px horizontally, well clear of the 1.25px overflow. Desktop (≥641px) styling untouched. ## TDD The failing E2E sub-test in `test-observer-iata-1188-e2e.js` (added in #1189 R1) IS the red. Mutation verified locally: | Variant | Result | |--------------------|--------| | WITHOUT this fix | ❌ `.badge-iata right edge 376.25 exceeds 375px viewport` | | WITH this fix | ✅ all 3 sub-tests pass | ## Local verification ``` $ go build -o /tmp/corescope-server ./cmd/server $ /tmp/corescope-server -port 13581 -db test-fixtures/e2e-fixture.db -public public & $ CHROMIUM_PATH=/usr/bin/chromium BASE_URL=http://localhost:13581 \ node test-observer-iata-1188-e2e.js Running observer-IATA E2E tests against http://localhost:13581 ✅ Packets table renders an IATA badge in an observer cell ✅ Filter grammar: observer_iata == "<code>" narrows the table ✅ Mobile viewport (375px): observer IATA badge stays visible — not clipped All observer-IATA E2E tests passed. ``` ## Constraints honored - All colors via existing CSS variables (no theming illusions; only `padding` / `margin-left` change inside `@media (max-width: 640px)`). - No JS changes. - Desktop badge display unaffected (selector scoped to narrow viewport). - `config.example.json`: no config field added. - PII preflight: clean. Co-authored-by: OpenClaw Bot <bot@openclaw.local> |
||
|
Kpa-clawbotandOpenClaw Bot
|
43203b09b7 |
fix(#1249): IATA badge missing on fixture + mobile clipping (#1252)
Failing test commit: `bdb4eefb` (added in #1189 R1) — original CI failure: https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995819598 Fixes #1249. ## Root cause Two independent bugs surfaced by the same E2E test: 1. **Fixture join broken.** `scripts/capture-fixture.sh` wrote the text observer hash into `observations.observer_idx`, but the v3 join in `cmd/server` is `observers.rowid = observations.observer_idx`. The join silently nulled out `observer_id` / `observer_iata` for every packet. 2. **Mobile clipping.** `.col-observer` had `data-priority=3` (hides at ≤1024px) and was in the narrow-viewport `defaultHidden` list, so at 375px the cell collapsed to `display:none` and `.badge-iata` had a 0×0 box. ## Changes - `test-fixtures/e2e-fixture.db`: remap `observer_idx` text hash → integer rowid (500/500 rows resolved). - `scripts/capture-fixture.sh`: build an `observer_id → rowid` map before insert; skip rows whose observer isn't in the fixture. Comment explains the trap. - `public/packets.js`: bump `.col-observer` priority `3 → 1` and drop `observer` from narrow-viewport `defaultHidden`. ## Verification All three sub-tests in `test-observer-iata-1188-e2e.js` pass locally against the freshened fixture. `curl /api/packets?limit=5` returns real IATA codes (OAK / MRY / SFO) instead of empty strings. Co-authored-by: OpenClaw Bot <bot@openclaw.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
45872c8371 |
fix(#1250): trim mobile VCR bar h-padding 8px→4px to clear 0.83px LCD clip (#1251)
Red: master CI run https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995768081 already fails on `test-e2e-playwright.js` `#1221 LCD clipped on right (right=375.828125, vw=375)`. No new test commit — the existing E2E assertion is the gate. **Root cause.** PR #1222's mobile rule set `.vcr-bar { padding: 4px 8px }`. The flex row holds three `flex-shrink: 0` children (controls + scope-btns + lcd) and one `flex: 1 1 0` absorber (`.vcr-timeline-container`, `min-width: 40px`). At 375px viewport the absorber hits its floor, so the intrinsic widths of the shrink-frozen children spill 0.83px past the padding box. **Fix.** Drop horizontal padding 8px → 4px inside the `@media (max-width: 640px)` block. That's 8px of new slack — order of magnitude above the 0.83px clip — keeping LCD's `getBoundingClientRect().right ≤ 375`. Desktop layout untouched (rule is mobile-scoped). VCR/feed overlap (#1206/#1213) not reintroduced because `--vcr-bar-height` is JS-measured by the ResizeObserver, not pinned in CSS. Fixes #1250 Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandOpenClaw Bot
|
356f001027 |
perf(#1240): steady-state background recompute for analytics endpoints (#1248)
RED commit: `27630f6a` — adds latency test that fails on master (p99=225ms > 50ms budget) and a stub `StartAnalyticsRecomputers` that returns a no-op so the assertion (not a build error) gates the change. GREEN commit: `20fbbceb` — wires real background recompute infrastructure. Test passes at p99=~1µs. ## What changed Replaces the on-request "compute-then-cache" pattern for the default-shape analytics queries with a steady-state background recompute loop. Reads always hit an `atomic.Value` snapshot in <1µs regardless of compute cost or writer contention. Operator principle: serving slightly stale data quickly beats real-time data slowly. ## Endpoints converted (default 5min interval each) | Endpoint | Cold compute | Recomputer interval | |---|---|---| | `/api/analytics/topology` | ~5s | 5 min | | `/api/analytics/rf` | ~4s | 5 min | | `/api/analytics/distance` | ~3s | 5 min | | `/api/analytics/channels` | ~0.5s | 5 min | | `/api/analytics/hash-collisions` | ~0.5s | 5 min | | `/api/analytics/hash-sizes` | ~22ms | 5 min | All intervals configurable per-endpoint via `analytics.recomputeIntervalSeconds.<name>` in `config.json`; documented in `config.example.json`. Default override via `analytics.defaultIntervalSeconds`. ## Scope: default query only Only the canonical shape `(region="", window=zero)` is precomputed. Region- or window-filtered requests fall back to the legacy TTL cache + on-request compute — keeps recomputer count bounded (6, not 6×N×M). ## Latency Test `TestAnalyticsRecomputerSteadyStateLatency`: 100 concurrent readers + 4 writers churning `s.mu.Lock` on 20k distHops. - Before: p50=188ms p99=225ms (assertion failed) - After: p50=240ns p99=1.1µs (atomic load + map return) ## Shutdown integration `StartAnalyticsRecomputers` returns a stop closure invoked from `main.go`'s SIGTERM handler BEFORE `dbClose()` so any in-flight SQLite compute drains cleanly. `TestAnalyticsRecomputerShutdownNoLeak` confirms all 6 goroutines are reaped (Δ=6 within 2s). ## Safety details - Initial compute is synchronous in `Start()` — first read after startup never sees nil. - `recover()` inside `runOnce` keeps a compute panic from killing the goroutine; previous snapshot remains valid. - `analyticsRecomputerMu` is a sync.RWMutex; recomputer pointers are read-locked in the hot path. The atomic.Value swap inside `runOnce` is lock-free. Fixes #1240. --------- Co-authored-by: OpenClaw Bot <bot@openclaw.local> |
||
|
|
b881a09f02 |
feat(#1188): show observer IATA on packets + filter grammar (#1189)
Red commit: 4ed272761ba94d99912a08e31533ee484ad3f1cd (CI run: https://github.com/Kpa-clawbot/CoreScope/actions/runs/25651898290) Fixes #1188 — observer IATA on packets in three UI surfaces + filter grammar. cross-stack: justified — feature spans API shape (Go), store, filter grammar (JS), three packets UI surfaces. ## Scope shipped - Packets table row: `.badge-iata` pill inline next to observer name - Expanded observation rows: per-observation IATA badge - Detail pane: Observer dd + per-observation list both render the badge - Filter grammar: `observer_iata` field + `iata` alias; `==`/`!=`/`contains`, plus a new `in (a, b, c)` list operator. Both names appear in autocomplete with descriptions. ## TDD red→green pairs 1. `271d72f` filter-grammar tests → `2c182eb` evaluator + suggest entries 2. `4ed2727` backend `observer_iata` API tests → `7856914` SQL join + struct/store wiring 3. `0e09371` display E2E → `7a3f45d` packets.js + style.css badge (E2E swapped for string-contract unit test in `ee414b4` — fixture `observations.observer_idx` stores text pubkeys, blocking the join the badge depends on) ## Backend - `cmd/server/db.go`: SELECT `obs.iata AS observer_iata` in `transmissionBaseSQL`, grouped query, observations-by-transmissions - `cmd/server/store.go`: `ObserverIATA` on `StoreTx`/`StoreObs`, load via all three ingest paths, surface in `txToMap`/`enrichObs`/`groupedTxsToPage` - `cmd/server/types.go`: field added to `TransmissionResp`/`ObservationResp`/`GroupedPacketResp` - Test fixture schemas declare `iata` on observers ## Perf Per #383, `obsIataBadge(packet)` reads `packet.observer_iata` directly (server-joined). Falls back to `observerMap.get(id).iata` only if absent — hot row-render loop avoids per-row Map lookup on fresh data. ## Display rules Missing IATA: nothing inline (Region column still shows `—`). No new hex — `.badge-iata` uses `var(--nav-bg)` / `var(--nav-text)`. E2E assertion added: test-observer-iata-1188.js:51 --------- Co-authored-by: OpenClaw Bot <bot@openclaw.dev> Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
e395c471ed |
fix(#1244): live mobile VCR single row + disable orphan gesture-hint pills on /live (#1246)
Red commit:
|
||
|
Kpa-clawbotandclawbot
|
74685ac82f |
fix(#1243): node detail mobile QR overlays map semi-transparently (#1245)
RED commit `fc9b619a` — CI: https://github.com/Kpa-clawbot/CoreScope/actions Fixes #1243. ## Problem On `#/nodes/<pubkey>` at 375×800, the QR code rendered as a separate ~250px-tall panel below the map. Desktop already overlays the QR semi-transparently via `.node-map-qr-overlay` for the compact view. ## Fix Extend the mobile breakpoint (`@media (max-width: 640px)`) so the full-screen `.node-top-row` mirrors the desktop overlay pattern: - `.node-top-row` → `position: relative`; map wrap expands to 100% - `.node-qr-wrap` → `position: absolute; bottom/right: 8px; z-index: 400` - Semi-transparent background (`rgba(255,255,255,0.85)` light / `0.4` dark) - Caption hidden in overlay (already shown above) Desktop (≥768px) flex layout untouched. ## TDD - RED `fc9b619a` — E2E at 375×800 asserts QR is `position: absolute|fixed`, overlaps map rect, and bg alpha < 1. - GREEN `ded978c0` — CSS adds overlay rule. ## Verification Preflight clean. Desktop layout unaffected — change is scoped inside `@media (max-width: 640px)`. ## Files - `public/style.css` (+29) - `test-e2e-playwright.js` (+57) --------- Co-authored-by: clawbot <clawbot@local> |
||
|
Kpa-clawbotandopenclaw-bot
|
2754251a53 |
perf(#1239): /api/analytics/distance — TTL 15s→60s + drop main RLock around compute (#1241)
## Summary Fixes #1239 — `/api/analytics/distance` 15s cold on staging under heavy ingest. Two independent fixes. First commit on this branch is the RED test for Fix B (`a539882`), demonstrating reader/writer contention against the main store lock. CI: see Actions tab for the run on the test-only commit — it asserts >150µs avg writer cycle and fails at 82367µs pre-fix. GREEN commit (`d3938f1`) brings it to 1µs. ## Fix A — TTL bump 15s → 60s (`5eae1e0`) - `rfCacheTTL` default in `cmd/server/store.go` changed from `15 * time.Second` to `60 * time.Second`. This is the shared TTL for RF / topology / distance / hash-sizes / subpath / channel analytics caches. - Per operator clarification (issue thread): distance analytics IS viewed live during analysis sessions, not background-glanced. 60s smooths the cold-miss churn during heavy ingest without freezing data. - `config.example.json`: documented `cacheTTL.analyticsRF` with new default + caveat. - Existing assertions (`TestCacheTTLDefaults`, `TestHashCollisionsCacheTTL`) updated to the new default. ## Fix B — Drop main RLock around compute (`a539882` red, `d3938f1` green) `computeAnalyticsDistance` previously held `s.mu.RLock()` for the entire iteration: region match-set construction, hop/path filtering, sort, dedup, histogram, category stats, time series. Readers serialized writers (ingest, `buildDistanceIndex`). Refactor: hold the RLock only long enough to snapshot the `distHops`/`distPaths` slice headers AND build the region match-set (which reads `tx.Observations`, mutated under `s.mu.Lock`). For `region=""` (the hot cold-call path) the lock hold is just the header snapshot — microseconds. Everything else runs on the locally-captured slices outside the lock. Safety: `distHops`/`distPaths` are append-only via re-slice in `buildDistanceIndex` / `updateDistanceIndexForTxs` (both under `s.mu.Lock`). If the backing array reallocates after the snapshot, the snapshot still references the prior array (GC-pinned) at the consistent length captured under the lock. Records are value types — no torn writes. ## Test results `cmd/server/distance_lock_contention_test.go` (8 reader goroutines × 20k synthetic distHops × 200 writer Lock/Unlock cycles): - pre-fix avg writer cycle: **82367µs** (16.5s for 200 cycles) - post-fix avg writer cycle: **1µs** (279µs for 200 cycles) - ~82000× reduction in writer contention; reader result shape unchanged Full `go test ./cmd/server/...` green with `-race`. ## Out of scope (per issue) - Same lock pattern in topology / RF / hash / subpath analytics — file separately if needed. - Per-region cache key sharding. - WebSocket-driven cache invalidation. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandcorescope-bot
|
aba20b3eda |
fix(#1234): Live mobile chrome pass 2 — single-row header, hide top-nav, VCR overflow (#1238)
## Summary Live page mobile chrome-reduction pass 2. Three coordinated trims at ≤640px: 1. **`.live-header` → single row, ≤44px.** Drop the MESH LIVE text label and the chart-icon (📊) header toggle. Promote `.live-stats-row` to a direct child of `.live-header` so beacon + pkts + nodes + active + rate + gear all sit on one row. The (now empty) `.live-header-body` collapses to `display:none`. `.live-controls-toggle` shrinks to 36×36 to fit the strip. 2. **Top app navbar hidden on `/live`.** `body:has(.live-page) .top-nav { display:none }` — scoped via `:has()` so other routes are unaffected. The `.live-page` height reclaims the freed 52px. 3. **VCR scope row: >6h collapsed into `More ▾`.** `12h` and `24h` get `.vcr-scope-btn--overflow`; the new `.vcr-scope-more-wrap` dropdown is desktop-hidden, mobile-shown. Dropdown items proxy `.click()` to the underlying scope buttons — single source of truth, existing handler unchanged. ## TDD - **RED** (`b975c828`): `test-issue-1234-live-chrome-pass2-e2e.js` — one E2E asserting all three acceptance items at 375×800 + desktop sanity at 1280×800. Wired into `deploy.yml`. Fails on master (no More button, navbar visible, MESH LIVE label visible). - **GREEN** (`1e529e63`): CSS + JS implementation. Updates `test-live-layout-1178-1179-e2e.js` and `test-issue-1204-live-panel-structure-e2e.js` in-place to match the new single-row contract (chart toggle gone, MESH LIVE label gone on mobile, gear shrunk to 36×36). ## Verification (local) - New E2E: 7/7 ✅ - `test-issue-1178-1179`: 10/10 ✅ - `test-issue-1204`: 10/10 ✅ - `test-issue-1205`: 18/18 ✅ - `test-issue-1206`: 7/7 ✅ - `test-live-mql-leak-1180`: 2/2 ✅ - `#1220` empty-chrome guard (in `test-e2e-playwright.js`): header = 38px collapsed ✅ Desktop (1280×800) layout unchanged — top-nav visible, all 4 VCR scopes inline, header behavior identical. Fixes #1234. --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandOpenClaw Bot
|
4ea1bf8ebc |
fix(#1236): map mobile — sticky panel header + remove right gutter (#1237)
RED:
|
||
|
Kpa-clawbotandbot
|
2e28aa3e04 |
fix(#1229): source-diversity confidence weighting in neighbor-graph tier-1 resolver (#1235)
RED |
||
|
Kpa-clawbotandclawbot
|
b21badbcbd |
fix(#1225): paginate channel messages at SQL level — 30s → <500ms (#1226)
## Summary Fixes #1225 — channel messages endpoint took ~30s on staging. ## Root cause `(*DB).GetChannelMessages` SELECTed every observation row for the channel (one row per observation, not per transmission), JSON-unmarshalled each row into a Go map, dedupe-folded by `(sender, packetHash)`, then sliced the tail in Go for pagination. On staging `#wardriving`: - `transmissions` rows with `channel_hash='#wardriving' AND payload_type=5`: **5,703** - `observations` joined to those: **274,632** (~48× amplification) - `time curl /api/channels/%23wardriving/messages?limit=50`: **30.04s / 31.41s / 31.48s / 35.33s / 34.05s** (5 calls before I killed the loop) `EXPLAIN QUERY PLAN` showed the index `idx_tx_channel_hash` was being used — the cost was entirely in fetching, unmarshalling, and folding the full observation set per request even for `limit=50`. Hypothesis #1 from the issue (full table scan on `messages/decoded`) is rejected; #2 (missing index) is rejected; the actual cause was **pagination in Go instead of SQL** — request cost was O(observations) not O(limit). ## Fix Move pagination into SQL on the `transmissions` table. Because `transmissions.hash` is `UNIQUE` and the original dedup key was `(sender, hash)`, each transmission collapses to exactly one logical message — paginating on transmissions is semantically equivalent to the prior in-Go dedup + tail slice. New shape: 1. `COUNT(*)` on transmissions for total (uses `idx_tx_channel_hash`). 2. `SELECT id FROM transmissions … ORDER BY first_seen DESC LIMIT ? OFFSET ?` to pick the page of newest transmissions. 3. `SELECT … FROM observations WHERE transmission_id IN (…page ids…)` — typically 50 ids → a few hundred observation rows. 4. Reassemble in pageIDs order, preserving the ASC-by-`first_seen` API contract. Region filtering, observation-count-as-`repeats`, and "first observation wins for hops/snr/observer" semantics are preserved (observations are scanned `ORDER BY o.id ASC`). ## Perf measurements **Before** (staging `#wardriving`, limit=50, 5 samples killed mid-loop): 30.04s, 31.41s, 31.48s, 35.33s, 34.05s. **Synthetic regression test** (`TestGetChannelMessagesPerfLargeChannel`): 3000 tx × 50 obs. - Broken impl: ~4.5s (test fails the 500ms budget — the RED commit). - Fixed impl: well under 500ms (test passes). **After (staging)**: will measure post-deploy and post-comment on issue with numbers. Synthetic scaling: staging is ~2× the test's transmission count, fixed-path cost scales with `limit` (50) + `COUNT(*)` (~5k rows on index) — expect <100ms p99. ## TDD - RED: `697c290d` — perf test asserts <500ms on 3k×50 dataset; fails at ~4.5s. - GREEN: `3f1f82d3` — fix; full suite green, perf test passes. ## Hypotheses status | # | Hypothesis | Verdict | |---|---|---| | 1 | Endpoint slow on prod-sized data | **CONFIRMED** (different mechanism — see root cause) | | 2 | Missing channel_hash index | Rejected (`idx_tx_channel_hash` exists & used) | | 3 | Frontend re-render storm | Not investigated (backend was clearly the bottleneck) | | 4 | Decode in request path | Rejected (decode is at ingest time; JSON unmarshal of cached `decoded_json` is the cost, addressed by reducing row count) | | 5 | WS subscription failure | Rejected | | 6 | Staging artifact | Rejected (reproducible) | ## Out of scope - The in-memory `(*PacketStore).GetChannelMessages` path (used when `s.db == nil`) has the same shape but operates on bounded in-memory data; not touched. If we ever fall back to it in production we'll revisit. --------- Co-authored-by: clawbot <bot@corescope> |
||
|
Kpa-clawbotandcorescope-bot
|
7179afcfde |
feat(#1228): reject geo-implausible neighbor-graph edges at build time (#1230)
Fixes #1228 — geo-implausible neighbor-graph edges are rejected at build time. Red commit: `5a6d9660` — failing tests for 4 cases (reject SF↔Berlin, accept local CA, accept no-GPS endpoint, counter increments). Live CI run (latest commit): https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1228 ## Why The disambiguator's tier-1 affinity graph is built blindly from path co-occurrence. On wide-geo MQTT deployments, a single bad hop disambiguation seeds an edge across geographically impossible distances (e.g. Bay Area ↔ Berlin), which then reinforces the same wrong resolution next time. Self-poisoning spiral. ## What changed - `upsertEdge` now consults a per-graph GPS index. When **both** endpoints have known GPS and their haversine distance exceeds the threshold, the edge is dropped and `NeighborGraph.RejectedEdgesGeoFar` (atomic) is incremented. - Either endpoint missing GPS ⇒ accept (no signal to reject), per acceptance criteria. - Threshold is configurable via `neighborGraph.maxEdgeKm` (default **500 km** — well above any plausible terrestrial LoRa hop, including satellite-assisted). 0 ⇒ use default; negative ⇒ disable the filter. Exposed via `Config.NeighborMaxEdgeKm()`. - New `BuildFromStoreWithOptions` carrying the threshold; `BuildFromStore` and `BuildFromStoreWithLog` are kept as thin wrappers. - Stats are surfaced under `GET /api/analytics/neighbor-graph` as `stats.rejected_edges_geo_far`. - All rejection logs PII-truncate pubkeys to 8 hex chars (public repo discipline). - `config.example.json` updated with the new field + comment. ## Follow-up #1229 (per-region scoped affinity graphs) depends on this landing first. --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
30ff45ad34 |
fix(#1220): collapse MESH LIVE mobile header into a single ~50px strip (#1223)
RED commit `c1a8cea` — E2E at 375x800 asserts MESH LIVE header is either ≤60px (collapsed) or ≥60px with a visible body. Fails on master with `height=118, bodyVisible=false, ctrlsVisible=false` — the empty-chrome middle state. CI for red commit: https://github.com/Kpa-clawbot/CoreScope/actions (will populate after push). ## Diagnosis On `(max-width: 768px)`, `#1180` collapses both `.live-header-body` and `.live-controls-body` to `display:none`. But `.live-controls` carries `flex: 0 0 100%` from the wide-viewport rule (introduced for `#1219` so the toggles wrap onto their own row below the title on tablet). On mobile, with the body hidden, that 100% basis still forces the gear button onto a full-width second row inside `#liveHeader`'s flex-wrap, ~60px tall — yielding the `~118-200px` empty panel the bug screenshot shows (the count badge + 📊 toggle on row 1, gear alone on row 2, nothing else). ## Fix — Option C Inside `@media (max-width: 768px)`, when `.live-controls.is-collapsed`: - drop `flex: 0 0 100%` → `flex: 0 0 auto; width: auto` so the gear inlines with the critical strip + 📊 toggle - when the header is also collapsed (`.is-collapsed:has(.live-controls.is-collapsed)`), zero the vertical padding so the strip hugs the 48px tap targets Result: collapsed mobile panel = single ~50px row, three icons inline. Expanded mobile = full toggle list (149px). Desktop unchanged (83px). Why Option C over A/B: a packet-watching mobile user keeps the map dominant and reaches for the gear when they want filters. The compact strip preserves both the WS-down red beacon (always visible) and the pkt count, with one-tap access to expand either body. Does not reintroduce #1204 (counter still attached to header) or #1205 (toggles still children of `#liveHeader`). Fixes #1220 --------- Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> |
||
|
Kpa-clawbotandclawbot
|
70855249c2 |
fix(#1224): channels page mobile UX overhaul (#1227)
## Summary RED test commit: `02652d0042b7cf65d1f9b3e96ce376bbb3064ba6` — CI: https://github.com/Kpa-clawbot/CoreScope/actions Mobile UX overhaul for the Channels page (#1224). At 375x800 the sidebar header was 112px tall (title + button stacked, analytics link + region filter each on their own row) and the channel-name column was clipped to 83px by the inline 📤 Share + ✕ Remove buttons. ## What changed - **Header is now ONE row**: title + region filter + `+ Add` chip + `📊` analytics overflow chip. Capped to ≤56px on mobile. - **`+ Add Channel` → `+ Add` chip** (no longer a full-width hero). Verified <65% of sidebar width. - **Analytics link** is an icon-only chip inside the header (was a full-row link below). - **Region filter** is inline inside the header (was its own row). - **Channel rows**: `.ch-item-name` takes `flex:1`, share button is icon-only (📤), remove button shrunk to 32px touch target. Name >150px on the first row. - **Empty state** is `max-height:30vh; padding:12px` on mobile — no longer dominates the viewport. ## Design decisions - Chose **inline chips** over an overflow `⋮` menu: header-level controls are few enough (4) that stacking pills + filter dropdown fits comfortably in 375px. Avoids the cost/complexity of a popover and matches the page's existing pill vocabulary (region filter). - Per-row share/remove kept inline but icon-only (`font-size:0` + `::before`) — preserves single-tap access without consuming the row. - Touch targets stay ≥32px (action chips) / 44px (other tappables); WCAG 2.5.5 spirit retained on the dominant interactive paths. - **Desktop layout (≥768px) is unchanged** — verified by a desktop guard in the E2E (`.ch-layout` flex-direction stays `row` at 1024px). ## Tests - `test-issue-1224-channels-mobile-ux-e2e.js` — 5 assertions at 375x800 + 1 desktop guard at 1024x800. Wired into CI. - Existing channel suites still pass: `test-channel-fluid-e2e.js` (11/11), `test-channel-issue-1087-e2e.js` (3/3), `test-channel-issue-1111-e2e.js` (2/2), `test-channel-modal-ux.js` (33/33), `test-channel-ux-followup.js` (29/29), `test-channel-sidebar-layout.js` + `test-channel-fluid-layout.js` (14/14). Fixes #1224 --------- Co-authored-by: clawbot <clawbot@users.noreply.github.com> |
||
|
Kpa-clawbotandopenclaw-bot
|
24f277e5c6 |
fix(#1221): VCR LED clock in-row with controls and unclipped on mobile (#1222)
Red commit:
|
||
|
|
ab34d9fb65 |
fix(#1206): keep VCR bar from occluding the live packet feed (#1213)
Red commit: `bcfc74de` (CI: https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1206) Fixes #1206. ## Problem On Live Map the VCR (timeline/playback) bar overlays the bottom of the viewport. Bottom-pinned overlays — the live packet feed, the legend, any corner panel — used hard-coded `bottom: 58–88px` offsets that are smaller than the real bar height (two-row mobile layout + `env(safe-area-inset-bottom)` push it to ~80px and beyond). The last N packet-feed rows slid under the bar and became unreadable / unclickable. ## Fix Publish the bar's measured height as a CSS variable on the live page and bind every bottom-anchored overlay to it. - `public/live.js` — new `initVCRHeightTracker()` runs after init; uses `ResizeObserver` + `resize` / `visualViewport.resize` to keep `--vcr-bar-height` on `.live-page` in sync with `#vcrBar`. - `public/live.css` — `.live-feed`, `.feed-show-btn`, and the `.live-overlay[data-position="bl"|"br"]` corner slots now use `bottom: calc(var(--vcr-bar-height, 58px) + 10px)`. The feed's `max-height` is also capped against `100dvh - top - vcr - margin` so its scroll container can never extend past the bar. - Stale per-breakpoint overrides (the `@supports(env(safe-area-inset))` hard-coded `78px + safe-area` for feed/legend) are removed in favor of the single tracked variable. ## TDD - Red commit `bcfc74de` adds `test-issue-1206-vcr-overlap-e2e.js`: asserts `#liveFeed.getBoundingClientRect().bottom <= #vcrBar.top` (and same for the last row) at desktop 1280x800 and mid 720x800. Verified locally that reverting the green commit makes the feed-bottom assertions fail (feed bottom 742px > VCR top 721px) — see PR body for exact numbers from the local run. - Green commit `1ad17e7f` makes all 5 assertions pass. ## Browser verified Local Go server with `test-fixtures/e2e-fixture.db`, headless Chromium via the new E2E test — all 5 assertions green. ## E2E assertion added `test-issue-1206-vcr-overlap-e2e.js:84` (bottom-row vs VCR-top) plus container check at `:74`. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: clawbot <bot@corescope.local> |
||
|
|
a1f9dca951 |
fix(live #1205): re-anchor settings toggles inside MESH LIVE panel (#1219)
Red commit: f80ce5248a81fb8907081765732ebe0814f4a21a (CI URL appears in the Checks tab once the workflow starts). Supersedes closed PR #1209 with the correct approach (toggles in MESH LIVE panel, not legend). Fixes #1205. ## Problem The Live Map settings toggle row (Heat / Ghosts / Realistic / Color by hash / Matrix / Rain / Audio / Favorites / node filter / region filter — `#liveControls`) rendered as a free-floating sibling `.live-overlay` pinned `position: fixed` at bottom-right with `bottom: calc(78px + var(--bottom-nav-height) + safe-area)`. On many viewports it visually orphaned across the middle of the map, anchored to no panel. ## Regression cause PR **#1180** (commit `127a1927` — "compact header, pin controls bottom-right, narrow toggles") extracted `.live-toggles` from inside `.live-header` (the MESH LIVE panel) into a brand-new sibling `.live-overlay.live-controls` cluster. Before #1180 the toggles lived as a direct child of `.live-header`. ## Fix Restore the pre-#1180 structural pattern: `#liveControls` is re-parented as a child of `#liveHeader`, breaking onto its own row via `flex: 0 0 100%`. No more `position: fixed` overlay, no more free-floating cluster — the toggles share the MESH LIVE panel's chrome (background, blur, border, padding). - `public/live.js`: re-parent the `#liveControls` block inside `#liveHeader`, drop the `.live-overlay` class. - `public/live.css`: - `.live-controls`: `position: static`, transparent (header supplies chrome), `flex: 0 0 100%`. - `.live-header`: `flex-wrap: wrap`, `row-gap: 6px`, `max-width: calc(100vw - 24px)`; drop the `max-height: 40px` cap. Why this beats PR #1209: that PR parked toggles inside `#liveLegend`, inverting the *data → key → controls* hierarchy and pushing the legend to 60vh on mobile. Anchoring back to the MESH LIVE panel keeps controls with the panel that already labels the live surface and inherits its corner / drag affordances. ## Tests - **Red** (`test-issue-1205-live-controls-anchor-e2e.js`): asserts `#liveHeader.contains(#liveControls)` AND not contained in `#liveLegend`, parent is not `<body>` / `.live-page` directly, and the controls rect stays within the viewport. Runs at **1440×900, 640×900, 320×800**. Fails on master. - **Updated** `test-live-layout-1178-1179-e2e.js`: - (a) `.live-header-critical` height ≤ 40px (the critical strip stays compact; header itself now wraps). - (b) `.live-controls` `position: static` AND descendant of `#liveHeader` (new contract replacing the retired "fixed/right ≤24px/bottom>0"). - Wired in `.github/workflows/deploy.yml` next to the other live-layout E2Es. ## Acceptance criteria - [x] Settings toggle row renders inside the MESH LIVE panel (`#liveHeader`) - [x] Not parked in `#liveLegend` (rejected by #1209 review) - [x] Tested at desktop + tablet + narrow phone viewport widths - [x] E2E DOM assertion: parent is the MESH LIVE panel, not body / `.live-page` / `#liveLegend` --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: clawbot <clawbot@users.noreply.github.com> |
||
|
|
170f0ac66d |
fix(#1212): MQTT per-attempt logging + stall watchdog — prevent silent reconnect-loop death (#1216)
RED commit: `1cd25f7b` — CI (failing on assertion): https://github.com/Kpa-clawbot/CoreScope/actions?query=sha%3A1cd25f7b1bdd0091f689dd64ce1bfec6d031191f Fixes #1212 ## Root cause NOT that `AutoReconnect` was off — it was set; `MaxReconnectInterval=30s` was set (PR #949); a `SetReconnectingHandler` was wired. The defect was an **observability gap**: `SetReconnectingHandler` fires only INSIDE paho's reconnect goroutine. If that goroutine never iterates (status race after the recovered handler panic at 21:07:13, or an internal abort), operators see ONLY the `disconnected: pingresp not received` line and then total silence. They cannot distinguish "paho is patiently retrying" from "paho gave up and the goroutine is gone." That ambiguity is what turned a 30s blip into 6h of downtime. ## Changes ### `cmd/ingestor/main.go` — `SetConnectionAttemptHandler` Fires on every TCP/TLS dial — the initial `Connect()` AND every reconnect — independent of paho's internal reconnect-loop state. Logs: ``` MQTT [staging] connection attempt #1 to tcp://broker:1883 MQTT [staging] connection attempt #2 to tcp://broker:1883 ``` Per-source attempt counter via `atomic.AddInt64`. ### `cmd/ingestor/mqtt_watchdog.go` (new) — per-source stall watchdog Satisfies the watchdog acceptance criterion. Even when paho reports `connected`, if no MQTT messages have flowed for >5m, log a WARN line every 60s: ``` MQTT [staging] WATCHDOG: client reports connected to tcp://broker:1883 but no messages received for 7m30s (threshold 5m) — possible half-open socket or upstream stall ``` Catches half-open TCP and broker-accepted-but-not-forwarding scenarios that look "connected" to paho. Hot-path cost: one `atomic.StoreInt64` per inbound message. Watchdog scans the registry once a minute. ### Tests (`cmd/ingestor/mqtt_reconnect_test.go`, new) - `TestBuildMQTTOpts_InstrumentsConnectionAttempt` — asserts `OnConnectAttempt` is wired in `buildMQTTOpts`. - `TestMQTTStallWatchdog_FiresOnSilentSource` — connected + 10m silent + 5m threshold → stall flagged. - `TestMQTTStallWatchdog_QuietWhenRecent` — recent message → no stall. - `TestMQTTStallWatchdog_QuietWhenDisconnected` — disconnected → no stall (paho's reconnect logging covers it). ## TDD - RED `1cd25f7b` — 2 assertion failures (compile OK, stub returns no-stall, `OnConnectAttempt` nil). - GREEN `2527be6f` — implementation; all ingestor tests pass. ## Out of scope - Slice-bounds decode panic (#1211, separate PR). - A full in-process MQTT broker integration test would require a new dep (mochi-mqtt) — the observability and watchdog behaviors are independently verifiable by the unit tests above, and the reconnect path itself is paho's responsibility (we already test it's configured via `mqtt_opts_test.go`). --------- Co-authored-by: bot <bot@example.com> Co-authored-by: OpenClaw Bot <bot@openclaw.local> Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> |
||
|
|
eba9e89a72 |
fix(#1203): path-inspector — singleflight + stale-while-revalidate (#1208)
Red commit:
|
||
|
|
11d2026bb1 |
feat(startup): hot startup — load hotStartupHours synchronously, fill retentionHours in background (#1187)
Closes #1183 ## Summary - Adds `packetStore.hotStartupHours` config key (float64, default 0 = disabled). When set, `Load()` loads only that many hours of data synchronously, reducing startup time on large DBs. Background goroutine fills the remaining `retentionHours` window in daily chunks after startup completes. - A background goroutine (`loadBackgroundChunks`) fills the remaining `retentionHours` window in daily chunks after startup completes. Analytics indexes are rebuilt once at the end. - `QueryPackets` and `QueryGroupedPackets` check `oldestLoaded` and fall back to `db.QueryPackets()` for any query whose `Since`/`Until` predates the in-memory window — covering days 8–30 permanently (beyond `retentionHours`) and the background-fill gap during startup. - `/api/perf` gains `hotStartupHours`, `backgroundLoadComplete`, and `backgroundLoadProgress` fields inside `packetStore` so operators can monitor the fill. ### Drive-by fixes - E2E: added `gotoPackets` navigation helper used across packet-related tests - E2E: rewrote stripe assertion to check per-row stripe parity rather than a fragile computed-style comparison - E2E: theme test updated to use `#/home` as the initial route (was `#/`) - `db.go`: removed the RFC3339→unix-timestamp subquery path in `buildTransmissionWhere`; `t.first_seen` is now always compared directly as a string for both RFC3339 and non-RFC3339 inputs ## Configuration ```json "packetStore": { "retentionHours": 168, "hotStartupHours": 24 } ``` `hotStartupHours: 0` (default) preserves existing behavior exactly. Recommended for large DBs to reduce startup time; set to 0 to disable (loads full retentionHours at startup, legacy behavior). ## Test plan - [x] `TestHotStartupConfig_Clamp` — clamping when `hotStartupHours > retentionHours` - [x] `TestHotStartupConfig_ZeroIsDisabled` — zero leaves feature disabled - [x] `TestHotStartup_LoadsOnlyHotWindow` — only hot-window packets in memory after `Load()` - [x] `TestHotStartup_DisabledWhenZero` — all retention packets loaded when disabled - [x] `TestHotStartup_loadChunk_AddsOlderData` — chunk merges correctly, ASC order maintained - [x] `TestHotStartup_BackgroundFillsToRetention` — background goroutine fills to `retentionHours` - [x] `TestHotStartup_ChunkErrorRecovery` — chunk SQL failure logged and skipped, loop terminates - [x] `TestHotStartup_SQLFallback_TriggeredForOldDate` — query before `oldestLoaded` routes to SQL - [x] `TestHotStartup_SQLFallback_NotTriggeredForRecentDate` — recent query stays in-memory - [x] `TestHotStartup_PerfStats` — new fields present in `GetPerfStoreStats()` (backs the perf endpoint) - [x] `TestHotStartup_PerfStoreHTTP` — HTTP-level: GET /api/perf returns `hotStartupHours`, `backgroundLoadComplete`, `backgroundLoadProgress` in `packetStore` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: CoreScope Bot <bot@corescope.local> |
||
|
Kpa-clawbotandcorescope-bot
|
3255395bd0 |
fix(#1204): MESH LIVE panel — header inherited column flex from .live-overlay (#1215)
Red commit:
|
||
|
|
85e97d2f37 |
fix(#1211): bounds-check path length to prevent slice [218:15] panic in MQTT decode (#1214)
**RED commit:** `65d9f57b` (CI run will appear at https://github.com/Kpa-clawbot/CoreScope/actions after PR opens) Fixes #1211 ## Root cause `decodePath()` returns `bytesConsumed = hash_size * hash_count` where both come straight from the wire-supplied `pathByte` (upper 2 bits → `hash_size`, lower 6 bits → `hash_count`). Max claimable: 4 × 63 = 252 bytes. A malformed packet on the wire claimed `pathByte=0xF6` (hash_size=4, hash_count=54 → 216 path bytes) inside a 15-byte buffer. The inner hop-extraction loop in `decodePath` did break early on overflow — but `bytesConsumed` was still returned at face value (216). `DecodePacket` then did `offset += 216` (offset=218) and `payloadBuf := buf[offset:]` panicked with the prod-observed signature: ``` runtime error: slice bounds out of range [218:15] ``` The handler-level `defer/recover` at `cmd/ingestor/main.go:258-263` caught it, but the message was silently dropped with no usable diagnostic. ## Fix Add a `if offset > len(buf)` guard at BOTH decoder sites (same pattern, same panic potential): - `cmd/ingestor/decoder.go` — DecodePacket after decodePath - `cmd/server/decoder.go` — DecodePacket after decodePath Return a descriptive error citing the claimed length and pathByte hex so operators can reproduce. Also: `cmd/ingestor/main.go` decode-error log now includes `topic`, `observer`, and `rawHexLen` so future malformed packets are reproducible without needing to attach a debugger. ## Tests (TDD red → green) Both packages got two new tests: - **`TestDecodePacketBoundsFromWire_Issue1211`** — feeds the exact wire shape from the prod log (`pathByte=0xF6` inside a 15-byte buf). Asserts `DecodePacket` does NOT panic and returns an error. - **`TestDecodePacketFuzzTruncated_Issue1211`** — sweeps every `(header, pathByte)` combination with tails 0..19 bytes (≈1.3M inputs). Asserts zero panics. ### Red commit proof On commit `65d9f57b` (RED), both tests fail with the panic: ``` === RUN TestDecodePacketBoundsFromWire_Issue1211 decoder_test.go:1996: DecodePacket panicked on malformed input: runtime error: slice bounds out of range [218:15] --- FAIL: TestDecodePacketBoundsFromWire_Issue1211 (0.00s) === RUN TestDecodePacketFuzzTruncated_Issue1211 decoder_test.go:2010: DecodePacket panicked during fuzz: runtime error: slice bounds out of range [3:2] --- FAIL: TestDecodePacketFuzzTruncated_Issue1211 (0.01s) ``` On commit `7a6ae52c` (GREEN), full suites pass: - `cmd/ingestor`: `ok 53.988s` - `cmd/server`: `ok 29.456s` ## Acceptance criteria - [x] Identify the slice op producing `[218:15]` — `payloadBuf := buf[offset:]` in `DecodePacket` (decoder.go), where `offset` had been advanced by an unchecked `bytesConsumed` from `decodePath()`. - [x] Bounds check added at the identified site(s) — both ingestor and server decoders. - [x] Test with crafted payload (length-field > remaining buffer) — `TestDecodePacketBoundsFromWire_Issue1211`. - [x] Log topic, observer ID, payload byte length on drop — updated `MQTT [%s] decode error` log line. - [x] Existing tests stay green — confirmed both packages. ## Out of scope Reconnect-after-disconnect (#1212) — handled by a separate subagent. This PR touches NO reconnect logic. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: corescope-bot <bot@corescope> |
||
|
|
4925770aa4 |
fix(#1207): empty-state placeholder for Live Feed panel (no more orphan chrome) (#1210)
Red commit: `6c28227884a1e79e277653465028365dc0863171` — CI: https://github.com/Kpa-clawbot/CoreScope/actions?query=branch%3Afix%2Fissue-1207 Fixes #1207 ## Diagnosis The Live Map page renders `#liveFeed` (bottom-left panel) with two header buttons — `◫` (panel-corner-btn) and `✕` (feed-hide-btn) — but its `.panel-content` body has zero children on first paint, before any packets have been ingested via WebSocket. The user-reported "X + book icons, no content" is exactly these two header buttons sitting on an empty body. **Verdict:** intended panel, missing content due to a data race — the chrome mounts in HTML before the WS pushes its first packet. Not orphaned, not a leftover from #1186. ## Fix - Always render a persistent `.live-feed-empty` placeholder ("Waiting for packets…") inside `#liveFeed .panel-content`. - CSS hides it via `.live-feed .panel-content:has(.live-feed-item) .live-feed-empty { display: none; }` when real feed items exist. - `rebuildFeedList` re-adds the placeholder defensively after a wipe; eviction loop counts `.live-feed-item` only so the placeholder is never trimmed out. All colors via CSS variables (`var(--text-muted)`). ## Test (RED → GREEN) - **RED** `6c28227884a1e79e277653465028365dc0863171` — `test-e2e-playwright.js` adds a new test ("#1207 Live Feed panel never renders as empty chrome") that wipes `.live-feed-item` children to simulate the empty state and asserts the panel body has visible text or children. Fails on master. - **GREEN** `a5af80960ac42759ec83fd5ca5a72e81856228d4` — adds the placeholder; test now passes. ## Acceptance criteria - [x] No empty panel chrome visible on Live Map page - [x] Panel renders "Waiting for packets…" while feed is empty - [x] CSS auto-hides placeholder when packets arrive - [x] E2E assertion in `test-e2e-playwright.js` enforces non-empty `.panel-content` on `#liveFeed` ## Files - `public/live.js` — HTML markup + `rebuildFeedList` re-add + eviction-loop guard - `public/live.css` — `.live-feed-empty` style + `:has()` hide rule - `test-e2e-playwright.js` — regression test --------- Co-authored-by: clawbot <clawbot@kpabap.local> Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
dbb013a6bf |
test(#1201): regression coverage for hop disambiguator tier-1 + end-to-end top-hops fixture (#1202)
Mutation test confirmed: reverting cmd/server/store.go:2975 (`setContext(buildHopContextPubkeys(tx, pm))` → `setContext(nil)`) in `buildDistanceIndex` produces failing assertion in `TestTopHopsRespectsContextAcrossAllCallSites`: top-hops ranking flips to `72dddd→8acccc@13.0km` (Berlin↔Berlin misresolution), CA↔CA pair absent. After reverting the mutation, the test passes again. Fixes #1201 ## Summary Pure test addition. No production code changed. Adds regression coverage for the hop disambiguator's tier-1 (neighbor affinity) path and an end-to-end fixture that catches revert-to-nil-context regressions across all 9 call sites of `pm.resolveWithContext`. ## Sub-tasks (all 4 landed) 1. **Tier-1 explicit** — `hop_disambig_tier1_test.go`: - `Tier1_StrongAffinityPicksX` (strong-X edge wins) - `Tier1_StrongAffinityPicksY` (reverse weights — proves score is read) - `Tier1_AmbiguousEdgeSkipsToTier2` (`Ambiguous=true` → skip) 2. **Tier ordering** — `Tier1_BeatsTier2WhenBothSignal` (tier 1 wins when both signal) 3. **Tier-1 fallback** — - `Tier1_EmptyGraphFallsThrough` (graph has no edges for context) - `Tier1_NilGraphFallsThrough` (graph is nil) - `Tier1_ScoresTooCloseFallsThrough` (best < `affinityConfidenceRatio` × runner-up) 4. **End-to-end fixture** — `hop_disambig_e2e_test.go`: - 9 nodes with intentional prefix collisions across SLO/LA/NYC/Berlin (prefix `72`) and SF/CA/Berlin (prefix `8a`); Berlin candidates have `obsCount=200` so they'd win tier-3 absent context. - 50 transmissions path `["72","8a"]`, sender + observer in CA. - Affinity graph seeded with strong `sender↔72aa` and `sender↔8aaa` edges. - Asserts: CA↔CA hop present, no Berlin pubkeys in `distHops`, max distance < 300 km cap. ## TDD exemption Net-new regression-sentinel tests for behavior already correct on master post-#1198. Each test passed on first run (no production bug surfaced). The mutation test on sub-task 4 is the gating proof: forcing `setContext(nil)` at `store.go:2975` makes the test fail with the exact misresolution class the issue describes (Berlin↔Berlin leaks into top-hops). ## Acceptance criteria - [x] Tier-1 affinity test added with 3 cases - [x] Tier-ordering test added - [x] Tier-1 fallback tests added (nil / empty / scores-too-close) - [x] End-to-end fixture added with multi-candidate-prefix nodes - [x] End-to-end fixture fails if any call site reverts to `nil` context (mutation-verified) - [x] Test files live in `cmd/server/` alongside `prefix_map_role_test.go` --------- Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
2beeb2b324 |
fix(#1199): 6 deferred quality items from PR #1198 r2 review (#1200)
Red commit: 75563ce (CI run: pending — pushed at branch open) Follows up PR #1198 round-2 adversarial review (issue #1199). Six robustness / perf-hot-path / maintenance items, one commit per logical change. Stacked on top of `fix/issue-1197` (PR #1198) — base must move to `master` after #1198 merges. | # | Item | Commit(s) | Discipline | |---|---|---|---| | 1 | Brittle static-grep regex → go/parser AST walk in `resolve_context_callsites_test.go` | 33d80b6 (RED) → 450236d (GREEN) | red→green | | 2 | `computeAnalyticsTopology` double-pass filter → materialize `filteredTxs` once | 00005f6 | refactor | | 3 | `BenchmarkBuildAggregateHopContextPubkeys` baseline + tiny smoke test | b520048 | net-new bench/test | | 4 | `hopResolverPerTx` CONCURRENCY doc — single-goroutine invariant | 155ff07 | doc-only | | 5 | `schemaDegradationLogged` package-level `sync.Map` → PacketStore field | 75563ce (RED) → 7dbf193 (GREEN) | red→green | | 6 | `buildHopContextPubkeys` `out` slice cap hint (`make([]string, 0, 16)`) | 2040962 | refactor | Items 2 & 6 are pure refactors — no test files modified for items 2 & 6 (per AGENTS.md exemption rule). Existing tests stay green and unaltered. Item 4 is doc-only (CONCURRENCY: comment); no behavior change. Item 3 adds a bench + a smoke assertion for the aggregate helper that previously had no coverage. Local arm64 baseline: ~72ms/op, 130k allocs at 5k txs. Items 1 & 5 follow red→green: 33d80b6 demonstrates the regex blindspot via a synthetic AST-detectable input the regex misses; 75563ce demonstrates per-store log dedup leaks across instances. Both flips visible in branch history. Full `go test ./cmd/server/...` runs clean post-amend. Fixes #1199 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
353c5264ad |
fix(#1197): plumb hop-context + observation-count tiebreak to disambiguator (#1198)
Red commit:
|
||
|
|
03b5d3fe28 |
fix(#1065): first-visit gesture discoverability hints (#1186)
Red commit:
|