## Why master is red
After PRs #1592 (route-window subpath regression test) and #1595
(background/chunked index build with 503 readiness gate) were merged
together, two tests in `cmd/server/subpaths_window_test.go` started
failing on master:
```
--- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel
subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[])
--- FAIL: TestSubpathsHandlerHonorsTimeWindow
subpaths_window_test.go:116: GET /api/analytics/subpaths?...: status=503 body={"error":"index loading","retryAfter":5}
```
Both branches passed in isolation; the conflict only manifested
post-merge. Reason:
- **#1592** added tests that call `store.Load()` then immediately query
`GetAnalyticsSubpathsWithWindow` / hit `/api/analytics/subpaths`.
- **#1595** moved the subpath + path-hop index builds off the critical
path of `Load()` into background goroutines, and hard-gated the
analytics handlers behind `SubpathIndexReady()` (returning 503 +
`Retry-After: 5` until the build completes).
So after `Load()` returns, `s.spIndex` is still empty for a short window
and the handler returns 503. The store-level test sees `totalPaths=0`;
the handler test sees the 503.
## Fix (test-only)
Add `store.WaitIndexesReady(5 * time.Second)` between `Load()` and the
assertions in both tests. This matches the established pattern already
used by `routes_test.go` and `repeater_enrich_recomputer_1008_test.go`.
The 503 readiness gate from #1595 is intentional production behavior and
is **not** touched. No production code is modified.
## Repro
Before:
```
$ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=1
--- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s)
subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[])
--- FAIL: TestSubpathsHandlerHonorsTimeWindow (0.02s)
subpaths_window_test.go:116: GET /api/analytics/subpaths?minLen=2&maxLen=8: status=503 body={"error":"index loading","retryAfter":5}
FAIL
```
After:
```
$ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=3
--- PASS: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s)
--- PASS: TestSubpathsHandlerHonorsTimeWindow (0.02s)
... (x3) ...
PASS
ok github.com/corescope/server 0.097s
$ go test ./cmd/server/ -count=1 -timeout 300s
ok github.com/corescope/server 46.292s
```
## Files changed
- `cmd/server/subpaths_window_test.go` (+11 lines, test-only)
## Notes
- TDD exemption: this is a test-fix PR for a merge-conflict-induced
failure. The "failing test" already exists on master; this PR makes it
pass correctly by waiting on the readiness gate the test was previously
unaware of.
- Unblocks staging deploys.
Co-authored-by: openclaw-bot <bot@openclaw>
## What
Per-component SQLite writer-lock instrumentation so the next
neighbor-builder-style write-lock starvation (root cause of #1339,
invisible to operators for ~3 days) is detectable from `/api/perf`.
Adds `Store.WriterExec` / `Store.WriterTx` wrappers that gate every
wrapped call on a package-level `writerMu` so the wait the SQLite driver
hides becomes Go-visible, and record `wait_ms` + `hold_ms` +
`contention_total` (wait_ms > 100ms) under a component tag.
Per-component p50/p95/p99 + max are published to
`/api/perf/write-sources` under `.writer_perf` via the existing ingestor
stats-file path. Slow-writer log line (`[db-slow-writer] component=X
duration=Yms query=<200ch>`) fires on `hold_ms > 500ms` (threshold
overridable via `CORESCOPE_DB_SLOW_WRITER_MS` env var).
## Tagged call sites
| Component | Location |
|-----------|----------|
| `mqtt_handler` | `InsertTransmission` (db.go) |
| `neighbor_builder` | `buildAndPersistNeighborEdges`
(neighbor_builder.go) |
| `prune_packets` | `PruneOldPackets` (maintenance.go) |
| `prune_observers` | `RemoveStaleObservers` + orphan-metrics cleanup
(db.go) |
| `prune_metrics` | `PruneOldMetrics` (db.go) |
| `vacuum` | `RunIncrementalVacuum` + `CheckAutoVacuum`'s full VACUUM
(db.go) |
## TDD red→green
- **Red commit** `68de585b` — `cmd/ingestor/db_writer_perf_test.go` +
`Store.Writer*` stubs at end of `db.go`. Test synthetically blocks the
writer for 60s tagged `neighbor_builder`, then asserts
`mqtt_handler.wait_ms.p99 > 50000ms` on concurrent inserts. Fails on the
assertion (p99 = 0.0ms) with the stub — not a build error.
- **Green commit** `6a9be174` — replaces stubs with real
wait/hold/contention aggregator + wires every writer call site. Same
test passes:
```
2026/06/05 04:36:47 [db-slow-writer] component=neighbor_builder duration=60059.0ms query=COMMIT
--- PASS: TestWriterStarvationVisibleInPerf (60.40s)
PASS
ok github.com/corescope/ingestor 60.408s
```
## Scope discipline
- **API**: no public `Store`/`DB` signature change. Only additive
exports.
- **Server**: extends existing `/api/perf/write-sources` JSON with
`.writer_perf` — does **not** add a new route, does **not** replace
`handlePerf`. Empty `.writer_perf` map when paired with an older
ingestor.
- **Read/write invariant** (#1283) preserved: all instrumentation lives
on the ingestor's writer connection.
- **Files touched** (6 total): `cmd/ingestor/db.go`,
`cmd/ingestor/db_writer_perf_test.go`, `cmd/ingestor/maintenance.go`,
`cmd/ingestor/neighbor_builder.go`, `cmd/ingestor/stats_file.go`,
`cmd/server/perf_io.go`, `config.example.json`.
## Deferred (acceptance items NOT in this PR)
- **`mbcap_persist` component tag** — `RunMultibyteCapPersist`'s tx is
intentionally NOT wrapped in this PR to stay within the implementation
brief's 3-files-outside-whitelist budget. One-file follow-up to
instrument.
- **CI smoke test** asserting "neighbor-builder hold_ms < 1000ms on
100k-obs fixture" — deferred to a separate PR per the brief; this PR is
scoped to instrumentation only.
## Preflight overrides
PREFLIGHT-MIGRATION-SCALE: <30s N=runtime — the async-migration gate
flagged five `instrumentedExec` / wrapped-`tx.Exec` lines on `DELETE
FROM observer_metrics`, `UPDATE observers`, `DELETE FROM
observer_metrics`, `DELETE FROM observations`, `DELETE FROM
transmissions`. These are **not** schema migrations — they are the
existing runtime prune / retention queries that already ran sync against
`s.db.Exec` / `tx.Exec` on every retention cycle on master. This PR only
swapped the surface call (sync → sync, via the wrapper) to record
wait/hold timing; no new sync schema work was introduced. Behavior on
production data is identical to master.
Also: red commit's synthetic `UPDATE nodes SET name = name WHERE 0` is a
test-only stub designed to acquire the writer without mutating any row
(the `WHERE 0` is a no-op predicate).
Fixes#1340
---------
Co-authored-by: corescope-bot <bot@corescope.local>
Red commit: 929da3c6dcc1b619c27478291125d1c91323db8f — CI:
https://github.com/Kpa-clawbot/CoreScope/commit/929da3c6dcc1b619c27478291125d1c91323db8f/checksFixes#1010.
## What
Adds `GOMEMLIMIT` support to both `cmd/server` and `cmd/ingestor` per
the locked triage scope on #1010.
Precedence (env wins):
1. `GOMEMLIMIT` env var
2. `runtime.maxMemoryMB` config field (new)
3. Server only: implicit `packetStore.maxMemoryMB * 1.5` (existing #836
behavior, unchanged when `runtime.maxMemoryMB` is absent)
4. Otherwise unset — default Go behavior preserved (backwards
compatible)
Each startup logs a `[memlimit]` line echoing the effective
source/limit, or an "unset → default" note when neither is set.
## Changes
- `cmd/ingestor/memlimit.go` — new, `applyMemoryLimit(runtimeMaxMB,
envSet)`.
- `cmd/ingestor/memlimit_test.go` — new, env/config/none/precedence
assertions.
- `cmd/ingestor/config.go` — new `RuntimeConfig{MaxMemoryMB int}` field.
- `cmd/ingestor/main.go` — wires `applyMemoryLimit` into startup right
after `LoadConfig`.
- `cmd/server/config.go` — new `RuntimeConfig` + `cfg.Runtime` field.
- `cmd/server/main.go` — adds explicit `runtime.maxMemoryMB` precedence
over packetStore-derived; existing `warnIfMemlimitUnderprovisioned`
(#1264) unchanged.
- `config.example.json` — new `runtime` block with
`_comment_runtime_maxMemoryMB` per the Config Documentation Rule.
- `README.md` — sizing-table row with ≥1.5× working set floor +
death-spiral warning.
## TDD
- Red: `929da3c6` — ingestor `applyMemoryLimit` stub returns
`(0,"none")`; four tests fail on assertions (`expected source=env, got
"none"`, etc.) — no compile errors.
- Green: `953ec9d8` — implements ingestor `applyMemoryLimit`, wires
startup, threads `runtime.maxMemoryMB` through server too.
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
→ clean (all gates pass, all warnings pass).
## Out of scope
- `pprof`-verified GC-trigger acceptance criterion from the original
issue — requires production tracing; the triage scope is the
operator-tunable plumbing.
- Container auto-detection of cgroup memory limit (already covered by
#1264's `warnIfMemlimitUnderprovisioned`).
---------
Co-authored-by: corescope-bot <bot@corescope>
## Summary
Closes#1608.
The ingestor's MQTT connect/subscribe loop ran **last** in `main()`,
after the synchronous startup-maintenance block. Because all writes
share a single SQLite writer (#1283), that maintenance — and the connect
loop after it — serialize behind any long-running async migration. The
subscription therefore came up minutes late (observed ~4.5 min after the
v3.8.3 `obs_observer_ts_idx_v1` index build over ~4.9M rows), and QoS-0
packets published in that window were dropped.
This decouples **receipt** from **write**:
- New `IngestBuffer` — a bounded FIFO drained by a **single** gated
consumer goroutine.
- The MQTT subscription is brought up first; its publish handler stamps
source liveness at receipt and enqueues a `handleMessage` closure.
- Startup maintenance runs, then `WaitForAsyncMigrations()`, then
`IngestBuffer.Ready()` opens the gate and the backlog drains.
A single consumer preserves the single-writer invariant (#1283);
buffering replays the original messages, so it introduces **no
duplicates** (unlike a QoS-1 broker queue). Broker-agnostic — helps
direct-connect and bridged operators alike.
## Changes
- `cmd/ingestor/ingest_buffer.go` — `IngestBuffer`
(`Submit`/`Start`/`Ready`/`Dropped`/`Pending`); non-blocking submit with
drop-on-full counter; single consumer.
- `cmd/ingestor/config.go` — `ingestBufferSize` knob (default 50000).
- `cmd/ingestor/main.go` — reorder boot: connect/subscribe **before**
startup maintenance; stamp liveness at receipt; `Ready()` after
maintenance + `WaitForAsyncMigrations()`; periodic stats log buffer
`pending`/`dropped`.
## Test plan
- [x] `go test ./...` in `cmd/ingestor` — `IngestBuffer` suite covers
gating-until-ready, FIFO order, drop-on-full, serial execution
(single-writer), and concurrent-submit.
- [ ] `go test -race` in CI (concurrency on `IngestBuffer`).
- [ ] Manual: restart with a pending heavy migration → `subscribed to
meshcore/#` appears within seconds; `[ingest-buffer] write path ready`
after the migration; packets received during the window are written
after `Ready()` (0 dropped under normal traffic); stall watchdog stays
quiet (liveness stamped at receipt).
## Out of scope
A hard crash while messages sit in the in-memory buffer still loses
them; crash-durability requires broker-side persistence, which is
topology-specific. This PR closes the startup-migration and deploy loss
windows.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Firmware 1.16.0 (`companion-v1.16.0`) ships variable-length
`PAYLOAD_TYPE_ACK` payloads: 4 bytes (legacy) → 5 bytes (4-byte CRC +
1-byte attempt, commit `f6e6fdaa`) → 6 bytes (+ 1-byte RNG, commit
`a130a95a`). CoreScope's decoder previously truncated past the 4-byte
CRC and discarded the attempt + RNG bytes.
This PR teaches `cmd/ingestor/decoder.go` to surface the extended bytes
on the decoded payload so the DB/UI can distinguish v1.15 vs v1.16
senders, with no schema or wire-compat changes.
Partial fix for #1610 — top-level ACK + multipart-inner ACK are covered.
PATH-extra ACK parsing (`decodePathPayload`) is deferred to #1612 per
triage.
## Changes
- `decodeAck` reads 4/5/6-byte payloads. Keeps `extraHash` (4-byte CRC)
for compat; adds optional `ackLen`, `ackAttempt`, `ackRand` JSON fields.
Legacy 4-byte ACKs leave attempt/rand `nil`.
- `decodeMultipart` ACK branch relaxes the `len >= 5` floor so the inner
blob can be 4/5/6 bytes (multipart `payload_len` 5/6/7). Adds
`innerAckLen`, `innerAckAttempt`, `innerAckRand`.
- All additions are `omitempty` — backwards-compatible JSON only. No DB
column, no schema migration, no frontend change.
## Out of scope (per issue triage)
- `decodePathPayload` PATH-extra parsing — tracked separately in #1612.
- Frontend rendering of attempt counter — leave for a follow-up if the
DB/UI eventually wants to display it.
## TDD
- **Red commit `3fce0465`** adds `cmd/ingestor/issue1610_test.go` with 6
new assertions (legacy 4-byte, extended 5/6-byte, multipart variants of
each). New fields are declared on `Payload` so the test compiles, but no
decoder populates them yet — tests fail on `ackLen=<nil> want 4` etc.
Verified isolation with `git stash` of decoder.go + re-run.
- **Green commit `5165c202`** implements the decoder changes. `go test
./...` in `cmd/ingestor` passes.
## Fixtures
Synthetic wire vectors built by hand against the firmware spec — the
issue did not provide real captures. Each test cites the firmware ref +
commit it derives from (`BaseChatMesh.cpp:218-234`, commits `f6e6fdaa`
and `a130a95a`).
## References
- Issue #1610
- Firmware tag `companion-v1.16.0` @ `07a3ca9e`
- Upstream PR meshcore-dev/MeshCore#2594
- Blog: https://blog.meshcore.io/2026/06/06/release-1-16-0
---------
Co-authored-by: corescope-bot <bot@corescope.local>
## Summary
Mirrors the distance-index lazy pattern (#1011): the subpath and
path-hop index builds are no longer part of `Load()`'s synchronous
critical section. They now run in **two parallel background goroutines**
kicked off after `s.loaded = true`, so HTTP comes up immediately even at
Cascadia scale (5M observations, previously ~60s blocked on these two
builds inside `Load()` under `s.mu`).
Fixes#1008.
## Approach
Two new `atomic.Bool` fields on `PacketStore` (`subpathReady`,
`pathHopReady`) plus a one-shot broadcast channel (`indexReadyChan`) for
waiters. `Load()` removes the synchronous `s.buildSubpathIndex()` /
`s.buildPathHopIndex()` calls and instead kicks
`s.startBackgroundIndexBuilds()` right before returning. That function
spawns **two independent goroutines** (review m7), one per index. Each
goroutine:
1. acquires `s.mu.Lock()` (blocks until `Load()`'s deferred Unlock
fires),
2. runs its builder, releases the lock, stores its `ready = true`,
3. closes the broadcast channel if both flags are now true,
4. logs `[startup] index build complete: subpath (Xs)` (or pathHop).
Analytics handlers whose entire response IS the index aggregate —
`/api/analytics/subpaths`, `/api/analytics/subpaths-bulk`,
`/api/analytics/subpath-detail`, `/api/nodes/{pubkey}/paths` — gate
reads behind the corresponding atomic and respond with `503 Service
Unavailable`, `Retry-After: 5`, body `{"error":"index
loading","retryAfter":5}` until the build completes — matching the
triage spec.
### Handler scope (review M2)
A second class of handlers also touches these indexes — `/api/nodes`,
`/api/nodes/{pubkey}`, the `GetRepeaterRelayInfoMap` /
`GetRepeaterUsefulnessScoreMap` / `GetBridgeScore` enrichment helpers,
and `repeater_liveness` / `repeater_usefulness`. These are
**intentionally NOT 503-gated**: they expose the index via optional
enrichment fields that callers already treat as "may be empty", and
503-ing the SPA bootstrap to wait for an index that only affects
relay-activity badges would be a worse UX than a 30–60s window of "—"
values. The rationale is documented in the package doc-comment at the
top of `index_ready_1008.go`.
The recomputer's synchronous prewarm path
(`StartRepeaterEnrichmentRecomputer`) gates on `WaitIndexesReady(60s)`
(review M1) so it never snapshots an empty `byPathHop` into
`s.repeaterRelayCache`; on timeout it skips the prewarm and lets the
5-minute ticker pick up the populated index.
## Concurrency safety
Each build goroutine acquires `s.mu.Lock()` before calling the existing
`buildSubpathIndex()` / `buildPathHopIndex()` helpers, which replace
`s.spIndex` / `s.spTxIndex` / `s.byPathHop` with freshly-allocated maps.
Visibility of the populated maps to handlers that observe
`Ready()==true` is established by Go 1.19+ sync/atomic acquire-release
semantics: the atomic store of `true` happens-after `s.mu.Unlock()`, and
the handler's atomic load synchronizes-with that store. The handler's
subsequent `s.mu.RLock` serializes against concurrent ingest writers,
not against the builder.
The existing `main.go` boot sequence does not start ingest goroutines
until after `store.Load()` returns and graph init completes, so the
brief window between `Load()` returning and the two goroutines acquiring
`s.mu` does not race with concurrent ingest writes.
## TDD: red → green
- **Red** commit `63e79e11`: `cmd/server/index_ready_1008_test.go` adds
four assertions; `cmd/server/index_ready_1008.go` adds compile-only
stubs returning `true` so the tests fail on assertions, not build
errors.
- **Green** commit `fb1d22b0`: implements the real atomic gates, the
background goroutine, and the four handler 503 branches; also updates
four existing tests that read indexes directly post-`Load()` to call
`store.WaitIndexesReady(5s)` first.
- **Race-fix commit `b77d56eb`** (review m8 — test-infra exemption):
adds `WaitIndexesReady` calls in test helpers/setup paths so the race
detector no longer flags the read-after-Load() pattern in existing
tests. Per AGENTS.md, race-detector flakes are observable evidence (test
crashes under `-race`) and qualify for the test-infra exemption from the
TDD red-commit requirement; no behavior change in production code.
- **Polish round 2 — M1 red `408c7462` / green `85e82c8a`**:
`TestIssue1008_M1_PrewarmWaitsForIndexes` asserts the recomputer prewarm
SKIPs when indexes are not ready. Red commit adds the assertion + a stub
`repeaterEnrichmentPrewarmWait` var; green commit wires
`WaitIndexesReady` into the prewarm path and adds the handler-scope docs
for M2.
- **Polish round 2 — minor cleanups `fd089bd0`** (m3..m7): chunk-loader
wires `markIndexesReadySync`, memory-model comment rewritten to cite
acquire-release, sentinel deleted, polling replaced with a broadcast
channel, two parallel goroutines for the builds.
`TestIssue1008_m7_BothFlagsSetAfterParallelStart` covers the parallel
path.
## Reproduction
```
git fetch origin fix/issue-1008
git checkout 63e79e11 # red commit
cd cmd/server && go test -run TestIssue1008_ -count=1 . # FAILs
git checkout fix/issue-1008 # latest green
cd cmd/server && go test -run TestIssue1008 -count=1 -race . # all pass
cd cmd/server && go test -count=1 -race -short ./... # full suite ok
```
## Files changed
| file | role |
|---|---|
| `cmd/server/store.go` | atomic.Bool fields + indexReadyChan broadcast
field; remove sync build calls in Load(); kick goroutines; wire
markIndexesReadySync from chunk loader |
| `cmd/server/index_ready_1008.go` | ready flags, two-goroutine
background builds, 503 helper, channel-based WaitIndexesReady,
handler-scope docs |
| `cmd/server/index_ready_1008_test.go` | red-commit contract tests +
parallel-start assertion |
| `cmd/server/repeater_enrich_recomputer.go` | gate prewarm on
WaitIndexesReady (M1) |
| `cmd/server/repeater_enrich_recomputer_1008_test.go` | M1 red+green
assertions |
| `cmd/server/routes.go` | 503 gate on 4 analytics handlers |
| `cmd/server/routes_test.go` | setup helpers wait for ready; collision
test waits |
| `cmd/server/coverage_test.go` | three tests wait for ready before
reading indexes |
## Out of scope
- Distance index (already deferred in #1011) — untouched.
- The `pickBestObservation` + `indexByNode` per-tx loop in `Load()` —
kept synchronous per triage Findings (ordering-sensitive,
contiguous-memory, fast).
---------
Co-authored-by: bot <bot@noreply.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>
Implements the locked spec from #1359.
Red commit: 68a140a8 — `distinctRelayCount` stub returns 0; test fails
on assertion (compiles + runs to assertion, not a build error).
Green commit: 48c2ddad — real implementation.
## Backend (in-memory, no SQL, no schema change)
- `cmd/server/relay_airtime_share.go`
- `distinctRelayCount(tx)` — unions the resolved-pubkey reverse index
for `tx.ID`. That index already dedups `(pubkey-hash, txID)` pairs
across every observation's `resolved_path`, so its length IS the count
of distinct repeaters that forwarded the packet. NOT length of any
single observation's resolved_path (the bug-trap from #1358).
- `computeRelayAirtimeShare(window)` — per-tx `score = payload_bytes ×
distinctRelays`, bucketed by `payload_type`, sorted desc by airtime_pct.
- `GetRelayAirtimeShareWithWindow` — cached behind existing `rfCache` +
`rfCacheTTL` pool. Shallow-copies the cached payload with `cached=true`
for the client.
- `cmd/server/routes.go` — `GET
/api/analytics/relay-airtime-share?window=…` returning
`{rows:[{payload_type,type,count,count_pct,score,airtime_pct}],
total_count, total_score, window, cached}`.
## Frontend
- `public/analytics.js`
- `renderRelayAirtimeDumbbell(data)` — horizontal dumbbell chart per
payload_type. Gray dot = count %, colored dot = airtime %, connector
line between them = the divergence, shared 0-100% axis, sorted desc by
airtime.
- Tooltip: payload_type, count %, count N, airtime %, raw score,
within-mesh caveat.
- Title: **Relay Airtime Share**.
- Subtitle (exact): `Score = payload bytes × distinct repeaters that
forwarded the packet. Counts relay re-transmissions; originator TX
excluded. Not comparable across meshes.`
- Mounted on the Overview tab immediately beneath Payload Type Mix.
## Tests
`TestRelayAirtimeShare_ADVERTvsACKDivergence` — the locked acceptance
scenario:
- 1 ADVERT (200 B, 8 distinct relays) → score 1600, airtime 100%
- 1000 ACKs (10 B, 0 relays each) → score 0, airtime 0%
- Count distribution is the inverse (ACK 99.9%, ADVERT 0.1%).
- Sort assertion: ADVERT is rows[0] by airtime_pct desc.
Full suite: `go test -short ./cmd/server/...` → PASS (25.9s).
## Acceptance criteria
- [x] In-memory `airtime_usage_score` accumulator in analytics path
- [x] `distinctRelayCount(tx)` helper unioning resolved-pubkey reverse
index across all observations of `transmission_id`
- [x] `/api/analytics/relay-airtime-share?window=…` endpoint
- [x] Cached via existing `rfCache` + `rfCacheTTL`; no new cache layer
- [x] Dumbbell chart on `/analytics` beneath Payload Type Mix;
gray=count, colored=airtime, shared axis, sorted desc by airtime
- [x] Title + subtitle exactly as specified
- [x] Tooltip with payload_type, count %, count N, airtime %, raw score,
caveat
- [x] Unit test demonstrates the ADVERT-vs-ACK divergence
- [x] No new SQL, no new index, no schema migration (verified via diff)
- [ ] Live staging bench (<5ms p99 uncached / <1ms cached) — deferred to
follow-up; cached behind 60s `rfCacheTTL` so steady-state cost is a map
lookup
## Preflight overrides
- Branch scope cross-stack: justified — backend endpoint and frontend
chart are a single deliverable per #1359 spec (one chart bound to one
endpoint, no incremental staging).
Fixes#1359
---------
Co-authored-by: bot <bot@local>
## Summary
Addresses the remaining acceptance gap on #1120: a true **5-minute
rolling-baseline anomaly detector** for the Perf-page Write Sources
table. The endpoints + ingestor wiring + UI scaffolding landed in #1123
(partial); this PR replaces the ad-hoc tx-rate comparison with the
rolling baseline the issue actually asks for, and adds a JS unit test
that proves the ⚠️ flag fires at 11× baseline.
## What changed
- **`public/perf.js`** — new pure helper `detectPerfAnomalies(history,
current, opts)`. Computes per-component current rate and rolling
baseline rate over a window (default 5 min). Flags components whose
current rate > 10× baseline. Includes a 0.05/s floor so a stale `0`
baseline doesn't false-positive at startup.
- **UI** — Write Sources table now shows `Rate/s`, `Baseline/s`, and
`Anomaly` columns. Operators can sanity-check the ⚠️ rather than
trusting opaque output. History is kept on `window` and pruned to a
6-min sliding ring.
- **`test-perf-anomaly.js`** — new VM-sandbox test asserting:
- ⚠️ fires when one component runs at 11× its 5-min baseline
- No ⚠️ at 5× (under threshold)
- No ⚠️ until ≥30s of history has accumulated
## TDD evidence (red → green)
- Red commit `590f04d3`: introduces the stub `detectPerfAnomalies`
(returns empty `{flags:{}}`) + the test. Test FAILS on the
`assert(r.flags.backfill_path_json === true, ...)` assertion — not a
build error.
```
❌⚠️ fires when backfill rate hits 11× the 5-minute baseline:
expected backfill_path_json flagged at 11× baseline, got flags={}
2 passed, 1 failed
```
- Green commit `726a5e78`: implements the rolling-baseline detector. All
3 tests pass; existing `test-packet-filter.js` (79 tests) still green;
`cmd/server` Go tests for `/api/perf/*` still green.
## What is NOT in this PR (deferred / out of scope per brief)
- **SQLite-stats subsection** (WAL size + cache hit rate + pending
checkpoint) — `/api/perf/sqlite` already exists (landed in #1123). Issue
body lists it as a metric category, brief explicitly marks it OPTIONAL.
Not regressed; no changes needed.
- **Ingestor `/proc/self/io` bridge** — already lives in the ingestor
stats file (`ProcIO` field, `internal/perfio`) and is rendered on the
Perf page. No change.
- **Issue #1340** (SQLite write-lock instrumentation) — separate PR in
flight, not piggybacked.
- **No new metrics backend** (no Prometheus, no OpenTelemetry). Pure
JSON over `/api/perf/*`.
## Hard-rule compliance
- Files changed: 2 (`public/perf.js`, `test-perf-anomaly.js`) — well
inside the 3-files-outside-allowed-set cap.
- `Stats` struct unchanged.
- All colors via CSS variables — no hex literals introduced (grep
clean).
- TDD: red commit fails on assertion, green commit passes — visible in
branch history.
- PII preflight: clean on both commits.
Partial fix language deliberately not used — this completes the issue's
UI acceptance criterion. Leaving `Fixes #1120` off so the user can
verify on the staging deploy before closing.
---------
Co-authored-by: meshcore-bot <bot@meshcore>
## What
The Route Patterns chart on `/#/analytics` ignored the Time window
picker — every selection returned identical data. This PR threads
`?window=` through to the backing endpoints and the store-level
computation.
## Root cause
`cmd/server/routes.go:2065` (`handleAnalyticsSubpaths`) and
`cmd/server/routes.go:2090` (`handleAnalyticsSubpathsBulk`) never called
`ParseTimeWindow(r)`. The store-level entry points
(`GetAnalyticsSubpaths`, `GetAnalyticsSubpathsBulk`) had no window-aware
variant. The frontend (`public/analytics.js`) didn't append `&window=`
to the `/analytics/subpaths-bulk` request.
## Fix
### Backend (`cmd/server/store.go`)
Added `GetAnalyticsSubpathsWithWindow` +
`GetAnalyticsSubpathsBulkWithWindow`. Zero `TimeWindow` →
byte-equivalent to the existing fast path (no perf regression on the
default view). Non-zero window → iterate `s.packets`, filter on
`tx.FirstSeen` via `TimeWindow.Includes`, reuse `rankSubpaths`. Cached
by `(region|area|window)`.
```diff
-data := s.store.GetAnalyticsSubpaths(region, minLen, maxLen, limit)
+window := ParseTimeWindow(r)
+data := s.store.GetAnalyticsSubpathsWithWindow(region, minLen, maxLen, limit, window)
```
```diff
-results := s.store.GetAnalyticsSubpathsBulk(region, groups)
+results := s.store.GetAnalyticsSubpathsBulkWithWindow(region, groups, ParseTimeWindow(r))
```
### Frontend (`public/analytics.js`)
`renderSubpaths` now appends `&window=<value>` to the
`/analytics/subpaths-bulk` request, matching how RF / topology /
channels tabs already wire the picker.
## Before / after
```
GET /api/analytics/subpaths?window=24h → totalPaths=2 (all data — ignored window)
GET /api/analytics/subpaths?window=24h → totalPaths=1 (24h-bounded — honored)
```
## Tests
`cmd/server/subpaths_window_test.go`:
- `TestSubpathsHonorsTimeWindow_StoreLevel` — seeds a 1h-old tx with
path `[aa,bb]` + a 30d-old tx with path `[cc,dd]`; asserts the unbounded
call sees both and the 24h-windowed call sees only the recent one.
- `TestSubpathsHandlerHonorsTimeWindow` — same scenario via the HTTP
handlers for `/api/analytics/subpaths` and
`/api/analytics/subpaths-bulk`.
TDD: red commit `eefc27d3` (test fails on assertion with stub that
ignores window), green commit `4c4c45d0` (implementation makes it pass).
Full `go test ./...` in `cmd/server` green locally (~47s).
## Performance
Default view (no window selected) is unchanged — `window.IsZero()`
short-circuits to the existing precomputed-index hot path. Windowed view
is O(N_tx · path²), same complexity as the existing region-filtered slow
path. Results cached per `(region|area|window)`.
Closes#1217
---------
Co-authored-by: Kpa-clawbot <bot@corescope>
Fixes#1616. Supersedes the soften-and-track approach from #1172 (now
closed).
## What
Architectural fix for the slide-over close path so it no longer
transitions through a `focused-but-hidden` state. Chromium-headless
cannot deterministically order focus/blur events when `panel.hidden =
true` happens in the same microtask as a delegated table re-render —
root cause of the flake family that was blocking ~8 unrelated PRs at a
time and flipping master CI ~50%.
## How (three changes per #1616 acceptance criteria)
1. **Panel detach on close.** `open()` attaches panel + backdrop to
`<body>`; `close()` removes them. `isOpen()` is now a boolean flag
(`panelOpen`) instead of `(!panel.hidden)` — the closed panel literally
does not exist in the document tree, so there is no focused-but-hidden
window.
2. **Focus restore by `data-value` lookup at restore time.** Sync
`tr.focus()` BEFORE detach. If `document.activeElement !== tr` after the
sync call, attach a one-shot `MutationObserver` on the table's `tbody`;
on a matching row re-attach, call `.focus()` once and `disconnect()`.
Observer has a 2s timeout fallback so it doesn't leak when the row is
genuinely gone.
3. **Permanent CI flake-gate.** New step in
`.github/workflows/deploy.yml`: runs `test-slideover-1056-e2e.js` 20
consecutive times. Any single non-zero exit aborts. If this step ever
turns red post-merge, the focused-but-hidden state has crept back in.
## Hard-asserted (no more soft-warn)
All three deferred assertions are now `assert(...)`:
- `focus-restore@800: Escape returns focus to originating row`
- `focus-restore@800: X-button click returns focus to originating row`
- `resize@800→1440 nodes: cleanup releases panel, backdrop, scroll-lock,
focus` (focusRestored portion)
## Commits
- `fce39304` — RED: un-skip the two soft-skipped assertions
- `cead78df` — GREEN: architectural fix (detach + MutationObserver)
- `4f6d5c47` — CI: permanent `--repeat-each=20` flake-gate
## Verification
The 20-run gate is the verification. Watch the new `Slide-over E2E
flake-gate (#1616, --repeat-each=20)` step on this PR's CI; merge only
if it passes.
## Why this is the right fix
Five prior patches (`7891b70`, `366af4f`, `36ebecc`, `df5397f`,
`d681505`) all targeted the focus call ordering and all flaked in CI
Chromium-headless. The unfixable bit is "hidden-but-was-focused" —
Chromium reorders blur/focus across that transition
non-deterministically. Removing the transition (detach instead of hide)
removes the race entirely.
Closes#1616. Closes#1172 (already closed).
---------
Co-authored-by: openclaw-bot <bot@openclaw>
Co-authored-by: CoreScope bot <bot@corescope.local>
Co-authored-by: clawbot <bot@clawbot.local>
Fixes#1614
## Problem
`window.getTileUrl()` in `public/roles.js` returned the active
provider's `url` property as-is. After #1533 added carto/osm/stamen
providers with lazy-resolved URLs (`url: function () { ... }`), the
helper returned the function itself instead of a URL template string.
Callers handed that function to `L.tileLayer()`, which stringified the
source as the template — every tile 404'd, the map went blank, and
Leaflet logged no error.
User-visible impact: node-detail inset map and analytics minimap
rendered zero tiles whenever a function-`url` provider was the active
dark-theme pick.
## Root cause
`public/roles.js:365-381` — `return p.url || p.baseUrl;` with no `typeof
=== 'function'` invocation. The provider registry in
`public/map-tile-providers.js:45-53` declares almost every provider with
`url: function() { ... }` for lazy config resolution (cartocdn domain,
OSM provider/token, Stamen API key).
## Fix
One-line change in the consumer (`getTileUrl()`). Invoke `url` /
`baseUrl` if it's a function; otherwise return it verbatim.
`map-tile-providers.js` is not touched — it remains the source of truth
for the lazy-resolver pattern.
```js
var u = p.url || p.baseUrl;
return (typeof u === 'function') ? u() : u;
```
## Callers reviewed
| Caller | Disposition |
| --- | --- |
| `public/nodes.js:94` (`_applyTilesToNodeMap`) | Routes through
`window.getTileUrl()` → fixed transitively |
| `public/analytics.js:2055` (`L.tileLayer(getTileUrl(), …)`) | Routes
through `getTileUrl()` → fixed transitively |
| No other `getTileUrl()` callers | `grep -n "getTileUrl\b" public/*.js`
confirms only the two above |
## Commits (red → green)
- `a2b23392` — `test(#1614): red — getTileUrl() must return string, not
function` — adds `test-issue-1614-tile-url-function.js`. Verified to
fail on assertion (not build error) before the fix landed; passes after.
- `26fcacd1` — `fix(#1614): invoke provider url() when it's a function`
— minimal one-line fix in `roles.js` plus wiring the new test into
`deploy.yml` and `test-all.sh`.
## Tests
Unit test asserts the public contract from three angles so any
regression of either branch fails CI:
1. Dark + `url: function()` → returns a string template containing
`{z}/{x}/{y}`.
2. Dark + `url: 'https://…'` → returns the string verbatim (no
double-invoke).
3. Dark + `baseUrl: function()` fallback → also invoked, also returns a
string.
Wired into CI via `.github/workflows/deploy.yml` and `test-all.sh`.
## E2E coverage
Skipped intentionally. The existing Playwright harness
(`test-e2e-playwright.js`) runs against a deployed BASE_URL and is not
invoked from the Go CI workflow (`deploy.yml`). Adding a new E2E flow
there would require standing up a leaflet/tile-loading harness for a
single one-line regression. The unit test covers the exact
`getTileUrl()` contract that this bug violates and would have caught it;
if reviewers want a Playwright assertion later we can add it as a
follow-up. Manual verification was performed against staging
(`http://analyzer-stg.00id.net/#/nodes/...`).
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
— clean (all gates pass, PII clean, red commit verified).
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
## Summary
Fixes#1606 — frontend `public/nodes.js` issued a single `?limit=5000`
fetch to `/api/nodes` and trusted the response as the complete node set.
After PR #1540 (v3.8.3) clamped `/api/nodes` `?limit` to 500 as a DoS
guard, that single fetch silently truncated to the top 500 rows by
`last_seen DESC`. On the reporter's 2313-node deployment, **78% of nodes
(1813) were invisible** in the Nodes page, with no UI indication
anything was missing.
Replaces the single fetch in `loadNodes()` with a pagination loop driven
by `data.total` from the first response. Stops when `_allNodes.length >=
total`, when the server returns a short page, or at a 10 000-row safety
cap. `counts` is taken from the first response and refreshed on each
subsequent page (last writer wins; the server returns the same `counts`
payload each call).
Scope is deliberately narrow per the (munger) finding in the triage
comment: the three sibling call sites (`analytics.js:2080,2817`,
`packets.js:791`) are **NOT** touched here. They get their own
follow-up.
## Repro
```bash
curl -s "https://analyzer.marwoj.net/api/nodes?limit=5000" | jq '{nodes_len: (.nodes | length), total}'
# Before fix on >500-node deployment:
# { "nodes_len": 500, "total": 2313 } ← frontend silently displays only 500
```
## Before / after evidence
Unit test `test-issue-1606-pagination.js` drives `loadNodes()` against a
mocked `api()` exposing 1200 fixture nodes with a 500-per-page server
cap (mirrors the real `/api/nodes` clamp).
| | `_allNodes.length` | `data.total` |
|---|---:|---:|
| Before (single fetch) | **500** | 1200 |
| After (pagination loop) | **1200** | 1200 |
Red commit: `700a5cc4` (test asserts `_allNodes.length === data.total`,
fails 500 ≠ 1200).
Green commit: `6d51da45` (pagination loop, test passes).
All 611 tests in `test-frontend-helpers.js` continue to pass — the
existing nodes.js WS-handler runtime tests are unaffected.
## Browser verified
Mocked-API unit test only — staging currently has <500 nodes so the bug
isn't reproducible there. The reporter's deployment
(`analyzer.marwoj.net`, 2313 nodes) is where the visible regression
occurs. The unit test reproduces the exact failure mode against a
controllable fixture.
## E2E assertion added
`test-issue-1606-pagination.js:170` — `assert.strictEqual(all.length,
env.fixtureTotal, ...)`
## Files changed
- `public/nodes.js` — `loadNodes()` single fetch → pagination loop
- `test-issue-1606-pagination.js` — new regression test (sandboxed
nodes.js + mock api)
## Out of scope (deferred to follow-up)
Per triage's (munger) note, these three siblings have the same
single-fetch bug and need their own focused PR:
- `public/analytics.js:2080` (`limit=10000`)
- `public/analytics.js:2817` (`limit=10000`)
- `public/packets.js:791` (`limit=2000`)
Closes#1606
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
fix for the focus-restore@800 E2E test that's currently failing on
master (see runs 26990436988, 26986419081)
Chromium headless is notorious for dropping synchronous or rAF-based
focus restores when elements are hidden. By manually blurring the active
element before hiding the panel, and staggering the focus restore with a
setTimeout macrotask after the rAF, we ensure the focus call lands after
the browser has completed all implicit focus resets and event handlers.
Furthermore, dynamically evaluating the focus resolver directly inside
the deferred focus attempt prevents the target element from becoming
stale if a live WebSocket packet triggers a background table re-render
in the intervening milliseconds.
## Summary
Partial fix for #1599 — replay from packets sidebar no longer freezes
the live map.
Clicking **Replay** on a packets-page row wrote the packet to
`sessionStorage['replay-packet']` and navigated to `/#/live`. On init,
`live.js` called `vcrPause()` to silence live WS traffic during the
replay. But `vcrPause()` sets `VCR.mode = 'PAUSED'`, and
`renderAnimations()` gates `anim.progress` advancement on `!isPaused` —
so the replayed animation never advanced and the map appeared frozen.
## Fix
Introduce a module-level `suppressLive` flag dedicated to muting live WS
traffic without entering `PAUSED`. The WS handler's `LIVE` branch honors
the flag (still ticking `updateTimeline` so the UI keeps reflecting
traffic). The replay handoff sets the flag for ~12 s — long enough for
the animation to play out — then clears it.
Files changed:
- `public/live.js` — module flag (`~145`), replay handoff (`~1502`), WS
LIVE branch (`~897`)
- `test-issue-1599-replay-freeze-e2e.js` — new Playwright E2E (seeds
`sessionStorage['replay-packet']`, asserts `activeAnimations` drains
after the handoff)
- `.github/workflows/deploy.yml` — wire the new E2E into the deploy E2E
block
## TDD trail
| Commit | Role |
| --- | --- |
| `8a0add00` | Red — failing E2E (asserts the queued animation drains;
pre-fix it never does → `FAIL: activeAnimations did NOT drain after
replay handoff (count=1) — replay freeze regression`) |
| `8069210d` | Green — `suppressLive` flag replaces `vcrPause()` in the
handoff |
| `c2a84a3e` | CI wiring |
Locally reproduced both states against the e2e-fixture DB (Chromium via
`CHROMIUM_PATH=/usr/bin/chromium`):
- HEAD red commit: `2 pass, 1 fail` (assertion-shaped, not compile)
- HEAD green commit: `3 pass, 0 fail`
Browser verified: local Chromium against `corescope-server -port 13581
-db /tmp/e2e-fixture.db -public public` — `replay-packet` key is
consumed by the init path, animation queues, and drains post-fix.
E2E assertion added: `test-issue-1599-replay-freeze-e2e.js:111`
(`activeAnimations drained to 0`).
## What this PR does NOT do
The reporter explicitly called out a second, separable problem on the
same issue: `renderPacketTree(packets, true)` runs with `isReplay =
true`, which skips `addFeedItem` (`public/live.js:3155`), so the
bottom-left feed shows "Waiting for packets…" even once the map
animates. That is a UX decision (should the replayed packet appear in
the feed?) and is intentionally **not** addressed here. Leaving #1599
open so the operator can decide.
Hence: **"Partial fix for #1599"** — no `Fixes #` keyword.
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
→ all hard gates ✅, no warnings.
---------
Co-authored-by: corescope-bot <bot@corescope>
## Summary
Build the distance analytics index lazily on the first
`/api/analytics/distance` request instead of eagerly inside `Load()`
(and its background-load chunked merge). Per the triage Fix path on the
issue:
- Eager startup build removed from `Load()` and from
`loadAllPacketsBackground()`'s post-merge pass.
- First request returns `202 Accepted` + `Retry-After: 5` and kicks off
the build in a background goroutine, gated by `sync.Once` so concurrent
first-window requests all observe 202 (single build, not N parallel
O(n²) computations).
- Once built, subsequent requests fall through to the existing
analytics-recomputer / TTL cache and serve 200 as before.
- Debounced rebuild policy: refire only when `Δobs > 5%` since last
build OR `>5 min` elapsed, whichever is more restrictive. Background
loader also resets the gate so the next request rebuilds against the
larger dataset.
Effect: operators who never visit distance analytics no longer pay the
O(n²) construction at startup. Acceptance criteria (a) no startup build,
(b) first request triggers build, (c) concurrent in-flight requests get
202 are encoded as failing-first tests.
## Red → green
- Red: `bc947ad1` — 3 assertion failures (`expected ... empty, got 3`,
`expected 202, got 200`, `expected all 10 ... got 0`).
- Green: `5264b68a` — production change makes them pass, no other tests
regress.
## Files changed
- `cmd/server/store.go` — lazy-build state
(`distLazyMu`/`Once`/`Built`/`Building`/`LastBuilt`/`LastObs`),
`TriggerDistanceIndexBuild`, `DistanceIndexBuilt`,
`DistanceIndexBuilding`; eager `buildDistanceIndex` calls in `Load()`
post-pass and chunked-background-load post-pass removed (Once reset
instead so the next request rebuilds against the full dataset).
- `cmd/server/routes.go` — `/api/analytics/distance` returns 202 +
`Retry-After` until built.
- `cmd/server/distance_lazy_index_test.go` — new tests (the three triage
acceptance criteria).
- `cmd/server/coverage_test.go`, `cmd/server/parity_test.go`,
`cmd/server/routes_test.go`, `cmd/server/hop_disambig_e2e_test.go` —
pre-warm the index via `TriggerDistanceIndexBuild()` +
`DistanceIndexBuilt()` poll where the test asserts the 200 JSON shape.
## Perf justification
Startup cost on a 500K-obs / 2K-node dataset: previously O(n²) hop scan
during `Load()` post-pass and again during the background-load merge —
measured at 10–20s in `specs/startup-audit.md`. New code: zero work at
startup, the same O(n²) work runs at most once per HTTP request cycle
(and only when the index is stale per debounce policy). Cold-path
concurrency is bounded by `sync.Once`, so N parallel first-window
requests never produce N parallel builds.
## Scope
No config field added (debounce thresholds are hardcoded constants per
the triage Fix path — `5%` / `5min`). No public API signature changes.
No DB-side migration. Tests cover the lazy invariant, the
202+Retry-After contract, and concurrent first-request behavior.
Closes#1011
---------
Co-authored-by: Kpa-clawbot <bot@corescope.local>
## Problem
`/analytics` Hash Usage Matrix 1-byte view excluded repeaters configured
for 2- or 3-byte hash prefixes. In MeshCore, 1-byte path-matching is a
first-byte equality check, so any packet routed by 1-byte hash collides
on that first byte regardless of the downstream repeater's configured
prefix size. Omitting multi-byte prefix repeaters under-reports real
conflicts in the 1-byte hash space.
## Fix
**Data layer — `cmd/server/store.go` (`computeHashCollisions`,
~L7907-L7918 before, L7907-L7941 after):**
Before — `one_byte_cells` was populated only from `prefixMap`, which
only contained repeaters with `hash_size == 1`:
```go
if bytes == 1 {
oneByteCells = make(map[string][]collisionNode)
for i := 0; i < 256; i++ {
hex := strings.ToUpper(fmt.Sprintf("%02x", i))
oneByteCells[hex] = prefixMap[hex]
if oneByteCells[hex] == nil {
oneByteCells[hex] = make([]collisionNode, 0)
}
}
} else if bytes == 2 { ... }
```
After — additionally project all `hash_size in {2,3}` repeaters to their
first byte:
```go
if bytes == 1 {
// ... (same baseline population) ...
for _, cn := range allCNodes {
if cn.Role != "repeater" { continue }
if cn.HashSize != 2 && cn.HashSize != 3 { continue }
if len(cn.PublicKey) < 2 { continue }
hex := strings.ToUpper(cn.PublicKey[:2])
if _, ok := oneByteCells[hex]; !ok { continue }
oneByteCells[hex] = append(oneByteCells[hex], cn)
}
}
```
The 2-byte view's bucketing is unchanged — that view continues to count
only repeaters configured for 2-byte prefixes (those semantics differ).
**UI — `public/analytics.js` L1459:** clarified the 1-byte view
description so the inclusion of multi-byte prefix repeaters is explicit.
## API shape
No response-shape change. `one_byte_cells[HEX]` is still
`[]collisionNode`; only the contents now include 2/3-byte prefix
repeaters in the appropriate first-byte buckets. The existing frontend
decoder is unaffected.
## Tests
-
`cmd/server/routes_test.go::TestHashCollisionsOneByteIncludesMultiBytePrefixRepeaters`
— seeds three repeaters with first byte `CC` configured for 1/2/3-byte
prefixes plus an unrelated `DD` repeater, asserts all three appear in
`one_byte_cells["CC"]`, and that the 2-byte view's `nodes_for_byte` is
unchanged.
Red commit `278bdf8d` (test only) fails on assertion ("got 1, want 3");
green commit `9127ea4e` passes.
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
→ clean.
Closes#1218
---------
Co-authored-by: clawbot <bot@corescope>
- Eliminated extra space to the right of the map filters.
- Made the map filters and mesh live a single line with a divider
- Resized the input and dropdowns in the map filters so they meet WCAG
2.5.5 by being at least 44px high, but appearing 30px high
- Turned the filters cog and the fullscreen button into native leaflet
icons that are large enough to meet WCAV 2.5.5 compliance
- Increased the size of the zoom buttons to meet WCAG 2.5.5 compliance
on both the live and map pages
- If the top nav bar is pinned, it won't disappear during fullscreen but
if it isn't pinned, it will disappear with everything else.
- The cog and full screen button change color to show they're active
Final Outcome in 4k
<img width="2878" height="1406" alt="image"
src="https://github.com/user-attachments/assets/28db46a2-f1bb-4d9c-9d77-30c444b4ef3d"
/>
Final Outcome in 1080p
<img width="1920" height="1080" alt="image"
src="https://github.com/user-attachments/assets/120be8ec-0279-40fc-925a-243e9c0bcc1c"
/>
## Problem
Node detail's bimodal-clock warning showed only `⚠️ N of last M adverts
had nonsense timestamps (likely RTC reset)` — no way to tell which
packets, no way to verify the heuristic, no way to drill in.
## Fix
Additive, two-sides:
**Backend** (`cmd/server/clock_skew.go`)
- New type `BadSample { Hash, AdvertTS, SkewSec }`.
- New field `NodeClockSkew.RecentBadSamples []BadSample` (`omitempty`).
- Populated from the **same** bimodal-bad classification pass that
produces `RecentBadSampleCount` — no heuristic change. `tsSkewPair`
carries `hash` + `advertTS` so the classifier can record per-sample
evidence without a second walk; drift code is unaffected (reads only
`ts`/`skew`).
**Frontend** (`public/nodes.js`)
- `bimodalWarning` preserves the existing count summary line, then
renders a `<ul>` of bad samples: each `<li>` is `<a
href="#/packets/HASH">hash[:8]</a> → formatTimestamp(advertTS)` with ISO
tooltip. Defensive `Array.isArray` so older API responses still render
the summary alone.
## TDD
- **Red:**
`cmd/server/clock_skew_issue1094_test.go::TestIssue1094_RecentBadSamples_ExposesHashAndTimestamp`
— seeds 3 healthy + 2 bimodal-bad adverts, asserts `RecentBadSamples`
has length 2 with the expected hashes and advert timestamps. Fails on
the assertion (`len = 0, want 2`) with the stub-only commit.
- **Green:** classifier populates the slice; existing #1285 and bimodal
tests stay green.
- Red commit: `ed501f4b`
- Green commit: `54305b06`
## Cross-stack
Backend + frontend ship together (`cross-stack: justified` commit). API
stays backward compatible (`omitempty` server, `Array.isArray` client)
but the feature only lights up with both halves present.
## Preflight
Clean — PII, branch scope, red-commit, CSS vars, XSS sinks, migrations,
fixture coverage all pass.
## Acceptance
- [x] Warning lists specific packet hashes
- [x] Each hash links to `#/packets/<hash>`
- [x] Bad advert timestamp shown next to the hash
- [x] Pattern is reusable — `BadSample` is a clean shape any future
heuristic that flags specific packets can adopt
Fixes#1094
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
Fixes#1170.
## What
1. **Doc comment** on `writeStatsAtomic` (`cmd/ingestor/stats_file.go`)
spelling out the two-sided symlink story:
- tmp side (`path+".tmp"`): protected by `O_NOFOLLOW` (existing
behavior, already noted).
- rename side (`path` itself): NOT protected by `O_NOFOLLOW`; instead
`os.Rename` semantics are relied upon — rename atomically replaces any
existing entry at `path` (including a symlink) with the new regular
file. The symlink target is never written through because all writes
happened to the unrelated tmp file before rename.
2. **Regression guardrail test**
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced` in
`cmd/ingestor/stats_file_test.go` that pre-plants a symlink at the
destination path pointing to an unrelated target file, calls
`writeStatsAtomic`, and asserts:
- (a) `os.Lstat(path).Mode()&os.ModeSymlink == 0` (post-write path is a
regular file, not a symlink)
- (b) the original symlink target's sentinel bytes are unchanged.
If a future refactor swaps `os.Rename` for a
destination-symlink-following primitive (e.g. `open(path, O_WRONLY)`
without `O_NOFOLLOW`, or a copy-then-truncate), the test fails loudly.
## TDD note (red-commit exemption)
The current `writeStatsAtomic` ALREADY satisfies the new test's
assertions — `os.Rename` does the right thing today. Per the fix-issue
skill's exemption for pure-documentation / guardrail tests on
already-correct behavior, no fabricated red commit was constructed; the
test stands as a pinning regression guard. The two commits are
therefore: (1) test addition, (2) doc comment.
## Scope
- `cmd/ingestor/stats_file.go` — doc comment only
- `cmd/ingestor/stats_file_test.go` — one new test function
No production behavior change. No public API change. No new
dependencies. No CI workflow changes. `O_NOFOLLOW` and the existing
tmp-side behavior are untouched.
## Preflight
All hard gates pass (PII, branch scope, red commit, CSS vars,
LIKE-on-JSON, sync/async migration, XSS sinks). No warnings.
---------
Co-authored-by: meshcore-bot <bot@meshcore.local>
Kpa-clawbot