dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-07-03 03:31:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,789 Commits 82 Branches 33 Tags
d437958474df87556b429e6a2e691c7bca741512
Commit Graph

5 Commits

Author SHA1 Message Date
Kpa-clawbot a8c99c61fd fix(#1659): block analytics endpoint until first pass complete (503 Retry-After) (#1688) 
## Summary

Fixes #1659 — analytics cards no longer show the post-restart slice when
"All data" is selected.

## Root cause

After server restart, `s.recompRF` / `s.recompTopology` /
`s.recompChannels` cache the FIRST computation, which is the small
in-RAM observations slice (background chunk-loader has not yet
backfilled history). The recomputer serves that slice through
`GetAnalyticsRFWithWindow`'s default shortcut for an entire recompute
interval, while the client pins it via `CLIENT_TTL.analyticsRF`. UX:
cards show a tiny window even when the user selects "All data".

## Fix shape (option B from the issue body)

Server-side per-recomputer warm-up gate:

- `cmd/server/analytics_warmup_1659.go` adds a per-recomputer
`firstPassDoneNs` atomic timestamp, set ONLY by the first successful
`runOnce()` (CAS-guarded for idempotency). `IsWarmingUp_1659()` /
`FirstPassDoneAt_1659()` are lock-free reads.
- `cmd/server/analytics_recomputer.go` `runOnce()` calls
`markFirstPassDone_1659()` after every successful compute.
- `cmd/server/routes.go` handlers for RF / Topology / Channels: when the
request is the default shape (`region=="" && area=="" &&
window.IsZero()`) AND the matching recomputer is still warming up,
return `503` + `Retry-After: 5` + `{"error":"analytics warming
up","retry_after_s":5}`. Windowed / region-filtered requests bypass the
gate (they already bypass the recomputer cache, so they are unaffected
by the warm-up bug).

Client-side:

- `public/app.js` `api()` helper retries any 503 response, honoring
`Retry-After`, with exponential backoff capped at 30s, max 6 attempts
(~63s total).
- Small "Computing analytics…" banner appears while any warm-up retry is
in flight, dismissed once the request resolves. Pages can override via
`window.onWarmup_1659`.

## Tests

RED commit `8b2b2d7` ships failing-on-assertion tests + a stub. GREEN
commit `2716c23` lands the fix and flips them green.

- `cmd/server/analytics_warmup_1659_test.go` — 3 cases: 503 during
warmup, 200 after first pass, windowed request bypasses gate.
- `test-1659-analytics-warmup.js` — 3 cases: Retry-After honored, retry
cap bounded, non-503 errors not retried. Wired into
`.github/workflows/deploy.yml`.

## Preflight overrides

- cross-stack: justified — server-side 503 contract MUST be paired with
client-side retry-and-banner handling; splitting across two PRs would
land a half-working fix.

Fixes #1659.

---------

Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: openclaw <openclaw@local>
 2026-06-12 21:02:59 +00:00
efiten 317b59ab10 feat: area-based visual node filter — attribute packets by transmitter GPS (#804) (#839) 
## Summary

- Adds configurable GPS polygon areas to `config.json`; nodes are
attributed to an area if their last-known position falls inside the
polygon
- New `Area: …` dropdown filter (matching the existing region filter
style) appears on all analytics, nodes, packets, map, and live screens
when areas are configured
- Backend resolves area membership with a 30s TTL cache; area filter
bypasses the 500-node cap on `/api/bulk-health` so all area nodes are
always returned
- Includes a polygon builder tool (`/area-map.html`) for drawing and
exporting area boundaries

## Changes

**Backend**
- `AreaEntry` type + `Areas` config field
- `GetNodePubkeysInArea` DB query + `resolveAreaNodes` (30s TTL,
`areaNodeMu` RWMutex)
- `PacketQuery.Area` + `filterPackets` polygon check
- `?area=` param propagated through all analytics, topology,
clock-health, and bulk-health routes
- `/api/config/areas` endpoint

**Frontend**
- `area-filter.js`: single-select dropdown, persists to localStorage,
cleans up stale keys on load
- Wired into analytics, nodes, packets, channels, map, and live pages
- Live map clears node markers on area change

**Docs & tools**
- `docs/user-guide/area-filter.md` — configuration and usage guide
- `docs/api-spec.md` — updated with new endpoint and `?area=` param
table
- `tools/area-map.html` — polygon builder for defining area boundaries
- Demo areas added to `config.example.json`

## Test plan

- [x] No areas configured → filter dropdown does not appear on any page
- [x] Areas configured → dropdown appears, "All" selected by default
- [x] Selecting an area filters nodes/packets/topology/map correctly
- [x] Selecting "All" restores unfiltered view
- [x] Selection persists across page reloads (localStorage)
- [x] Stale localStorage key (area removed from config) is cleared on
load
- [x] `/api/bulk-health?area=X` returns all nodes in area (no 500-node
cap)
- [x] `/api/config/areas` returns correct list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
 2026-05-21 14:00:15 -07:00
Kpa-clawbot 4cd8445233 perf(#1265): wire /api/observers/clock-skew + /api/nodes/clock-skew into analytics recomputer (#1266) 
RED: 97f49a0c · CI:
https://github.com/Kpa-clawbot/CoreScope/actions/runs/26046530920

Fixes #1265.

## Problem
On staging two clock-skew endpoints serve compute-on-request:

- `/api/observers/clock-skew` — 3.3s
- `/api/nodes/clock-skew` — 8.9s

Both drive a full `clockSkew.Recompute` over 100k+ adverts while holding
`s.mu.RLock`, blocking under concurrent reader load.

## Fix
Wire both endpoints into the established `analytics_recomputer.go`
pattern (PRs #1248 / #1259 / #1263). Two new slots:

- `recompObserversClockSkew` — wraps `computeObserverCalibrations()`
- `recompNodesClockSkew` — wraps `computeFleetClockSkew()`

Accessors `GetObserverCalibrations` / `GetFleetClockSkew` now prefer the
atomic-pointer snapshot; on-request compute is fallback-only for the
brief window before initial sync compute lands (and for tests that skip
the recomputer).

Default interval **300s**, overridable via:

```json
"analytics": {
  "recomputeIntervalSeconds": {
    "observersClockSkew": 300,
    "nodesClockSkew": 300
  }
}
```

`config.example.json` + the `_comment_analytics` doc updated.

## TDD
- RED `97f49a0c` — `TestClockSkewRecomputersRegistered` +
`TestClockSkewHandlersSteadyStateLatency` (8 concurrent readers × 25
reqs per endpoint, p99 < 100ms gate). Fails on master: recomputer slots
nil.
- GREEN `19599375` — wire + accessor switch. p99 well under 5ms on the
test fixture.

## Verification
```
cd cmd/server && go test ./... -count=1   # ok 42s
bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master   # all gates pass
```

---------

Co-authored-by: CoreScope Bot <bot@corescope.local>
 2026-05-18 12:27:44 -07:00
Kpa-clawbot f81ed5b3cf perf(#1256): wire /api/analytics/roles into steady-state recomputer (#1259) 
RED commit: `0190466d` — failing CI:
https://github.com/Kpa-clawbot/CoreScope/actions (will populate after PR
creation)

## Problem
On staging (commit `d69d9fb`, 78k tx, 2.3M obs), `curl
http://localhost/api/analytics/roles` times out at 60s with 0 bytes —
the Roles tab is unusable. Issue #1256.

PR #1248's steady-state recomputer fan-out (topology / rf / distance /
channels / hash-collisions / hash-sizes) **didn't include roles**. The
legacy handler:

1. Holds `s.mu.RLock` for the entire compute.
2. Calls `GetFleetClockSkew()`, which drives `clockSkew.Recompute(s)`
over all ADVERT transmissions — O(78k) per request.
3. Concurrent ingest writers compound the latency through
writer-starvation.

Result: every request hits the cold path; the response never comes back
inside the 60 s HTTP budget.

## Fix
Add `roles` as the 7th endpoint in the recomputer fan-out — same pattern
as #1248:

- `PacketStore.recompRoles` slot, registered in
`StartAnalyticsRecomputers` with default 5-min interval.
- `PacketStore.GetAnalyticsRoles()` → atomic-pointer load from the
snapshot (sub-ms), with a `computeAnalyticsRoles()` fallback only for
the brief startup window before the initial sync compute completes.
- Handler is now a thin wrapper — no lock-held work on the request path.
- New optional `roles` key under `analytics.recomputeIntervalSeconds` in
config; `config.example.json` and `_comment_analytics` updated.

## Latency (unit-scope benchmark)
- Worst-of-50 handler latency: **<100 ms** (test budget; well under the
2 s p99 acceptance).
- Compute itself is bounded by the existing 5-min recompute window — it
runs once in the background, never on the request path.

## Tests
- RED `0190466d`: asserts `recompRoles` is registered and the handler
returns under the latency budget. Fails on master with `recompRoles not
registered`.
- GREEN `d7784f76`: registers the recomputer + snapshot accessor — both
tests pass.

Fixes #1256

---------

Co-authored-by: openclaw-bot <bot@openclaw.local>
 2026-05-18 07:36:28 -07:00
Kpa-clawbot 356f001027 perf(#1240): steady-state background recompute for analytics endpoints (#1248) 
RED commit: `27630f6a` — adds latency test that fails on master
(p99=225ms > 50ms budget) and a stub `StartAnalyticsRecomputers` that
returns a no-op so the assertion (not a build error) gates the change.

GREEN commit: `20fbbceb` — wires real background recompute
infrastructure. Test passes at p99=~1µs.

## What changed

Replaces the on-request "compute-then-cache" pattern for the
default-shape analytics queries with a steady-state background recompute
loop. Reads always hit an `atomic.Value` snapshot in <1µs regardless of
compute cost or writer contention. Operator principle: serving slightly
stale data quickly beats real-time data slowly.

## Endpoints converted (default 5min interval each)

| Endpoint | Cold compute | Recomputer interval |
|---|---|---|
| `/api/analytics/topology` | ~5s | 5 min |
| `/api/analytics/rf` | ~4s | 5 min |
| `/api/analytics/distance` | ~3s | 5 min |
| `/api/analytics/channels` | ~0.5s | 5 min |
| `/api/analytics/hash-collisions` | ~0.5s | 5 min |
| `/api/analytics/hash-sizes` | ~22ms | 5 min |

All intervals configurable per-endpoint via
`analytics.recomputeIntervalSeconds.<name>` in `config.json`; documented
in `config.example.json`. Default override via
`analytics.defaultIntervalSeconds`.

## Scope: default query only

Only the canonical shape `(region="", window=zero)` is precomputed.
Region- or window-filtered requests fall back to the legacy TTL cache +
on-request compute — keeps recomputer count bounded (6, not 6×N×M).

## Latency

Test `TestAnalyticsRecomputerSteadyStateLatency`: 100 concurrent readers
+ 4 writers churning `s.mu.Lock` on 20k distHops.
- Before: p50=188ms p99=225ms (assertion failed)
- After:  p50=240ns p99=1.1µs (atomic load + map return)

## Shutdown integration

`StartAnalyticsRecomputers` returns a stop closure invoked from
`main.go`'s SIGTERM handler BEFORE `dbClose()` so any in-flight SQLite
compute drains cleanly. `TestAnalyticsRecomputerShutdownNoLeak` confirms
all 6 goroutines are reaped (Δ=6 within 2s).

## Safety details

- Initial compute is synchronous in `Start()` — first read after startup
never sees nil.
- `recover()` inside `runOnce` keeps a compute panic from killing the
goroutine; previous snapshot remains valid.
- `analyticsRecomputerMu` is a sync.RWMutex; recomputer pointers are
read-locked in the hot path. The atomic.Value swap inside `runOnce` is
lock-free.

Fixes #1240.

---------

Co-authored-by: OpenClaw Bot <bot@openclaw.local>
 2026-05-17 17:33:30 +00:00