mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-07-09 07:51:59 +00:00
f0d85bdcab29c968bdd11fcd4e2d0c2f08fd1f44
395 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 clawbot
|
5c10e11293 |
test(#1735): annotate test ALTER probes for preflight
The PREFLIGHT migration-scale gate flags every ALTER TABLE statement in the repo unless it carries the async=true annotation. The new TestIsDuplicateColumnErr_DriverStringPinned test runs ALTER on an in-memory DB to provoke and pin the driver's duplicate-column error wording — surgical addition to keep the gate green. |
||
|
clawbot
|
8e15637bf3 |
feat(#1735): warm-up banner — dismiss + auto-dismiss failed migrations
Group B from PR #1735 round-1 review (must-fix #2). Previously a failed async migration pinned the banner forever: isSteadyState returned false as long as any migration was in 'failed' status, with no path to clear. Operators lost trust in the banner; real new failures got lost in the noise. Fix: - FAILED_AUTO_DISMISS_MS = 10 min from endedAt — past that window the failed entry auto-clears from the banner. The failure is still visible via /api/perf/async-migrations and /api/healthz; only the banner stops blocking. - Per-line × button: explicit user ack immediately removes the failure from the banner. - Fail closed: if endedAt is missing or unparseable, the failure does NOT auto-dismiss (operator must see it). - isSteadyState gets an optional nowMs param (defaults to Date.now) for testability and to make the auto-dismiss math re-render-deterministic. CSS additions: .warmup-banner__item--failed coloring + .warmup-banner__dismiss button styling using existing CSS variable patterns. Tests added: test-warmup-banner-failed-dismiss-1735.js pins: - within window: failure still blocks steady state + appears in messages - past window: failure auto-clears from both - explicit dismiss: immediate removal - missing/malformed endedAt: fails closed (no auto-dismiss) |
||
|
clawbot
|
6d8709be51 |
test(#1735): /api/perf/async-migrations handler tests + tighter reader-yield assertion + orphan-tx test doc
Group F from PR #1735 round-1 review (must-fix #11, #12, #13). #11 — Add cmd/server/async_migrations_handler_test.go covering the four states of /api/perf/async-migrations: - success with rows: 200 + JSON array - empty list: 200 + '[]' (not 'null', so warmup-banner.js can iterate) - readAsyncMigrations error: HTTP 500 + JSON error body (not silently empty — that was the round-1 must-fix) - nil db (server pre-DB-init): 200 + '[]' #13 (kent-beck BLOCKER) — TestChunkedBackfill_YieldsToReaderBetweenBatches: the original threshold (12K rows, 500ms reader-latency bound) was loose enough that a single-tx fake whose total wall time was <500ms could pass. Tightened to: - sample BASELINE reader latency BEFORE backfill starts (avg of 5 probes) - sample BEST reader latency during backfill - assert bestDuring < 80ms absolute AND ratio < 5x baseline (with 5ms floor to avoid sub-ms flakiness) A single-tx implementation that holds the writer the entire wall time would push the during-latency ratio into the 50-100x range and fail deterministically. Comment in the test body explains why. #12 — TestChunkedBackfill_OrphanTxTerminates: doc-only — explain why the orphan insert and seedTransmissions run in separate transactional contexts (orphan has no observation row; can't share seed's tx; the backfill loop is committed-state-only so the split has no effect on what's being asserted). |
||
|
clawbot
|
63ac2df1c8 |
fix(#1735): backfill correctness — suppress redundant terminal fire + recover panicking progress callback
Group E from PR #1735 round-1 review (must-fix #10, #14). - chunkedTxLastSeenBackfill: track lastFired (p, total) and skip the final terminal callback when it would re-fire identical counts already reported by the last in-loop fire. Previously, when the last batch was exactly batchSize-sized, the next chunk returned n=0 and we fired (processed,total) a second time. Operators saw duplicate progress events. - Wrap the progress callback in defer-recover. A panicking callback (operator-supplied or buggy bookkeeping write) is converted to an error and returned, NOT propagated to the ingestor goroutine. RunAsyncMigration already converts a returned error to status=failed with the message in the error column, so end-to-end the migration is properly marked failed with the recovered panic text. Tests added: TestChunkedBackfill_TerminalSuppressedWhenRedundant TestChunkedBackfill_PanicInCallbackRecovered TestChunkedBackfill_PanicViaRunAsyncMigrationMarksFailed |
||
|
clawbot
|
905cf32fc9 |
fix(#1735): schema-setup robustness — sync.Once for ALTER storm, rate-limited warn log, pinned driver-string test
Group D from PR #1735 round-1 review (must-fix #8, #9). - ensureAsyncMigrationProgressColumns: guard with sync.Once so a process that runs many async migrations doesn't re-run 3 ALTER TABLE statements every call. The column set is fixed at build time, so once-per-process is the correct scope. - Remove progressSchemaWarnOnce (sync.Once) for the per-write warn log. Replace with a wall-clock rate-limiter (1/min). sync.Once silenced all future errors — destroying observability of an ongoing problem. The rate-limited approach lets every error remain visible without flooding the log on rapid retries. - isDuplicateColumnErr: the modernc.org/sqlite driver does not expose a typed sentinel for duplicate-column ADD COLUMN failures. Document why the substring match is correct AND add TestIsDuplicateColumnErr_DriverStringPinned which provokes the actual driver error so a future driver upgrade that changes the wording fails CI loudly. - Add TestEnsureAsyncMigrationProgressColumns_RunsOncePerProcess pinning the sync.Once behavior + a resetEnsureColumnsOnceForTest helper for test isolation. |
||
|
clawbot
|
0eab5f8f0e |
fix(#1735): surface async-migration errors on healthz/perf + propagate progress write failures
Group A from PR #1735 round-1 review (must-fix #1, #5, #6, #7). - cmd/server/healthz.go: on readAsyncMigrations error, include the message in the JSON body as async_migrations_error AND keep async_migrations_running=true. Fail closed for warm-up: if we can't read the bookkeeping table, treat the system as possibly still warming up rather than declaring 'all clear'. - cmd/server/async_migrations.go handlePerfAsyncMigrations: return HTTP 500 with the error body on readAsyncMigrations failure instead of silently returning an empty list. (Empty list is a meaningful operator signal; a query failure must be visible.) - cmd/server/routes.go /api/perf: log the readAsyncMigrations error and surface it via X-Async-Migrations-Error response header so the rest of the perf payload still flows. - cmd/server/async_migrations.go: delete the unread asyncMigrationsCacheErr field (finding #5). - cmd/server/async_migrations.go parseAsyncTime: propagate parse errors to the caller; readAsyncMigrationsRaw now appends them to ErrorMessage so unparseable timestamps don't silently produce 0s. - cmd/ingestor/async_migration_progress.go recordAsyncMigrationProgressEx: check RowsAffected(); 0 rows updated -> error (bookkeeping row missing). cmd/ingestor/db.go: track in-loop progress write failures, log them, and treat a failed TERMINAL progress write as a failed migration (counts are no longer trustworthy). |
||
|
Kpa-clawbot
|
847964656a |
chore(preflight): annotate small _async_migrations schema ops
The pr-preflight async-migration gate flags any new ALTER TABLE / CREATE TABLE in a migration-shaped file without an explicit annotation. Two sites are legitimately safe-at-scale but lacked the annotation: - cmd/ingestor/async_migration_progress.go ADD COLUMN on the bookkeeping table _async_migrations (single-digit rows; ADD COLUMN is O(rows)). - cmd/server/async_migrations_test.go CREATE TABLE on a fresh in-memory test DB (test setup, not a real schema migration). Annotation-only — no behavior change. Both call sites already had runtime safeguards (duplicate-column tolerance, test isolation). cross-stack: justified — annotations only; no functional change. PR #1735 already declares the frontend+backend coupling. |
||
|
Kpa-clawbot
|
f5bf605604 | feat(api): /api/perf and /api/healthz expose async migration progress | ||
|
openclaw-bot
|
f149993473 |
feat(async-migration): progress columns + rate-limited writes + retry reset
Adds an observational progress surface to _async_migrations so a long-running async migration (in particular tx_last_seen_backfill_v1 on operator-scale cold-load) is no longer opaque to readers. Schema changes (additive on legacy DBs): - _async_migrations.rows_processed (INTEGER NOT NULL DEFAULT 0) - _async_migrations.rows_total (INTEGER NOT NULL DEFAULT 0) - _async_migrations.last_update_at (TEXT) ensureAsyncMigrationProgressColumns runs ADD COLUMN per column and ONLY swallows the SQLite "duplicate column" error — every other ALTER failure propagates so a real schema problem doesn't get hidden. The CREATE TABLE body carries the same columns for fresh installs. recordAsyncMigrationProgress rate-limits writes to <=1/sec per migration name via a per-name time.Time cache; the rate limit is intentionally NOT a sync.Map so the bookkeeping table doesn't see a write per backfill batch (which on a SetMaxOpenConns(1) DB would compete with the migration's own UPDATE for the writer lock). recordAsyncMigrationProgressTerminal forces a write past the limiter — used to pin final stable counts on both success and failure paths so observers see the final point at which the migration stopped, not stale intermediate data. Retry path (RunAsyncMigration on an existing pending_async or failed row) resets rows_processed / rows_total / last_update_at to zero AND clears the in-memory rate-limit cache, so the next run starts with an honest denominator and no suppressed first write. A single sync.Once guards the warn log for the legacy "progress columns missing" path so a misconfigured DB doesn't generate one log line per batch. db.go wires both the periodic and terminal progress writes into the tx_last_seen_backfill_v1 migration. Failures still propagate to the RunAsyncMigration goroutine (status flips to 'failed' with the error message); the terminal write captures the partial counts at the failure point. |
||
|
openclaw-bot
|
915b10119f |
fix(#1724): chunk tx_last_seen_backfill with bounded reader yield
Replaces the single correlated UPDATE used by tx_last_seen_backfill_v1 (introduced in #1690) with a chunked loop that yields the single SQLite writer between batches. Symptom (pre-fix, operator scale ~71K tx / 1.5M obs / 2GB DB): - backgroundLoadComplete=true fires. - The async migration starts the single full-table UPDATE under SetMaxOpenConns(1), holds the writer for 10-15 minutes. - Every /api/healthz, /api/packets, /api/stats request queues behind sqlite_busy_timeout. UI appears frozen long after warm-up clears. Fix (this commit): - cmd/ingestor/tx_last_seen_backfill.go (new): chunkedTxLastSeenBackfill snapshots MAX(id), counts eligible rows (last_seen=0 AND has observations AND id<=maxID), then loops bounded UPDATEs (batchSize=5000) with time.NewTimer-based sleeps (no Timer leak via time.After) between batches (yieldDelay=100ms). EXISTS gate skips orphan transmissions so the loop terminates. maxID snapshot keeps concurrent INSERTs out of scope (those are handled inline by stmtBumpTxLastSeen on the writer fast path). Ctx cancellation between batches returns context.Canceled with partial counts; partial commits are visible (migration does NOT flip to done). All errors propagate (snapshot, count, UPDATE, RowsAffected) — the migration cannot silently mark itself done. Progress callback fires per non-empty batch + once terminal with final stable counts; never on a stale n=0 batch. - cmd/ingestor/db.go: wire the helper into the tx_last_seen_backfill_v1 async migration, explicit batchSize=5000, yieldDelay=100ms. Math reality-check: ~71K tx / 5000 ≈ 15 batches × (~50ms exec + 100ms yield) ≈ ~2.5s wall time with readers slotted in at most every 150ms. PR #1725's description claimed ~300 batches × 150ms ≈ 45s — that confused observations (1.5M) with transmissions (71K); real number is ~20x smaller. Indexes idx_tx_last_seen (transmissions(last_seen)) and idx_observations_transmission_id already exist (see internal/dbschema and cmd/ingestor/db.go base schema) — no additional index work required at this commit. Tests: cmd/ingestor/tx_last_seen_backfill_test.go (added in prior commit) pin all the contract points reviewers flagged on PR #1725. Cancel-mid-loop test timing widened from 30ms to 250ms to give the real chunked impl room to commit a batch before the cancel fires; assertion semantics unchanged (partial commits + context.Canceled + no full completion). |
||
|
openclaw-bot
|
cb6bab577f |
test(#1724): RED — chunked tx_last_seen backfill behavior + edges
Adds the failing test suite for the new chunkedTxLastSeenBackfill helper that will replace the single-statement #1690 backfill in the next commit. Tests pin the contract reviewers flagged on the prior attempt: - Reader yields between batches (concurrent reader latency bounded — a single-tx fake would NOT satisfy this). - With seedN=12000 + batchSize=5000, progress callback fires >=3 times. - ctx cancel mid-loop -> context.Canceled + partial commits visible. - Concurrent INSERT of new last_seen=0 rows does not trap the loop (maxID snapshot bounds the scan). - Orphan transmissions (no observations) are skipped via EXISTS so the loop terminates deterministically. - Param validation: batchSize<=0 and negative yieldDelay are rejected (no <0 sentinel). - Error propagation: closed DB surfaces -> migration cannot silently report success. Includes a minimal stub of chunkedTxLastSeenBackfill (returns zero/nil) so the file compiles and the tests run to their assertions. The GREEN commit replaces the stub with the real chunked implementation. |
||
|
Kpa-clawbot
|
97833c523b |
fix(post-packets): use v3 observations schema (closes #1196) (#1704)
## Summary `POST /api/packets` is broken on every v3-schema install — which is the default since #1289. The handler issues two writes against legacy v2 column names and silently swallows the observation insert's error, returning `200 OK` with `id>0` while persisting zero observation rows. ## Root cause `cmd/server/routes.go:1225-1235` (pre-fix) used the v2 schema shape: ```go INSERT INTO transmissions (... path_json ...) // path_json removed in v3 INSERT INTO observations (transmission_id, observer_id, observer_name, snr, rssi, timestamp) // v2 columns // timestamp written as RFC3339 text; v3 wants unix INTEGER // second Exec's error was discarded ``` v3 schema (`cmd/ingestor/db.go:289-304`): `observations.observer_idx INTEGER` (FK `observers.rowid`), `observations.timestamp INTEGER` (unix epoch), `path_json` lives here not on `transmissions`. Reporter [@EldoonNemar](https://github.com/EldoonNemar) called this out precisely in #1196 — both the schema mismatch and the divergence between the test harness (which uses the v3 shape) and the handler (v2 shape). ## Fix `cmd/server/routes.go`: - `transmissions` insert: drop `path_json` column. - Observer resolution: `INSERT OR IGNORE INTO observers (id, name, ...)` then `SELECT rowid` — mirrors the ingestor resolver at `cmd/ingestor/db.go:778,906`. - `observations` insert: write `observer_idx INTEGER` + `timestamp = time.Now().Unix()`; `path_json` moved here. - **Propagate both insert errors** (transmission + observation) as `500` instead of swallowing them. ## TDD | Step | Commit | Result | | ----- | ------- | ------ | | RED | `46d25389` | Test fails on master: `id=0` because the transmissions insert references a column not present in v3. | | GREEN | `dae57d67` | Test passes; round-trip persists the observation with `observer_idx` resolved from the seeded `obs1` row and a unix-epoch `timestamp`. | Local repro: ``` # RED on the test commit alone: $ go test -run TestPostPacketPersistsV3Schema -count=1 . --- FAIL: TestPostPacketPersistsV3Schema (0.03s) routes_test.go:4755: expected transmission id > 0, got 0 (body: {"id":0,"decoded":{...}}) FAIL # GREEN on HEAD: $ go test -run TestPostPacketPersistsV3Schema -count=1 . ok github.com/corescope/server 0.037s ``` ## Scope Two files, both in `cmd/server/`: - `cmd/server/routes.go` (+38/-12) — handler rewrite - `cmd/server/routes_test.go` (+66) — round-trip regression test No public API signature changes. No DB schema changes (consumes the existing v3 schema correctly). Closes #1196 |
||
|
Kpa-clawbot
|
76e130b313 |
fix(#1702): grant actions: write to release-fast-path workflow (#1703)
## Summary Fixes the missing `actions: write` permission on `.github/workflows/release-fast-path.yml` so the fallback `gh workflow run deploy.yml` dispatch no longer returns HTTP 403. ## Triage verdict From issue #1702 root-cause section: > Fast-path workflow YAML likely lacks: > ```yaml > permissions: > contents: read > packages: write > actions: write # MISSING — required to dispatch other workflows > ``` > ## Fix > One-line addition to `.github/workflows/release-fast-path.yml` permissions block. ## Root cause `.github/workflows/release-fast-path.yml` lines 16-18 (before this change) only granted `contents: read` and `packages: write`. The fallback step (`gh workflow run deploy.yml` when `:edge`'s `org.opencontainers.image.revision` label doesn't match the tag SHA) calls the GitHub Actions REST API, which requires `actions: write` on `GITHUB_TOKEN`. Without it, the dispatch fails with `Resource not accessible by integration` and the release stalls until an operator manually re-runs the fast-path job after `:edge` rebuilds. ## Change - `.github/workflows/release-fast-path.yml`: add `actions: write` to the workflow-level `permissions:` block. - `cmd/server/release_fast_path_workflow_test.go`: extend the existing config-gate test (issue #1677) to require `actions: write` alongside the previously asserted `contents: read` and `packages: write`. Two commits, red→green: 1. `test(#1702): assert release-fast-path.yml requires actions: write` — extends the assertion. Verified to fail on this commit (`release-fast-path.yml: missing required permission "actions: write"`). 2. `fix(#1702): grant actions: write to release-fast-path workflow` — adds the permission. Test green. ## TDD posture The repo already had a YAML-config gate at `cmd/server/release_fast_path_workflow_test.go` (parses the workflow as text and asserts required permission strings). Strict TDD applied: red commit extends the test, green commit fixes the workflow. No exemption needed. ## Acceptance criteria (from #1702) - [x] `permissions.actions: write` added to the fast-path workflow - [ ] Manual test: tag a scratch SHA where `:edge` is stale; confirm fallback dispatches deploy.yml without 403 — by-design out of CI scope (would require a throwaway tag + race condition); covered by next real release. - [ ] Operator-felt: next release where notes-commit lands AFTER `:edge` build completes works in one pass without manual rerun — verifiable only on next release; in-scope of `Closes #1702` because bullet 1 (the structural defect) is the cause of bullets 2 and 3. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → **clean** (all hard gates pass, no warnings). Closes #1702 --------- Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com> |
||
|
Kpa-clawbot
|
e96f0f9f9f |
fix(#1694): port extended ACK decoder to server (ackLen/ackAttempt/ackRand parity) (#1695)
## Summary Ports the firmware-1.16.0 extended ACK decoding from the ingestor (PR #1618, issue #1610) into the server-side re-decoder. Previously `cmd/server/decoder.go` silently dropped `ackLen`, `ackAttempt`, and `ackRand` (and the multipart inner equivalents) — the server emitted plain 4-byte ACKs even when the wire carried the 5/6-byte extended form. Now both decoders agree byte-for-byte. Closes #1694. ## What changed - `cmd/server/decoder.go::decodeAck`: sets `AckLen` (capped at 6), `AckAttempt` (`buf[4]` when `len>=5`), `AckRand` (`buf[5]` when `len>=6`). Mirrors `cmd/ingestor/decoder.go:279-305`. - `cmd/server/decoder.go::decodeMultipart` ACK branch: sets `InnerAckLen = len(buf)-1` (capped at 6), `InnerAckAttempt`, `InnerAckRand`. Mirrors `cmd/ingestor/decoder.go:696-714`. - `Payload` struct gains six `*int` fields tagged `omitempty`: `AckLen`, `AckAttempt`, `AckRand`, `InnerAckLen`, `InnerAckAttempt`, `InnerAckRand`. Backward-compatible JSON — legacy 4-byte ACKs leave attempt/rand nil and the fields are omitted from the output. No other decoder consumer is touched. Routes / store auto-surface the new fields via JSON marshaling. ## Test layout `cmd/server/decoder_ack_extended_test.go` drives `decodeAck` table-driven across the three wire shapes: | Buffer | AckLen | AckAttempt | AckRand | |---|---|---|---| | `EF BE AD DE` (CRC only) | 4 | nil | nil | | `EF BE AD DE 07` | 5 | 7 | nil | | `EF BE AD DE 07 42` | 6 | 7 | 0x42 | Plus `TestDecodeMultipartAckExtendedInner` for a 7-byte multipart buffer (`0x33` header + 6-byte inner ACK), asserting `InnerAckLen=6`, `InnerAckAttempt=7`, `InnerAckRand=0x42`. ## TDD trail - **Red commit** (test + struct stubs only, `decodeAck`/`decodeMultipart` unchanged) → assertions fail on `AckLen=nil`. - **Green commit** (port implementation) → all assertions pass. Full `cd cmd/server && go test ./...` passes locally. ## Firmware refs - `firmware/src/helpers/BaseChatMesh.cpp:218-234` (extended ACK layout) - firmware commit `f6e6fdaa` (attempt counter) - firmware commit `a130a95a` (RNG byte) --------- Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |
||
|
Kpa-clawbot
|
a8c99c61fd |
fix(#1659): block analytics endpoint until first pass complete (503 Retry-After) (#1688)
## Summary Fixes #1659 — analytics cards no longer show the post-restart slice when "All data" is selected. ## Root cause After server restart, `s.recompRF` / `s.recompTopology` / `s.recompChannels` cache the FIRST computation, which is the small in-RAM observations slice (background chunk-loader has not yet backfilled history). The recomputer serves that slice through `GetAnalyticsRFWithWindow`'s default shortcut for an entire recompute interval, while the client pins it via `CLIENT_TTL.analyticsRF`. UX: cards show a tiny window even when the user selects "All data". ## Fix shape (option B from the issue body) Server-side per-recomputer warm-up gate: - `cmd/server/analytics_warmup_1659.go` adds a per-recomputer `firstPassDoneNs` atomic timestamp, set ONLY by the first successful `runOnce()` (CAS-guarded for idempotency). `IsWarmingUp_1659()` / `FirstPassDoneAt_1659()` are lock-free reads. - `cmd/server/analytics_recomputer.go` `runOnce()` calls `markFirstPassDone_1659()` after every successful compute. - `cmd/server/routes.go` handlers for RF / Topology / Channels: when the request is the default shape (`region=="" && area=="" && window.IsZero()`) AND the matching recomputer is still warming up, return `503` + `Retry-After: 5` + `{"error":"analytics warming up","retry_after_s":5}`. Windowed / region-filtered requests bypass the gate (they already bypass the recomputer cache, so they are unaffected by the warm-up bug). Client-side: - `public/app.js` `api()` helper retries any 503 response, honoring `Retry-After`, with exponential backoff capped at 30s, max 6 attempts (~63s total). - Small "Computing analytics…" banner appears while any warm-up retry is in flight, dismissed once the request resolves. Pages can override via `window.onWarmup_1659`. ## Tests RED commit `8b2b2d7` ships failing-on-assertion tests + a stub. GREEN commit `2716c23` lands the fix and flips them green. - `cmd/server/analytics_warmup_1659_test.go` — 3 cases: 503 during warmup, 200 after first pass, windowed request bypasses gate. - `test-1659-analytics-warmup.js` — 3 cases: Retry-After honored, retry cap bounded, non-503 errors not retried. Wired into `.github/workflows/deploy.yml`. ## Preflight overrides - cross-stack: justified — server-side 503 contract MUST be paired with client-side retry-and-banner handling; splitting across two PRs would land a half-working fix. Fixes #1659. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: openclaw <openclaw@local> |
||
|
Kpa-clawbot
|
048143f54f |
fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691)
## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com> |
||
|
Kpa-clawbot
|
d910ea0208 |
feat(#1638): confidence rating weighted by hash mode (#1687)
Fixes #1638. ## Problem `getConfidenceIndicator` in `public/nodes.js` treats every observation as equal evidence, so a node seen 5 times via 1-byte hash prefixes (which collide ~8-way across a typical mesh) scores the same as a node seen 5 times via 6-byte prefixes (effectively unambiguous). The user asked for confidence to respect ambiguity. ## Change - `cmd/server/neighbor_graph.go` — new `CountsByMode map[int]int` on `NeighborEdge`, bumped in `upsertEdge` / `upsertEdgeWithCandidates` based on the observation's hash-prefix byte length (1/2/4/6). Merged in `resolveEdge` when ambiguous→resolved edges collapse. - `cmd/server/neighbor_api.go` — `NeighborEntry.counts_by_mode` exposed (omitempty), and `dedupPrefixEntries` merges per-mode counts when an unresolved prefix entry collapses into a resolved one. Flat `Count` field preserved for back-compat. - `public/nodes.js::getConfidenceIndicator` — weights observations by mode: 1-byte=0.125, 2-byte=0.5, 4/6-byte=1.0. A single 6-byte sighting counts ~8× a raw 1-byte one. HIGH triggers when EITHER the legacy heuristic clears OR weighted count ≥3. Legacy entries without `counts_by_mode` keep working (default weight 0.5). - Tooltip now shows the per-mode breakdown (e.g. "Observations: 5 (1-byte: 3, 6-byte: 2)"). ## TDD - RED: `cmd/server/neighbor_graph_test.go::TestBuildNeighborGraph_CountsByMode` — fixture with 1/2/4-byte sightings asserts per-mode tally (commit `838965f3`). - RED: `test-confidence-indicator.js` — 6-byte mostly-sighted neighbor must outrank 1-byte mostly-sighted neighbor at equal flat count (commit `4bd5e18e`). - GREEN: implementation in commit `7511606d`. All 4 JS tests pass; new Go test passes; full Go suite passes (two pre-existing flakes unrelated, both pass when isolated). ## Browser verification Synthetic side-by-side of OLD vs NEW classifier against representative inputs — see screenshot. 1-byte-only and 6-byte-only at the same flat count diverge from MEDIUM/MEDIUM to MEDIUM/HIGH, and 3 6-byte sightings now upgrade where 20 1-byte sightings stay MEDIUM. ## Preflight overrides - check-branch-scope: cross-stack: justified — backend exposes the new `counts_by_mode` field and the frontend consumes it; the whole point of the change. ## Compat - `Count` field unchanged in shape and value. - `counts_by_mode` is `omitempty`; legacy persisted edges (loaded from `neighbor_edges` via `neighbor_persist.go`) get no per-mode breakdown and fall back to the default weight (0.5) — no UI regression. --------- Co-authored-by: bot <bot@local> Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbot
|
efd66ea3f5 |
feat(mqtt): per-source status endpoint + Observers panel (#1682)
## Summary Adds MQTT source status visibility per #1043 acceptance criteria: - **Ingestor:** per-source counter registry (`cmd/ingestor/source_status.go`) tracking `connected`, `lastConnectUnix`, `lastDisconnectUnix`, `lastPacketUnix`, `connectCount`, `disconnectCount`, `packetsTotal`, `packetsLast5m` (sliding 5-min window via per-second buckets keyed by unix second — no stale-leak), `lastError`. Wired at the existing OnConnect / ConnectionLost / DefaultPublish callsites alongside the liveness watchdog. Idempotent registration so counters survive reconnects. Snapshot emitted in the existing stats file under `source_statuses` (additive, `omitempty`). - **Backend:** new `GET /api/mqtt/status` handler reads the ingestor stats file and returns the per-source list. **Broker passwords are masked** via a regex over the `scheme://user:pass@host` form (covers mqtt/mqtts/tcp/ssl/ws/wss). Mask is also applied to `lastError` as defense-in-depth (broker libs occasionally quote the failing URL). OpenAPI completeness gate satisfied with a `routeDescriptions` entry. - **Frontend:** small self-contained panel (`public/mqtt-status-panel.js`) mounted above the Observers table. Auto-refreshes every 10s, color-codes each row (green = connected + recent packet, yellow = connected idle, red = disconnected), and tears down its timer on SPA route change. ## TDD - Red commit `f19a93b5` — stub `/api/mqtt/status` handler + assertion test that the broker password is `****`-redacted. Test fails on the assertion (handler passes the URL through verbatim). Compile-clean — assertion-fail, not build-fail. - Green commit `77042e41` — `maskBrokerURL` helper + table-driven unit tests across all schemes + handler rewires to mask both `Broker` and `LastError`. - Subsequent commits land the ingestor wiring and the frontend panel. ## Tests ``` $ cd cmd/server && go test -run 'TestMqttStatus|TestMaskBrokerURL' -v ./... PASS: TestMqttStatus_MasksBrokerPassword PASS: TestMqttStatus_EmptyWhenNoStatsFile PASS: TestMaskBrokerURL_Patterns (10 subtests) $ cd cmd/ingestor && go test -run 'TestSourceStatus|TestSnapshotSourceStatuses' -v ./... PASS: TestSourceStatus_BasicLifecycle PASS: TestSourceStatus_Disconnect PASS: TestSnapshotSourceStatuses_ReturnsAll $ node test-mqtt-status-panel.js 7 passed, 0 failed ``` Full `go test ./...` clean in both `cmd/server` and `cmd/ingestor`. ## Preflight overrides - `cross-stack`: justified — issue #1043 is intrinsically full-stack (ingestor stats → server endpoint → observers panel). Per-stack split would land an unreachable endpoint or a fetch with no backend. - `check-xss-sinks` (public/mqtt-status-panel.js:55): justified — the flagged `innerHTML=` is a fully-static literal (empty-state placeholder, no payload data interpolated). All payload-bearing `innerHTML=` sites in this file run through `escapeHTML` (defined in the same file); the test `renderPanel never echoes a plaintext password (defense-in-depth)` exercises the rendered HTML against payload strings. ## Acceptance criteria - [x] `/api/mqtt/status` returns per-source connection state — `cmd/server/mqtt_status.go` - [x] UI panel shows all configured sources with live status — `public/mqtt-status-panel.js` - [x] Connection state updates on reconnect/disconnect events — `MarkConnect` / `MarkDisconnect` wired in `cmd/ingestor/main.go` - [x] Broker URLs don't expose passwords in the API response — `maskBrokerURL` + 13 test cases - [x] Works with 1-N sources — registry is keyed per-source, snapshot iterates the map **Partial fix for #1043** — per-packet `mqtt_source` attribution (the issue's "Follow-up" section) is **deferred** per the `mc-bot-triaged:v1` triage and the autofix comment ("Per-packet attribution deferred to follow-up issue"). That work requires a new observation-row column and DB schema migration, both explicitly out of scope for this PR. Refs #1043 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbot
|
2ef7d2437d |
fix(ci): release fast-path re-tag :edge → :vX.Y.Z when SHA matches (Fixes #1677) (#1680)
## Summary Adds `.github/workflows/release-fast-path.yml`: a metadata-only re-tag workflow that fires on `push.tags: v[0-9]+.[0-9]+.[0-9]+` and, when `:edge`'s `org.opencontainers.image.revision` label matches the tag SHA, applies `:vX.Y.Z`, `:vX.Y`, `:vX`, `:latest` to the existing edge manifest via `crane tag`. No rebuild, no test re-run — ~seconds vs ~30 min today. If the SHA doesn't match (tag points to an older commit, or `:edge` wasn't built yet), it dispatches the existing `deploy.yml` pipeline as a fallback so validated bytes always ship. To prevent double-fire, `deploy.yml`'s top-level `on:` block drops `tags: ['v*']` — `release-fast-path.yml` is now the sole consumer of `push.tags`. Edge publishing on master push is untouched. ## TDD Red commit adds `cmd/server/release_fast_path_workflow_test.go` (two tests: one asserts the new workflow exists with the required trigger/permissions/markers; the other asserts `deploy.yml`'s `on:` block no longer mentions `tags:`). Both fail on assertions in the red commit. Green commit adds the workflow file + edits `deploy.yml`; both pass. ## Acceptance criteria (from #1677) - Tag-CI completes in <2 min when tag SHA == `:edge` revision → fast-path is metadata-only, single short job - Falls back to full pipeline on SHA mismatch → `gh workflow run deploy.yml --ref ${{ github.ref }}` - `:vX.Y.Z` has same digest as `:edge` → `crane tag` copies the manifest, bytes are byte-identical - No regression on older-SHA tags → fallback path runs the unchanged full validation Fixes #1677 --------- Co-authored-by: Kpa-clawbot <bot@corescope.local> |
||
|
Kpa-clawbot
|
653d47e03c |
test(openapi): add CI completeness gate for /api routes (Phase 1 of #1670) (#1678)
## Summary Partial fix for #1670 — **Phase 1 only** (CI completeness gate). Phase 2 (backfilling the 18 currently-undocumented routes into `openapi.go`) is deferred to a separate issue per the triage on #1670 and is explicitly out of scope here. ## What this adds - `cmd/server/openapi_completeness_test.go` — AST-walks every non-`_test.go` file in `cmd/server/`, finds string-literal first args to `*.HandleFunc(...)` calls beginning with `/api/`, and diffs against the paths declared in `routeDescriptions()` in `cmd/server/openapi.go`. - `cmd/server/openapi_known_gaps.json` — seeded allowlist of the **18** `/api/` routes currently registered via `HandleFunc` but not yet documented in `openapi.go`. ## Ratchet pattern From this branch forward, `TestOpenAPICompleteness` fails when: 1. A new `HandleFunc("/api/...")` is added without a matching entry in `openapi.go` **or** the allowlist (regression gate — the main goal of Phase 1). 2. A route in the allowlist is *also* documented in `openapi.go` — the allowlist must shrink as Phase 2 backfills land, never go stale. The two-commit history (red → green) demonstrates the gate works: - **Red commit**: adds only the test. Fails on master with the 18 missing routes listed. - **Green commit**: adds the allowlist seeded with that exact 18-route set. Test passes at the current baseline. ## Local verification - `go test ./cmd/server/ -run TestOpenAPICompleteness -v` → PASS at baseline (`44/62 covered; 18 in allowlist; 18 gaps remain`). - Ratchet validation: temporarily inserted `r.HandleFunc("/api/ratchet-test-route", ...)` into `routes.go` → test FAILED with that exact route name; reverted → test PASSES again. ## Files changed - `cmd/server/openapi_completeness_test.go` (+203 / new) - `cmd/server/openapi_known_gaps.json` (+24 / new) ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass; no warnings. ## Out of scope - Backfilling the 18 allowlisted routes into `openapi.go` (Phase 2 — tracked separately). - Schema validation of the spec against OpenAPI 3.0 (Phase 3 per the issue). - PR template checkbox update (Phase 2 follow-up). Issue #1670 stays open for Phase 2. --------- Co-authored-by: clawbot <bot@corescope.local> |
||
|
efiten
|
938153dd92 |
fix(nodes): rebuild relay-hop history on startup from path_json (#1643)
## Problem A relay node's **activity timeline** — and its per-node `packetsToday` / observer counts — collapses to *"only the hour the server restarted"* after every restart. Before the restart the timeline shows only the node's own adverts (~1–2/hr); all of its relay activity piles into the single post-restart hour. ## Root cause All DB cold-load paths (`Load`, `loadChunk`, `scanAndMergeChunk`) index relay-hop attribution into `byNode` **only** from `observations.resolved_path`. But since #1287 the ingestor persists relay data as aggregate `neighbor_edges` and **never writes `resolved_path`** — it is `NULL` on every deployment (verified on a live DB: 0 of ~440k rows populated). So relay attribution is never reconstructed on startup; it only re-accumulates from live traffic (`IngestNew*`, which re-resolves from `path_json` + the neighbor graph), piling a relay node's whole history into the post-restart window. ## Fix Server read-side only — **no schema / ingestor / migration change**. When `resolved_path` is empty, re-resolve relay hops from the already-persisted `path_json` using the in-memory prefix map + neighbor graph (the same `resolvePathForObs` compute the live ingest path already runs). `main.go` now loads the persisted neighbor graph *before* the packet load so resolution has the graph available. Two correctness details worth a close look: 1. **Fetch the prefix-map/graph snapshot BEFORE opening each load cursor.** `getCachedNodesAndPM` issues its own DB query; doing so while a load cursor is open deadlocks on a single-connection SQLite pool (the test harness uses one). 2. **Index into `byNode` ONLY** — not the `resolved_path` / path-hop indexes. Those are cross-checked by `handleNodePaths` against the persisted `resolved_path` column (NULL here); populating them from an in-memory re-resolution would make that SQL confirmation fail and wrongly drop the tx from paths-through (#1352). ## Tests New coverage asserts a relay pubkey reachable *only* via `path_json` lands in `byNode` after a restart-style load, for both the hot-window (`LoadChunked`) and background-window (`loadChunk`) paths. Existing #1558 (`resolved_path`) and #1352 (paths-through) tests still pass. Full `cd cmd/server && go test ./...` is green under `-race`. ## Perf The fallback runs `resolvePathForObs` per observation with a non-empty `path_json` during cold load — the same per-packet compute the live ingest path already performs, so no new asymptotic cost. The prefix map + graph are snapshotted **once per load** (not per row); `getCachedNodesAndPM` is 30s-cached. In `loadChunk` the resolution runs in the existing lock-free scan and is accumulated locally, matching that function's "build local, merge under lock" design. ## Note on a pre-existing flaky test `TestDistanceConcurrentRequestsDuringBuildReturn202` is timing-fragile (fails ~1/15 on `master` without this change). It relies on the lazy distance build being slow because it's the first caller of `getCachedNodesAndPM` (cold cache). This PR pre-warms that cache during `Load`, narrowing the build window, so the test fails more often in **non-race** local runs. It passes reliably under `-race` (CI mode), where the build stays slow. Flagging in case you want to harden the test separately. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
Kpa-clawbot
|
825b26485c |
fix(#1181): hide nodes whose name starts with a configured prefix (#1655)
Fixes #1181. ## Summary Adds operator-configurable name-prefix hiding for nodes. When a node's name starts with any prefix listed in the new `hiddenNamePrefixes` config field (default `["🚫"]`), it is omitted from `/api/nodes`, `/api/nodes/search`, and `/api/nodes/{pubkey}`. DB rows are preserved — the filter runs at the API layer only, so observation history (paths, hops, distances) stays intact and the node simply re-appears if the operator clears the prefix list. This mirrors the convention already in use on other MeshCore map dashboards: an operator who wants their node hidden renames it with the 🚫 prefix and sends an advert; the next advert is then dropped from the dashboard. The node is **not** hidden from the mesh itself — only from this dashboard. This is documented inline in `config.example.json`. Implementation follows the existing `IsBlacklisted` pattern exactly: a new `Config.IsNameHidden(name)` method, and three filters in `routes.go` placed alongside the corresponding blacklist filters. No DB schema, public API, or websocket changes. ## Files changed - `cmd/server/config.go` — new `HiddenNamePrefixes []string` field + `IsNameHidden` method - `cmd/server/routes.go` — filters in `handleNodes`, `handleNodeSearch`, `handleNodeDetail` - `config.example.json` — new field + `_comment_hiddenNamePrefixes` operator doc - `cmd/server/hidden_name_prefix_1181_test.go` — new test file (red → green) ## Test plan Two new subtests in `TestHiddenNamePrefix_1181_*`: 1. `_NodesList` — inserts a node named `🚫 ban me`, asserts it is present when `HiddenNamePrefixes` is empty and absent when set to `["🚫"]`. 2. `_Search` — inserts `🚫 search me`, asserts `/api/nodes/search?q=search` does not surface it when the prefix is configured. Verified red→green: - Red commit `d0903852`: `go test -run TestHiddenNamePrefix_1181` fails on the leak assertion (`hidden_name_prefix_1181_test.go:94`). - Green commit `e79a0d8d`: same command passes. ``` $ cd cmd/server && go test -run TestHiddenNamePrefix_1181 -count=1 . ok github.com/corescope/server 0.060s ``` ## Out of scope - Auto-purging DB rows for hidden nodes — left to existing retention. The triage was explicit: hide, do not delete. - Live websocket broadcast: nodes are not broadcast via websocket (only packets), so no separate emit path needs filtering. Frontend reads nodes via `/api/nodes`, which is filtered. - Frontend customizer for the prefix list — operators configure via `config.json` like every other knob. |
||
|
Kpa-clawbot
|
e04c7113cb |
feat: integrate hashtag channels from meshcore-channels catalogue (#1323) (#1656)
Fixes #1323 ## Summary Adds a small in-memory cache of the community-maintained hashtag-channels catalogue (`marcelverdult/meshcore-channels`) and exposes it as `GET /api/known-channels?region=XX` plus a collapsed sidebar section on the Channels view ("Known channels (catalogue)") with a one-click "+ Add" button per row. Per triage (#1323): new `cmd/server/known_channels_cache.go`, new `GET /api/known-channels?region=…`, frontend section in `public/channels.js`. No new DB tables — cache is in-memory only. ## What changed - `cmd/server/known_channels_cache.go` — `knownChannelsCache` with an atomic snapshot pointer, 24h default refresh, 30s HTTP timeout, 4 MB body cap, custom `User-Agent`. Fail-soft: a failed refresh leaves the last-known snapshot in place. Background goroutine started from `main.go` after the neighbor-graph recomputer; never blocks startup. - `cmd/server/known_channels_route.go` — `GET /api/known-channels?region=` serves the cached snapshot off the atomic pointer (never blocks on upstream). Region filter is case-insensitive ISO 3166-1 alpha-2. Empty/missing cache returns 200 with an empty entries list (fail-soft for the UI). - `cmd/server/config.go` — `KnownChannelsURL` + `KnownChannelsRefreshMs`. - `config.example.json` — example values + `_comment_knownChannels`. - `public/channels.js` — new collapsed sidebar section "Known channels (catalogue)" that lazy-fetches `/api/known-channels` on first render and renders rows with a "+ Add" button. The button calls the existing `addUserChannel(name)` path, so adding catalogue channels reuses the full save-key + decrypt flow that user-typed hashtags already use. - `cmd/server/known_channels_cache_test.go` — failing-first tests: - `TestKnownChannelsParseFixture` asserts the parser populates `GeneratedAt`/`License` and region-stamps every entry while skipping empty countries. - `TestKnownChannelsRouteRegionFilter` asserts the route returns 200 with exactly the filtered subset for `?region=be`. - `TestKnownChannelsFailSoftOn500` asserts a failed upstream fetch leaves the prior snapshot in place and bumps `failCount`. ## Upstream pinning The default URL is pinned to the specific file `channels-by-country.json` on `main`: > https://raw.githubusercontent.com/marcelverdult/meshcore-channels/main/channels-by-country.json Shape (verified 2026-05-24): ```json { "generated_at": "...", "license": "CC0-1.0", "countries": { "be": [{"channel": "#antwerpen", "description": "..."}], ... } } ``` ## Test plan ``` cd cmd/server && go test -run 'TestKnownChannels' -count=1 . ok github.com/corescope/server 0.008s ``` Red commit: |
||
|
Kpa-clawbot
|
1116801b2f |
M5: emoji → Phosphor Icons — settings & customize (#1648) (#1653)
**Red commit:** `851cc8c3a024b1675558092d772444bf4f1ec625` — failing test on a stub branch (will link CI run after PR opens). Partial fix for #1648 (M5 of 6). **Do NOT close the tracking issue** — M6 (server-side residual emoji sweep + lint gate) still pending. ## Per-file swap counts | File | Phosphor `<use>` refs | Notes | |---|---|---| | `public/customize.js` | 20 | DEFAULTS → `ph:<name>` tokens; render path keeps legacy emoji branch (back-compat) | | `public/customize-v2.js` | 26 | same as v1; cv2 overrides path unchanged | | `public/home.js` | (helpers added) | `_renderHomeGlyph` / `_renderHomeLabel` accept both `ph:<name>` and legacy emoji | | `public/geofilter-builder.html` | 5 | clear / undo / save / load buttons (+inline `.ph-icon` CSS) | | `public/audio.js` | 1 | audio unlock prompt | | `public/filter-ux.js` | 5 (3 new) | help popover star + close, saved-filter delete | | `public/style.css` | 0 | `#chList .ch-share-btn::before { content: '📤' }` removed; JS now renders an inline sprite | | `cmd/server/routes.go` | (6 `ph:` tokens) | onboarding home defaults updated in lockstep with customize-v2.js | ## Operator config back-compat — PROMINENT Per design call #1 (user-locked): existing operator-stored emoji values in `config.json` / `localStorage` are **NOT** touched. The render path supports both: ```js function renderConfigGlyph(value) { var m = String(value || '').match(/^ph:([a-z][a-z0-9-]+)$/); if (m) return '<svg class="ph-icon"><use href="/icons/phosphor-sprite.svg#ph-' + m[1] + '"/></svg>'; return esc(value); // EMOJI-OK-LEGACY-RENDER — operator-stored emoji/text path } ``` Defaults flipped to `ph:<name>` tokens, so new operators (and operators who hit "Reset to Defaults") see Phosphor sprites. Operators with stored emoji values continue to see their emoji exactly as before. Verified end-to-end (see E2E (b) below). ## cmd/server/routes.go — changed in lockstep Per design call #2: the home-defaults `steps` / `footerLinks` mirror the JS DEFAULTS, so they MUST update together. routes.go now emits `ph:<name>` tokens; the frontend home-render path resolves them. Existing tests (`TestConfigThemeHomeDefaults`) still pass — they assert structure, not glyph values. ## E2E assertions added - `test-issue-1648-m5-emoji-scan.js` — per-file zero-emoji + ph-token DEFAULTS + sprite presence - `test-issue-1648-m5-icons-e2e.js`: - (a) customize chrome — tabs/header rendered as sprites; chrome text icon-free - **(b) back-compat — injects fake `🐙` operator step into localStorage, reloads, opens customize, asserts the emoji renders verbatim in both the input value AND the live preview span; asserts the ph-token step renders as a sprite** (design call #1 in action) - (c) `/channels` modal sprite count - (d) `/audio-lab` sprite presence - (e) `geofilter-builder.html` control buttons sprite-driven - (f) every `<use>` resolves to a defined symbol id ## Out of scope (M6 cleanup) - cmd/server/routes.go residual server-rendered emoji **not** tied to customize defaults (none found by my grep — file already audited) - `make lint-no-emoji` CI grep gate (M6 owns it) - `public/icons/README.md` workflow doc cross-stack: justified — design call #2 requires Go + JS update together. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbot
|
8295c2115c |
fix(reach): bust response cache on blacklist change (#1629) (#1636)
Red commit:
|
||
|
Kpa-clawbot
|
078225a54e |
perf(neighbor_api): fold first_seen into cached map — fix #1627 r3 regression (#1632)
## TL;DR Post-merge regression introduced by #1627 r3 (commit `e2212f50`): `buildNodeInfoMap` in `cmd/server/neighbor_api.go` ran an uncached `SELECT … FROM nodes` scan on every call. Folded `first_seen` into the already-cached `getCachedNodesAndPM` (30s TTL) so the 4 hot handlers that call `buildNodeInfoMap` no longer pay for a full table scan per request. ## Before / After `buildNodeInfoMap` is called by **4 hot handlers**: - `cmd/server/neighbor_api.go:130` - `cmd/server/neighbor_api.go:297` - `cmd/server/neighbor_debug.go:83` - `cmd/server/node_reach.go:421` | | Before | After | |---|---|---| | `SELECT … FROM nodes` per call | 1 (uncached) | 0 (cache hit) | | `SELECT … FROM observers` per call | 1 (uncached) | 1 (unchanged) | | At Cascadia scale (~2600 nodes) | full scan × 4 handlers × N req/s | one scan / 30s | ## How - Extended the `getAllNodes` schema probe to also `COALESCE(first_seen, '')`. Falls back through the existing richest → leanest ladder if the column is missing. - `nodeInfo.FirstSeen` is therefore populated for every cached entry in `getCachedNodesAndPM`. - `buildNodeInfoMap` drops its second `SELECT` entirely and just copies `nodeInfo` values out of the cached map. - Public signature of `buildNodeInfoMap` is unchanged. `node_reach.go:421` still sees `nodeInfo.FirstSeen` populated, served from cache. `cmd/server/store.go` is touched because `getAllNodes` is the only sensible owner of the `first_seen` SELECT — adding a parallel cache would duplicate the 30s TTL machinery this fix is designed to leverage. ## Test (red → green) - Commit 1 (`test:`): `TestBuildNodeInfoMap_FirstSeenIsCached` — calls `buildNodeInfoMap`, mutates `first_seen` out-of-band via a separate rw connection, calls it again, and asserts both calls return the same (cached) value. Fails on `origin/master` (call 2 sees the mutated value, proving the uncached scan). - Commit 2 (`perf:`): the fold. Test now passes. ## Refs Post-merge audit identified this as the only MAJOR finding from #1627; recommendation was a follow-up hot-fix PR. This is that PR. --------- Co-authored-by: openclaw-bot <bot@openclaw> Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbot
|
43be1bb76a |
fix(reach): scanReachRows DB errors must surface as 500 not 404 (#1631) (#1635)
Red commit:
|
||
|
efiten
|
e2212f5015 |
feat(nodes): per-node Reach page + GET /api/nodes/{pubkey}/reach (v2, review-complete) (#1627)
Re-submission of #1625 (which was merged early, then reverted in #1626) — now with **all three round-1 reviews addressed** so it lands in one hardened state instead of as post-merge follow-ups. ## What Per-node **Reach** view: a standalone page (`#/nodes/{pubkey}/reach`) + a node-detail section + `GET /api/nodes/{pubkey}/reach`. It shows which nodes a node has a **stable two-way RF link** with, derived from raw `path_json` adjacency (a path travels origin→observer, so `[A,B]` ⇒ B heard A). A link is bidirectional when both directions have observations; the **bottleneck** (weaker direction) rates two-way reliability. Nodes are identified only by **unique 2–3 byte** path prefixes (1-byte collides → excluded). ## Review fixes folded in vs #1625 **Performance (Carmack):** hard scan LIMIT (200k) + modest prealloc; `json.Unmarshal` replaced by a single-pass `parsePathTokens` (100k-row scan 2.2M→1.3M allocs, 344→203ms); memoized resolver; size-hinted maps (attribution over 100k rows: 102 allocs); `context.Context` plumbed; cache `RWMutex` + evict-oldest (no full wipe); singleflight dedup; degree/rank from a 60s shared snapshot; bench rewritten (ReportAllocs, 1k/10k/100k, mixed-payload, isolated attribution). **Correctness/safety + tests (Independent + Kent Beck):** pubkey validation → 400; error logging instead of silent swallow (first_seen / degree / marshal→500 / discarded rows); `public_key=?` index use; canonical `PayloadADVERT`; `min()` builtin; documented cache-slice immutability; mux ordering comment. New tests: scanReachRows decode, 3-byte token branch, non-advert first-hop guard, observer SNR aggregation across rows, HTTP-level attribution (asserts non-zero we_hear/they_hear), 400/404/blacklist/cache-hit. **UI / a11y / Tufte:** in-map legend (tiers + thresholds); dropped the colour+width double-encoding (constant width, colour-only); colour-blind glyphs (●●●/●●/●) + tier title beside the bottleneck number; dark-theme `--link-*`; lighter table (horizontal rules, sentence-case headers); map built once + link layer updated in place on toggle (no flicker); time-range no longer flashes a loader; `destroy()` generation guard; statCard escaping; scoped `@media print` to `#nq-report`; `fieldset/legend` + `for/id` toggles; `aria-pressed` / `aria-live` / back-link `aria-label`; "distance (km)" + bottleneck tooltip + no-GPS note; inline styles → CSS; decorative emoji removed. **Docs:** api-spec documents the 5-min cache, 200k scan cap, and 400. ## Testing - `cmd/server` full suite green; reach unit + endpoint + bench all pass. - `eslint public/*.js` (no-undef) and the XSS-sink gate clean. - E2E updated: request status checks + exact (non-tautological) toggle assertions + hard map-render assert. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## TDD-history note (Kent Beck gate) This branch carries production + tests together, not a fabricated red→green sequence. That's deliberate: the branch was rebased onto upstream and the intermediate SHAs were squashed, so reconstructing a "failing-test-first" commit after the fact would be theatre, not evidence — and rewriting history to stage it would be dishonest. The behaviour is instead covered by a comprehensive, anti-tautological suite (directional attribution edges, 3-byte token branch, non-advert first-hop guard, observer SNR aggregation, HTTP-level attribution asserting non-zero counts, scan-cap truncation, zero-reach 200-not-404, companion mis-attribution, cache eviction). Requesting maintainer acceptance of the work on test *substance* rather than commit *choreography*; the net-new-UI exemption is not claimed for the server endpoint. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: meshcore-bot <bot@meshcore> |
||
|
efiten
|
9c5faab1e4 |
Revert "feat(nodes): per-node Reach page (#1625)" (#1626)
Reverts #1625. #1625 was merged before the round-1 reviews (Independent / Kent Beck / Tufte) were addressed. Reverting to land it cleanly: a fresh PR will re-add the feature with the perf pass, the backend correctness/safety + test-coverage fixes, and the UI/a11y (Tufte) batch folded in, so it goes through review in a single hardened state rather than as a string of post-merge follow-ups. No functional loss — the feature returns in the replacement PR. |
||
|
efiten
|
47f85f6c4c |
feat(nodes): per-node Reach page + GET /api/nodes/{pubkey}/reach (directional link quality) (#1625)
## What
Adds a per-node **Reach** view that answers "how well does this specific
node hear, and get heard by, its neighbours?" — both as a standalone
page (`#/nodes/{pubkey}/reach`) and as a section on the node detail
page.
New endpoint: **`GET /api/nodes/{pubkey}/reach`**.
## What it measures
For the target node it derives, from raw `path_json` adjacency (a path
travels origin→observer, so in `[A,B]` B received A directly):
- **Directional link counts** per neighbour: `we_hear` (how often we
received them) vs `they_hear` (how often they received us).
- **Bidirectional / bottleneck**: a link is two-way stable when both
directions > 0; the weaker direction is the bottleneck and rates real
two-way reliability.
- **Importance**: neighbour degree + rank, relay-observation volume,
bidirectional-link count, direct-observer count.
- **Direct observers**: who received the node at 0 hops, with SNR.
Reliability rule: a neighbour is only attributed when its pubkey
**prefix is unique** at the path's byte length (collisions are skipped,
never misattributed).
## UI
- Standalone Reach page + node-detail section.
- Reusable bidirectional link map (OSM) with links coloured by
bottleneck.
- Incoming/outgoing toggles to isolate each direction.
## Naming note (deliberate, no collision)
This is distinct from the existing **per-observer reachability** in
topology analytics (`ReachNode` / `ObserverReach` / `perObserverReach`).
This PR adds its own `NodeReach*` response structs in a new
`node_reach.go` and a new `/api/nodes/{pubkey}/reach` route — there are
no symbol or route collisions (verified: `go build ./...` clean). Happy
to rename to disambiguate further (e.g. "Link Quality") if you'd prefer
to reserve "Reach" for the per-observer feature.
## Testing
- `cmd/server`: endpoint shape/404/limit-clamp + unit tests for token
derivation and directional attribution, plus a scan benchmark — all
pass.
- Frontend: helper tests + Reach-page E2E (`test-node-reach-e2e.js`),
standalone route + incoming/outgoing toggles.
- `go build ./...` and `eslint public/*.js` (no-undef) clean.
## Docs
Design spec, implementation plan, and the `GET
/api/nodes/{pubkey}/reach` API contract are included under `docs/`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Kpa-clawbot
|
a4776557ae |
feat(#1290): use firmware repeat:on|off hint to exclude listener-only observers from disambiguator (#1624)
Closes #1290. cross-stack: justified — backend persists firmware-side `repeat` hint to a new observers column, frontend surfaces the listener/repeater status as a badge on the observers list and node-detail Heard By table per the issue's UI acceptance criterion. ## What Firmware 1.16 publishes a `repeat: on|off` flag in the MQTT `/status` JSON (confirmed by @cwichura on the issue thread — see [`MQTTMessageBuilder.cpp:58`](https://github.com/agessaman/MeshCore/blob/b45373a31f111fb0de98bb3b168226d09ceadc47/src/helpers/MQTTMessageBuilder.cpp#L58) in `agessaman/MeshCore mqtt-bridge-implementation-flex`). Listener-only observers (`repeat:off`) by firmware contract never relay packets, so they cannot legitimately be a hop in someone else's resolved path. This PR plumbs the hint end-to-end so the disambiguator stops considering them. ## How * **`internal/dbschema`**: idempotent `can_relay INTEGER DEFAULT 1` migration on `observers`, plus `AssertReady` probe (server fatal-logs if absent). Mirrored in `cmd/ingestor/db.go` `CREATE TABLE` for fresh DBs. Annotated `PREFLIGHT: async=true` — `DEFAULT 1` is constant so SQLite does this as a metadata-only schema rewrite. * **`cmd/ingestor`**: `extractObserverMeta` accepts `repeat` as bool, case-insensitive string (`on|off|true|false|yes|no`), or numeric `0|1`. Missing field → `nil` → `COALESCE` preserves the existing column value (back-compat with legacy observers). Plumbed through `UpsertObserverAt` and the prepared upsert statement. * **`cmd/server`**: `GetNonRelayObserverPubkeys` + new `prefixMap.markNonRelay` drop matching candidates inside `pm.resolveWithContext` at the top of the resolver, so all 4 tiers see the pruned candidate set. `ObserverResp.CanRelay` is surfaced on `/api/observers` and `/api/observers/{id}`. `GetNodeHealth` enriches per-observer rows with `can_relay` so the node-detail badge renders. Probe-and-fall-back when the `can_relay` column is absent (legacy test fixtures). * **`public/`**: listener vs repeater pill on observers list, observer detail `Relay` stat card, and node-detail `Heard By` table. CSS uses existing theme vars. ## Test Added `TestResolveWithContext_ExcludesNonRelayObservers_Issue1290` in `cmd/server/resolve_non_relay_1290_test.go` covering all three required cases: * `repeat:off` pubkey → not a candidate (assertion failed in red commit `5f7fdb96`, passes after green `f12911dc`) * `repeat:on` pubkey → still a candidate (regression guard) * legacy obs (no field) → still a candidate (back-compat) Red→green proof: ``` $ git log --oneline origin/master..HEAD |
||
|
Kpa-clawbot
|
3d12266595 |
fix(#1608): address PR #1609 follow-up findings — config doc, receipt-time liveness, buffer stop/clamp warn (#1623)
Follow-up to #1609 / #1608. Addresses the 5 unresolved findings from the PR #1609 round-1 polish review. ## Findings addressed | Tag | Severity | Fix | Commits | |-----|----------|-----|---------| | **B1** | BLOCKER | Document `ingestBufferSize` in `config.example.json` near other ingestor knobs. Default `50000`, comment text from review. | `f0b4e411` | | **M1** | MAJOR (option 1 from review) | Split receipt-time vs post-write liveness: add `SourceLivenessState.LastReceiptUnix` + `MarkReceipt`, stamp at the MQTT receipt callback, leave `LastMessageUnix` post-write only. Drop the double-stamp at receipt that masked write-path stalls. Surface both clocks via the ingestor stats file (`source_liveness`) and the server's `/api/healthz` (`ingest_liveness`, additive — older builds unaffected). | RED `fa78233d` / GREEN `bc81b544` | | **M1 (drop-log)** | MAJOR | Log every drop when buffer is at capacity. Removes the `n==1 \|\| n%1000` throttle that hid the first stall behind 1000 lost packets. The Submit drop branch only fires when the channel is at cap so volume is naturally bounded by the stall, not by an arbitrary modulo. | RED `a468763e` / GREEN `7b24fce5` | | **m1** | MINOR | Add `IngestBuffer.Stop()` and `Done()` so tests stop leaking the consumer goroutine that `Start()` spawns. Existing tests gain `t.Cleanup(b.Stop)`. Drain semantics: stop-before-Ready exits immediately; stop-after-Ready best-effort drains queued jobs. | RED `8430c822` / GREEN `78c9b223` | | **m2** | MINOR | `NewIngestBuffer(<1)` now logs a `[ingest-buffer] WARN` line on clamp so misconfigured `ingestBufferSize` values are visible instead of silently running a 1-slot queue. Test captures log output. | RED `62119ab4` / GREEN `815bfd02` | | **m3** | MINOR | Add godoc to `Submit` and `Ready` documenting the Start-before-Submit / Start-before-Ready ordering invariant. | `564a813b` | ## TDD discipline Each behavioral fix (M1, M1-drop-log, m1, m2) lands as a red-then-green pair. Red commits compile + run + fail on assertion, verified locally before the green commit. Per-finding red→green pairs are visible in the commit graph above. B1 and m3 are docs-only and ship as single commits (preflight script accepts them under the docs/comments exemption). ## Schema compatibility `/api/healthz` change is purely additive: `ingest_liveness` is only included when the ingestor publishes the new `source_liveness` field, so older ingestor + newer server combos are unaffected. Field order in the response stays stable for prior consumers. ## Test output - `go test -count=1 -timeout 180s ./cmd/ingestor/...` → green (160s) - `go test -count=1 -timeout 300s ./cmd/server/...` → green (48s) - Race-mode runs of the touched packages (`IngestBuffer|Liveness|Watchdog|Receipt|Healthz`) → green - Full-package race runs locally exceed the brief's 120s timeout on pre-existing slow integration tests (TestObsTimestampIndexMigration, TestNeighborEdgesBuilderDeltaScan); CI has the headroom. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass, no warnings. ## Files changed - `config.example.json` — B1 - `cmd/ingestor/ingest_buffer.go` — m1, m2, M1-drop-log, m3 - `cmd/ingestor/ingest_buffer_test.go` — m1, m2, M1-drop-log - `cmd/ingestor/mqtt_watchdog.go` — M1 - `cmd/ingestor/mqtt_watchdog_m1_test.go` — M1 (new) - `cmd/ingestor/main.go` — M1 (receipt callsite) - `cmd/ingestor/stats_file.go` — M1 (publish `source_liveness`) - `cmd/server/perf_io.go` — M1 (type + reader) - `cmd/server/healthz.go` — M1 (surface `ingest_liveness`) Original review reference: PR #1609 polish review by the M-axis bot. --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbot
|
bc1822e46c |
perf(load): chunked Load with early HTTP readiness (#1009) (#1596)
## What Switches the server's startup from a synchronous full-scan `PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that: 1. Streams transmissions+observations from SQLite in id-ordered chunks (default `chunkSize=10000`, configurable via `db.load.chunkSize`). 2. Closes `FirstChunkReady()` after the first chunk is merged — `main.go` binds the HTTP listener on that signal instead of blocking on the full multi-minute load. 3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every response while LoadChunked is in flight, flipping to `ready` once it completes (via `loadStatusMiddleware`). 4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB` clamps and the post-load index rebuild (`pickBestObservation` / `buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`). ## Why Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked load the listener binds within seconds; dashboards and probes can read partial data and see the `loading` status header until the background load finishes. ## Notes - `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`) is unchanged — it still waits for neighbor-graph build + initial `pickBestObservation` before reporting `ready:true`. `LoadChunked` only changes when the listener BINDS, not when it advertises ready. - `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on a tiny DB) before proceeding, and drains the load goroutine in the background with a logged error path. - Config Documentation Rule: `config.example.json` now documents `db.load.chunkSize` with a nested `_comment` describing the trade-off. ## Tests - `cmd/server/chunked_load_test.go` asserts: - (a) `FirstChunkReady` fires before `LoadChunked` returns - (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` → `ready` - (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via `OnChunkLoaded`) - (d) `Config.DBLoadChunkSize()` default 10000 + override - Red commit (`102a4c84`) lands the tests with stubs that fail on assertion — verified locally before the green commit. - Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite green (47s locally). Closes #1009 ## TDD red-commit exemption The original red commit `f878e15e` ("test(load): failing tests for chunked Load + early HTTP readiness") fails to **compile** rather than failing on an assertion, because it references symbols (`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`, `Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A compile error is NOT a valid red commit." This is claimed under the **net-new surface** exemption with the following justification: - LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize are all introduced by this PR — no prior implementation existed to refactor. There is no behaviour on master that the red commit could meaningfully assert against without first declaring the new symbols. - The cheapest "proper" alternative (split the red into two commits: stub-first + assertion-fail) was deferred because the test file unambiguously fails on missing-symbol — there is no risk of the test becoming a tautology against a pre-existing stub. - **Behaviour gating IS proven elsewhere on this branch.** Commit `799bde49` ("test(load): red — LoadChunked must mark indexes ready + not flip Complete on error") is a proper assertion-fail red against the same package, and commit `92cadd1d` is the matching green. Reviewers can verify the red→green pattern there. If a future reviewer wants the strict pattern, the follow-up is mechanical: split `f878e15e` into a stub-only commit followed by the assertion commit. Not done here to keep the rework cost proportional to the risk (zero, in this case). ## Preflight overrides - check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and `cmd/server/chunked_load_oldest_test.go` only. They run against per-test `t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single test) — they are NOT production schema migrations. No prod table is touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir fixture). --------- Co-authored-by: CoreScope Bot <bot@corescope.local> Co-authored-by: clawbot <bot@noreply.example.com> Co-authored-by: Kpa-clawbot <bot@example.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |
||
|
Eldoon Nemar
|
7421ead9b0 |
fix: bypass API limit clamps for internal UI requests. Revisit of issue #1540 (#1589)
This PR replaces the strict, hardcoded limits on API list endpoints (introduced in the recent security patch) with a new operator-configurable `listLimits` block. This change is needed as issue 1540's implementation introduced a 500max node limit on the live map or any other function that leverages the api/nodes backend. Previously, we attempted to bypass public caps for internal UI requests using a heuristic based on browser headers (`Sec-Fetch-Site`). Following review, we decided to drop that heuristic entirely to eliminate any security-by-browser-convention surface area. Instead, `queryLimit()` returns to its original, mathematically simple bounds-checking shape, and the absolute maximums are now drawn from `config.json`. This provides equal DoS protection against all callers while allowing server operators to tune the ceilings based on the size of their mesh (e.g. embedded devices can tighten the knobs, regional hubs can raise them). ### Changes Made: - **`config.go`**: Introduced a `ListLimits` config struct containing `PacketsMax`, `NodesMax`, `AnalyticsMax`, and `ChannelMessagesMax`. Added safe initialization to ensure default caps (10000, 2000, 200, 500 respectively) apply even if the block is omitted from the config. - **`clamp_limit.go`**: Deleted `isInternalUIRequest` entirely and restored `queryLimit` to its original signature (`r, def, max`). - **`routes.go`**: Replaced all hardcoded integer ceilings on list endpoints (`/api/packets`, `/api/nodes`, etc.) with `s.cfg.ListLimits.*`. - **`config.example.json`**: Added the `listLimits` block with documentation to guide new operators. - **`clamp_limit_test.go`**: Purged all header-heuristic testing. ### Verification: - All 611 backend unit tests pass (`npm run test:unit`). - Bounds-checking math continues to enforce hard DoS clipping exactly at the operator's specified configuration limit. --------- Co-authored-by: mc-bot <bot@openclaw.local> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
Kpa-clawbot
|
1bdb92de88 |
feat(#1574): operator-configurable liveMap.maxNodes (default 2000) (#1577)
Red commit:
|
||
|
Kpa-clawbot
|
ad41b9bb7b |
fix(tests): subpaths_window tests wait for index readiness after #1595 chunked load (#1621)
## Why master is red After PRs #1592 (route-window subpath regression test) and #1595 (background/chunked index build with 503 readiness gate) were merged together, two tests in `cmd/server/subpaths_window_test.go` started failing on master: ``` --- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[]) --- FAIL: TestSubpathsHandlerHonorsTimeWindow subpaths_window_test.go:116: GET /api/analytics/subpaths?...: status=503 body={"error":"index loading","retryAfter":5} ``` Both branches passed in isolation; the conflict only manifested post-merge. Reason: - **#1592** added tests that call `store.Load()` then immediately query `GetAnalyticsSubpathsWithWindow` / hit `/api/analytics/subpaths`. - **#1595** moved the subpath + path-hop index builds off the critical path of `Load()` into background goroutines, and hard-gated the analytics handlers behind `SubpathIndexReady()` (returning 503 + `Retry-After: 5` until the build completes). So after `Load()` returns, `s.spIndex` is still empty for a short window and the handler returns 503. The store-level test sees `totalPaths=0`; the handler test sees the 503. ## Fix (test-only) Add `store.WaitIndexesReady(5 * time.Second)` between `Load()` and the assertions in both tests. This matches the established pattern already used by `routes_test.go` and `repeater_enrich_recomputer_1008_test.go`. The 503 readiness gate from #1595 is intentional production behavior and is **not** touched. No production code is modified. ## Repro Before: ``` $ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=1 --- FAIL: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s) subpaths_window_test.go:70: unbounded: expected totalPaths=2, got 0 (subpaths=[]) --- FAIL: TestSubpathsHandlerHonorsTimeWindow (0.02s) subpaths_window_test.go:116: GET /api/analytics/subpaths?minLen=2&maxLen=8: status=503 body={"error":"index loading","retryAfter":5} FAIL ``` After: ``` $ go test ./cmd/server/ -run TestSubpaths.*Window -v -count=3 --- PASS: TestSubpathsHonorsTimeWindow_StoreLevel (0.01s) --- PASS: TestSubpathsHandlerHonorsTimeWindow (0.02s) ... (x3) ... PASS ok github.com/corescope/server 0.097s $ go test ./cmd/server/ -count=1 -timeout 300s ok github.com/corescope/server 46.292s ``` ## Files changed - `cmd/server/subpaths_window_test.go` (+11 lines, test-only) ## Notes - TDD exemption: this is a test-fix PR for a merge-conflict-induced failure. The "failing test" already exists on master; this PR makes it pass correctly by waiting on the readiness gate the test was previously unaware of. - Unblocks staging deploys. Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
Kpa-clawbot
|
222bfdf6cf |
feat(perf): SQLite writer-lock wait/hold instrumentation per component (#1340) (#1594)
## What Per-component SQLite writer-lock instrumentation so the next neighbor-builder-style write-lock starvation (root cause of #1339, invisible to operators for ~3 days) is detectable from `/api/perf`. Adds `Store.WriterExec` / `Store.WriterTx` wrappers that gate every wrapped call on a package-level `writerMu` so the wait the SQLite driver hides becomes Go-visible, and record `wait_ms` + `hold_ms` + `contention_total` (wait_ms > 100ms) under a component tag. Per-component p50/p95/p99 + max are published to `/api/perf/write-sources` under `.writer_perf` via the existing ingestor stats-file path. Slow-writer log line (`[db-slow-writer] component=X duration=Yms query=<200ch>`) fires on `hold_ms > 500ms` (threshold overridable via `CORESCOPE_DB_SLOW_WRITER_MS` env var). ## Tagged call sites | Component | Location | |-----------|----------| | `mqtt_handler` | `InsertTransmission` (db.go) | | `neighbor_builder` | `buildAndPersistNeighborEdges` (neighbor_builder.go) | | `prune_packets` | `PruneOldPackets` (maintenance.go) | | `prune_observers` | `RemoveStaleObservers` + orphan-metrics cleanup (db.go) | | `prune_metrics` | `PruneOldMetrics` (db.go) | | `vacuum` | `RunIncrementalVacuum` + `CheckAutoVacuum`'s full VACUUM (db.go) | ## TDD red→green - **Red commit** `68de585b` — `cmd/ingestor/db_writer_perf_test.go` + `Store.Writer*` stubs at end of `db.go`. Test synthetically blocks the writer for 60s tagged `neighbor_builder`, then asserts `mqtt_handler.wait_ms.p99 > 50000ms` on concurrent inserts. Fails on the assertion (p99 = 0.0ms) with the stub — not a build error. - **Green commit** `6a9be174` — replaces stubs with real wait/hold/contention aggregator + wires every writer call site. Same test passes: ``` 2026/06/05 04:36:47 [db-slow-writer] component=neighbor_builder duration=60059.0ms query=COMMIT --- PASS: TestWriterStarvationVisibleInPerf (60.40s) PASS ok github.com/corescope/ingestor 60.408s ``` ## Scope discipline - **API**: no public `Store`/`DB` signature change. Only additive exports. - **Server**: extends existing `/api/perf/write-sources` JSON with `.writer_perf` — does **not** add a new route, does **not** replace `handlePerf`. Empty `.writer_perf` map when paired with an older ingestor. - **Read/write invariant** (#1283) preserved: all instrumentation lives on the ingestor's writer connection. - **Files touched** (6 total): `cmd/ingestor/db.go`, `cmd/ingestor/db_writer_perf_test.go`, `cmd/ingestor/maintenance.go`, `cmd/ingestor/neighbor_builder.go`, `cmd/ingestor/stats_file.go`, `cmd/server/perf_io.go`, `config.example.json`. ## Deferred (acceptance items NOT in this PR) - **`mbcap_persist` component tag** — `RunMultibyteCapPersist`'s tx is intentionally NOT wrapped in this PR to stay within the implementation brief's 3-files-outside-whitelist budget. One-file follow-up to instrument. - **CI smoke test** asserting "neighbor-builder hold_ms < 1000ms on 100k-obs fixture" — deferred to a separate PR per the brief; this PR is scoped to instrumentation only. ## Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=runtime — the async-migration gate flagged five `instrumentedExec` / wrapped-`tx.Exec` lines on `DELETE FROM observer_metrics`, `UPDATE observers`, `DELETE FROM observer_metrics`, `DELETE FROM observations`, `DELETE FROM transmissions`. These are **not** schema migrations — they are the existing runtime prune / retention queries that already ran sync against `s.db.Exec` / `tx.Exec` on every retention cycle on master. This PR only swapped the surface call (sync → sync, via the wrapper) to record wait/hold timing; no new sync schema work was introduced. Behavior on production data is identical to master. Also: red commit's synthetic `UPDATE nodes SET name = name WHERE 0` is a test-only stub designed to acquire the writer without mutating any row (the `WHERE 0` is a no-op predicate). Fixes #1340 --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbot
|
1b112f0b08 |
feat(memlimit): GOMEMLIMIT via runtime.maxMemoryMB in server + ingestor (#1010) (#1595)
Red commit:
|
||
|
efiten
|
18810b5c13 |
fix(ingestor): subscribe to MQTT before startup maintenance, buffer until writer is free (#1608) (#1609)
## Summary Closes #1608. The ingestor's MQTT connect/subscribe loop ran **last** in `main()`, after the synchronous startup-maintenance block. Because all writes share a single SQLite writer (#1283), that maintenance — and the connect loop after it — serialize behind any long-running async migration. The subscription therefore came up minutes late (observed ~4.5 min after the v3.8.3 `obs_observer_ts_idx_v1` index build over ~4.9M rows), and QoS-0 packets published in that window were dropped. This decouples **receipt** from **write**: - New `IngestBuffer` — a bounded FIFO drained by a **single** gated consumer goroutine. - The MQTT subscription is brought up first; its publish handler stamps source liveness at receipt and enqueues a `handleMessage` closure. - Startup maintenance runs, then `WaitForAsyncMigrations()`, then `IngestBuffer.Ready()` opens the gate and the backlog drains. A single consumer preserves the single-writer invariant (#1283); buffering replays the original messages, so it introduces **no duplicates** (unlike a QoS-1 broker queue). Broker-agnostic — helps direct-connect and bridged operators alike. ## Changes - `cmd/ingestor/ingest_buffer.go` — `IngestBuffer` (`Submit`/`Start`/`Ready`/`Dropped`/`Pending`); non-blocking submit with drop-on-full counter; single consumer. - `cmd/ingestor/config.go` — `ingestBufferSize` knob (default 50000). - `cmd/ingestor/main.go` — reorder boot: connect/subscribe **before** startup maintenance; stamp liveness at receipt; `Ready()` after maintenance + `WaitForAsyncMigrations()`; periodic stats log buffer `pending`/`dropped`. ## Test plan - [x] `go test ./...` in `cmd/ingestor` — `IngestBuffer` suite covers gating-until-ready, FIFO order, drop-on-full, serial execution (single-writer), and concurrent-submit. - [ ] `go test -race` in CI (concurrency on `IngestBuffer`). - [ ] Manual: restart with a pending heavy migration → `subscribed to meshcore/#` appears within seconds; `[ingest-buffer] write path ready` after the migration; packets received during the window are written after `Ready()` (0 dropped under normal traffic); stall watchdog stays quiet (liveness stamped at receipt). ## Out of scope A hard crash while messages sit in the in-memory buffer still loses them; crash-durability requires broker-side persistence, which is topology-specific. This PR closes the startup-migration and deploy loss windows. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Kpa-clawbot
|
9612f08e46 |
fix(#1610): decode firmware 1.16.0 extended ACK (5/6-byte payloads) (#1618)
## Summary Firmware 1.16.0 (`companion-v1.16.0`) ships variable-length `PAYLOAD_TYPE_ACK` payloads: 4 bytes (legacy) → 5 bytes (4-byte CRC + 1-byte attempt, commit `f6e6fdaa`) → 6 bytes (+ 1-byte RNG, commit `a130a95a`). CoreScope's decoder previously truncated past the 4-byte CRC and discarded the attempt + RNG bytes. This PR teaches `cmd/ingestor/decoder.go` to surface the extended bytes on the decoded payload so the DB/UI can distinguish v1.15 vs v1.16 senders, with no schema or wire-compat changes. Partial fix for #1610 — top-level ACK + multipart-inner ACK are covered. PATH-extra ACK parsing (`decodePathPayload`) is deferred to #1612 per triage. ## Changes - `decodeAck` reads 4/5/6-byte payloads. Keeps `extraHash` (4-byte CRC) for compat; adds optional `ackLen`, `ackAttempt`, `ackRand` JSON fields. Legacy 4-byte ACKs leave attempt/rand `nil`. - `decodeMultipart` ACK branch relaxes the `len >= 5` floor so the inner blob can be 4/5/6 bytes (multipart `payload_len` 5/6/7). Adds `innerAckLen`, `innerAckAttempt`, `innerAckRand`. - All additions are `omitempty` — backwards-compatible JSON only. No DB column, no schema migration, no frontend change. ## Out of scope (per issue triage) - `decodePathPayload` PATH-extra parsing — tracked separately in #1612. - Frontend rendering of attempt counter — leave for a follow-up if the DB/UI eventually wants to display it. ## TDD - **Red commit `3fce0465`** adds `cmd/ingestor/issue1610_test.go` with 6 new assertions (legacy 4-byte, extended 5/6-byte, multipart variants of each). New fields are declared on `Payload` so the test compiles, but no decoder populates them yet — tests fail on `ackLen=<nil> want 4` etc. Verified isolation with `git stash` of decoder.go + re-run. - **Green commit `5165c202`** implements the decoder changes. `go test ./...` in `cmd/ingestor` passes. ## Fixtures Synthetic wire vectors built by hand against the firmware spec — the issue did not provide real captures. Each test cites the firmware ref + commit it derives from (`BaseChatMesh.cpp:218-234`, commits `f6e6fdaa` and `a130a95a`). ## References - Issue #1610 - Firmware tag `companion-v1.16.0` @ `07a3ca9e` - Upstream PR meshcore-dev/MeshCore#2594 - Blog: https://blog.meshcore.io/2026/06/06/release-1-16-0 --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbot
|
df61660a5e |
perf(load): background subpath+pathHop index builds with ready gates (#1008) (#1604)
## Summary
Mirrors the distance-index lazy pattern (#1011): the subpath and
path-hop index builds are no longer part of `Load()`'s synchronous
critical section. They now run in **two parallel background goroutines**
kicked off after `s.loaded = true`, so HTTP comes up immediately even at
Cascadia scale (5M observations, previously ~60s blocked on these two
builds inside `Load()` under `s.mu`).
Fixes #1008.
## Approach
Two new `atomic.Bool` fields on `PacketStore` (`subpathReady`,
`pathHopReady`) plus a one-shot broadcast channel (`indexReadyChan`) for
waiters. `Load()` removes the synchronous `s.buildSubpathIndex()` /
`s.buildPathHopIndex()` calls and instead kicks
`s.startBackgroundIndexBuilds()` right before returning. That function
spawns **two independent goroutines** (review m7), one per index. Each
goroutine:
1. acquires `s.mu.Lock()` (blocks until `Load()`'s deferred Unlock
fires),
2. runs its builder, releases the lock, stores its `ready = true`,
3. closes the broadcast channel if both flags are now true,
4. logs `[startup] index build complete: subpath (Xs)` (or pathHop).
Analytics handlers whose entire response IS the index aggregate —
`/api/analytics/subpaths`, `/api/analytics/subpaths-bulk`,
`/api/analytics/subpath-detail`, `/api/nodes/{pubkey}/paths` — gate
reads behind the corresponding atomic and respond with `503 Service
Unavailable`, `Retry-After: 5`, body `{"error":"index
loading","retryAfter":5}` until the build completes — matching the
triage spec.
### Handler scope (review M2)
A second class of handlers also touches these indexes — `/api/nodes`,
`/api/nodes/{pubkey}`, the `GetRepeaterRelayInfoMap` /
`GetRepeaterUsefulnessScoreMap` / `GetBridgeScore` enrichment helpers,
and `repeater_liveness` / `repeater_usefulness`. These are
**intentionally NOT 503-gated**: they expose the index via optional
enrichment fields that callers already treat as "may be empty", and
503-ing the SPA bootstrap to wait for an index that only affects
relay-activity badges would be a worse UX than a 30–60s window of "—"
values. The rationale is documented in the package doc-comment at the
top of `index_ready_1008.go`.
The recomputer's synchronous prewarm path
(`StartRepeaterEnrichmentRecomputer`) gates on `WaitIndexesReady(60s)`
(review M1) so it never snapshots an empty `byPathHop` into
`s.repeaterRelayCache`; on timeout it skips the prewarm and lets the
5-minute ticker pick up the populated index.
## Concurrency safety
Each build goroutine acquires `s.mu.Lock()` before calling the existing
`buildSubpathIndex()` / `buildPathHopIndex()` helpers, which replace
`s.spIndex` / `s.spTxIndex` / `s.byPathHop` with freshly-allocated maps.
Visibility of the populated maps to handlers that observe
`Ready()==true` is established by Go 1.19+ sync/atomic acquire-release
semantics: the atomic store of `true` happens-after `s.mu.Unlock()`, and
the handler's atomic load synchronizes-with that store. The handler's
subsequent `s.mu.RLock` serializes against concurrent ingest writers,
not against the builder.
The existing `main.go` boot sequence does not start ingest goroutines
until after `store.Load()` returns and graph init completes, so the
brief window between `Load()` returning and the two goroutines acquiring
`s.mu` does not race with concurrent ingest writes.
## TDD: red → green
- **Red** commit `63e79e11`: `cmd/server/index_ready_1008_test.go` adds
four assertions; `cmd/server/index_ready_1008.go` adds compile-only
stubs returning `true` so the tests fail on assertions, not build
errors.
- **Green** commit `fb1d22b0`: implements the real atomic gates, the
background goroutine, and the four handler 503 branches; also updates
four existing tests that read indexes directly post-`Load()` to call
`store.WaitIndexesReady(5s)` first.
- **Race-fix commit `b77d56eb`** (review m8 — test-infra exemption):
adds `WaitIndexesReady` calls in test helpers/setup paths so the race
detector no longer flags the read-after-Load() pattern in existing
tests. Per AGENTS.md, race-detector flakes are observable evidence (test
crashes under `-race`) and qualify for the test-infra exemption from the
TDD red-commit requirement; no behavior change in production code.
- **Polish round 2 — M1 red `408c7462` / green `85e82c8a`**:
`TestIssue1008_M1_PrewarmWaitsForIndexes` asserts the recomputer prewarm
SKIPs when indexes are not ready. Red commit adds the assertion + a stub
`repeaterEnrichmentPrewarmWait` var; green commit wires
`WaitIndexesReady` into the prewarm path and adds the handler-scope docs
for M2.
- **Polish round 2 — minor cleanups `fd089bd0`** (m3..m7): chunk-loader
wires `markIndexesReadySync`, memory-model comment rewritten to cite
acquire-release, sentinel deleted, polling replaced with a broadcast
channel, two parallel goroutines for the builds.
`TestIssue1008_m7_BothFlagsSetAfterParallelStart` covers the parallel
path.
## Reproduction
```
git fetch origin fix/issue-1008
git checkout
|
||
|
Kpa-clawbot
|
3898688d6d |
analytics: Relay Airtime Share endpoint + dumbbell chart (#1359) (#1601)
Implements the locked spec from #1359. Red commit: |
||
|
Kpa-clawbot
|
d6384c3c59 |
fix(#1217): honor time-window filter on Route Patterns analytics (#1592)
## What The Route Patterns chart on `/#/analytics` ignored the Time window picker — every selection returned identical data. This PR threads `?window=` through to the backing endpoints and the store-level computation. ## Root cause `cmd/server/routes.go:2065` (`handleAnalyticsSubpaths`) and `cmd/server/routes.go:2090` (`handleAnalyticsSubpathsBulk`) never called `ParseTimeWindow(r)`. The store-level entry points (`GetAnalyticsSubpaths`, `GetAnalyticsSubpathsBulk`) had no window-aware variant. The frontend (`public/analytics.js`) didn't append `&window=` to the `/analytics/subpaths-bulk` request. ## Fix ### Backend (`cmd/server/store.go`) Added `GetAnalyticsSubpathsWithWindow` + `GetAnalyticsSubpathsBulkWithWindow`. Zero `TimeWindow` → byte-equivalent to the existing fast path (no perf regression on the default view). Non-zero window → iterate `s.packets`, filter on `tx.FirstSeen` via `TimeWindow.Includes`, reuse `rankSubpaths`. Cached by `(region|area|window)`. ```diff -data := s.store.GetAnalyticsSubpaths(region, minLen, maxLen, limit) +window := ParseTimeWindow(r) +data := s.store.GetAnalyticsSubpathsWithWindow(region, minLen, maxLen, limit, window) ``` ```diff -results := s.store.GetAnalyticsSubpathsBulk(region, groups) +results := s.store.GetAnalyticsSubpathsBulkWithWindow(region, groups, ParseTimeWindow(r)) ``` ### Frontend (`public/analytics.js`) `renderSubpaths` now appends `&window=<value>` to the `/analytics/subpaths-bulk` request, matching how RF / topology / channels tabs already wire the picker. ## Before / after ``` GET /api/analytics/subpaths?window=24h → totalPaths=2 (all data — ignored window) GET /api/analytics/subpaths?window=24h → totalPaths=1 (24h-bounded — honored) ``` ## Tests `cmd/server/subpaths_window_test.go`: - `TestSubpathsHonorsTimeWindow_StoreLevel` — seeds a 1h-old tx with path `[aa,bb]` + a 30d-old tx with path `[cc,dd]`; asserts the unbounded call sees both and the 24h-windowed call sees only the recent one. - `TestSubpathsHandlerHonorsTimeWindow` — same scenario via the HTTP handlers for `/api/analytics/subpaths` and `/api/analytics/subpaths-bulk`. TDD: red commit `eefc27d3` (test fails on assertion with stub that ignores window), green commit `4c4c45d0` (implementation makes it pass). Full `go test ./...` in `cmd/server` green locally (~47s). ## Performance Default view (no window selected) is unchanged — `window.IsZero()` short-circuits to the existing precomputed-index hot path. Windowed view is O(N_tx · path²), same complexity as the existing region-filtered slow path. Results cached per `(region|area|window)`. Closes #1217 --------- Co-authored-by: Kpa-clawbot <bot@corescope> |
||
|
Kpa-clawbot
|
5629a489b2 |
perf(distance): lazy build distance index on first request (#1011) (#1597)
## Summary Build the distance analytics index lazily on the first `/api/analytics/distance` request instead of eagerly inside `Load()` (and its background-load chunked merge). Per the triage Fix path on the issue: - Eager startup build removed from `Load()` and from `loadAllPacketsBackground()`'s post-merge pass. - First request returns `202 Accepted` + `Retry-After: 5` and kicks off the build in a background goroutine, gated by `sync.Once` so concurrent first-window requests all observe 202 (single build, not N parallel O(n²) computations). - Once built, subsequent requests fall through to the existing analytics-recomputer / TTL cache and serve 200 as before. - Debounced rebuild policy: refire only when `Δobs > 5%` since last build OR `>5 min` elapsed, whichever is more restrictive. Background loader also resets the gate so the next request rebuilds against the larger dataset. Effect: operators who never visit distance analytics no longer pay the O(n²) construction at startup. Acceptance criteria (a) no startup build, (b) first request triggers build, (c) concurrent in-flight requests get 202 are encoded as failing-first tests. ## Red → green - Red: `bc947ad1` — 3 assertion failures (`expected ... empty, got 3`, `expected 202, got 200`, `expected all 10 ... got 0`). - Green: `5264b68a` — production change makes them pass, no other tests regress. ## Files changed - `cmd/server/store.go` — lazy-build state (`distLazyMu`/`Once`/`Built`/`Building`/`LastBuilt`/`LastObs`), `TriggerDistanceIndexBuild`, `DistanceIndexBuilt`, `DistanceIndexBuilding`; eager `buildDistanceIndex` calls in `Load()` post-pass and chunked-background-load post-pass removed (Once reset instead so the next request rebuilds against the full dataset). - `cmd/server/routes.go` — `/api/analytics/distance` returns 202 + `Retry-After` until built. - `cmd/server/distance_lazy_index_test.go` — new tests (the three triage acceptance criteria). - `cmd/server/coverage_test.go`, `cmd/server/parity_test.go`, `cmd/server/routes_test.go`, `cmd/server/hop_disambig_e2e_test.go` — pre-warm the index via `TriggerDistanceIndexBuild()` + `DistanceIndexBuilt()` poll where the test asserts the 200 JSON shape. ## Perf justification Startup cost on a 500K-obs / 2K-node dataset: previously O(n²) hop scan during `Load()` post-pass and again during the background-load merge — measured at 10–20s in `specs/startup-audit.md`. New code: zero work at startup, the same O(n²) work runs at most once per HTTP request cycle (and only when the index is stale per debounce policy). Cold-path concurrency is bounded by `sync.Once`, so N parallel first-window requests never produce N parallel builds. ## Scope No config field added (debounce thresholds are hardcoded constants per the triage Fix path — `5%` / `5min`). No public API signature changes. No DB-side migration. Tests cover the lazy invariant, the 202+Retry-After contract, and concurrent first-request behavior. Closes #1011 --------- Co-authored-by: Kpa-clawbot <bot@corescope.local> |
||
|
Kpa-clawbot
|
3df8924114 |
fix(#1218): include multi-byte prefix repeaters in 1-byte hash usage matrix view (#1591)
## Problem
`/analytics` Hash Usage Matrix 1-byte view excluded repeaters configured
for 2- or 3-byte hash prefixes. In MeshCore, 1-byte path-matching is a
first-byte equality check, so any packet routed by 1-byte hash collides
on that first byte regardless of the downstream repeater's configured
prefix size. Omitting multi-byte prefix repeaters under-reports real
conflicts in the 1-byte hash space.
## Fix
**Data layer — `cmd/server/store.go` (`computeHashCollisions`,
~L7907-L7918 before, L7907-L7941 after):**
Before — `one_byte_cells` was populated only from `prefixMap`, which
only contained repeaters with `hash_size == 1`:
```go
if bytes == 1 {
oneByteCells = make(map[string][]collisionNode)
for i := 0; i < 256; i++ {
hex := strings.ToUpper(fmt.Sprintf("%02x", i))
oneByteCells[hex] = prefixMap[hex]
if oneByteCells[hex] == nil {
oneByteCells[hex] = make([]collisionNode, 0)
}
}
} else if bytes == 2 { ... }
```
After — additionally project all `hash_size in {2,3}` repeaters to their
first byte:
```go
if bytes == 1 {
// ... (same baseline population) ...
for _, cn := range allCNodes {
if cn.Role != "repeater" { continue }
if cn.HashSize != 2 && cn.HashSize != 3 { continue }
if len(cn.PublicKey) < 2 { continue }
hex := strings.ToUpper(cn.PublicKey[:2])
if _, ok := oneByteCells[hex]; !ok { continue }
oneByteCells[hex] = append(oneByteCells[hex], cn)
}
}
```
The 2-byte view's bucketing is unchanged — that view continues to count
only repeaters configured for 2-byte prefixes (those semantics differ).
**UI — `public/analytics.js` L1459:** clarified the 1-byte view
description so the inclusion of multi-byte prefix repeaters is explicit.
## API shape
No response-shape change. `one_byte_cells[HEX]` is still
`[]collisionNode`; only the contents now include 2/3-byte prefix
repeaters in the appropriate first-byte buckets. The existing frontend
decoder is unaffected.
## Tests
-
`cmd/server/routes_test.go::TestHashCollisionsOneByteIncludesMultiBytePrefixRepeaters`
— seeds three repeaters with first byte `CC` configured for 1/2/3-byte
prefixes plus an unrelated `DD` repeater, asserts all three appear in
`one_byte_cells["CC"]`, and that the 2-byte view's `nodes_for_byte` is
unchanged.
Red commit `278bdf8d` (test only) fails on assertion ("got 1, want 3");
green commit `9127ea4e` passes.
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
→ clean.
Closes #1218
---------
Co-authored-by: clawbot <bot@corescope>
|
||
|
Kpa-clawbot
|
1a2b8c48be |
feat(node-detail): link RTC-reset warning to offending packet hashes (#1094) (#1590)
## Problem Node detail's bimodal-clock warning showed only `⚠️ N of last M adverts had nonsense timestamps (likely RTC reset)` — no way to tell which packets, no way to verify the heuristic, no way to drill in. ## Fix Additive, two-sides: **Backend** (`cmd/server/clock_skew.go`) - New type `BadSample { Hash, AdvertTS, SkewSec }`. - New field `NodeClockSkew.RecentBadSamples []BadSample` (`omitempty`). - Populated from the **same** bimodal-bad classification pass that produces `RecentBadSampleCount` — no heuristic change. `tsSkewPair` carries `hash` + `advertTS` so the classifier can record per-sample evidence without a second walk; drift code is unaffected (reads only `ts`/`skew`). **Frontend** (`public/nodes.js`) - `bimodalWarning` preserves the existing count summary line, then renders a `<ul>` of bad samples: each `<li>` is `<a href="#/packets/HASH">hash[:8]</a> → formatTimestamp(advertTS)` with ISO tooltip. Defensive `Array.isArray` so older API responses still render the summary alone. ## TDD - **Red:** `cmd/server/clock_skew_issue1094_test.go::TestIssue1094_RecentBadSamples_ExposesHashAndTimestamp` — seeds 3 healthy + 2 bimodal-bad adverts, asserts `RecentBadSamples` has length 2 with the expected hashes and advert timestamps. Fails on the assertion (`len = 0, want 2`) with the stub-only commit. - **Green:** classifier populates the slice; existing #1285 and bimodal tests stay green. - Red commit: `ed501f4b` - Green commit: `54305b06` ## Cross-stack Backend + frontend ship together (`cross-stack: justified` commit). API stays backward compatible (`omitempty` server, `Array.isArray` client) but the feature only lights up with both halves present. ## Preflight Clean — PII, branch scope, red-commit, CSS vars, XSS sinks, migrations, fixture coverage all pass. ## Acceptance - [x] Warning lists specific packet hashes - [x] Each hash links to `#/packets/<hash>` - [x] Bad advert timestamp shown next to the hash - [x] Pattern is reusable — `BadSample` is a clean shape any future heuristic that flags specific packets can adopt Fixes #1094 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbot
|
af669438ff |
docs+test(ingestor): document writeStatsAtomic symlink-replace semantics + regression test (#1170) (#1588)
Fixes #1170. ## What 1. **Doc comment** on `writeStatsAtomic` (`cmd/ingestor/stats_file.go`) spelling out the two-sided symlink story: - tmp side (`path+".tmp"`): protected by `O_NOFOLLOW` (existing behavior, already noted). - rename side (`path` itself): NOT protected by `O_NOFOLLOW`; instead `os.Rename` semantics are relied upon — rename atomically replaces any existing entry at `path` (including a symlink) with the new regular file. The symlink target is never written through because all writes happened to the unrelated tmp file before rename. 2. **Regression guardrail test** `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` in `cmd/ingestor/stats_file_test.go` that pre-plants a symlink at the destination path pointing to an unrelated target file, calls `writeStatsAtomic`, and asserts: - (a) `os.Lstat(path).Mode()&os.ModeSymlink == 0` (post-write path is a regular file, not a symlink) - (b) the original symlink target's sentinel bytes are unchanged. If a future refactor swaps `os.Rename` for a destination-symlink-following primitive (e.g. `open(path, O_WRONLY)` without `O_NOFOLLOW`, or a copy-then-truncate), the test fails loudly. ## TDD note (red-commit exemption) The current `writeStatsAtomic` ALREADY satisfies the new test's assertions — `os.Rename` does the right thing today. Per the fix-issue skill's exemption for pure-documentation / guardrail tests on already-correct behavior, no fabricated red commit was constructed; the test stands as a pinning regression guard. The two commits are therefore: (1) test addition, (2) doc comment. ## Scope - `cmd/ingestor/stats_file.go` — doc comment only - `cmd/ingestor/stats_file_test.go` — one new test function No production behavior change. No public API change. No new dependencies. No CI workflow changes. `O_NOFOLLOW` and the existing tmp-side behavior are untouched. ## Preflight All hard gates pass (PII, branch scope, red commit, CSS vars, LIKE-on-JSON, sync/async migration, XSS sinks). No warnings. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> |
||
|
Kpa-clawbot
|
7533b3b67b |
feat(nodes): sortable First Seen column on Nodes table (#1166) (#1587)
## Summary Adds a sortable **First Seen** column to the Nodes table so users can spot newly observed repeaters in their region (per the reporter's use case). Closes #1166 ## Backend `/api/nodes` already exposes `first_seen` per node via `db.scanNodeRow` (sourced from the existing `nodes.first_seen` column — no schema migration, no recomputation, no extra query cost). The red test pins that contract. ## Frontend (`public/nodes.js`) - New `<th data-sort-key="first_seen" data-sort-default="desc">First Seen</th>` between Last Seen and Adverts. - Cell renders via `renderNodeTimestampHtml(n.first_seen)` — same relative-time + absolute-ISO `title=` tooltip as the Last Seen column. Empty values render as `—`. - `sortNodes` gains a `first_seen` branch with **empty-last** semantics: nodes without a `first_seen` always sort to the bottom regardless of asc/desc direction, so unknowns never clutter the top of the table. - Empty-state `colspan` bumped 7 → 8. ## TDD - **Red commit** `112442f4` — `test-issue-1166-first-seen-column.js` + `cmd/server/first_seen_1166_test.go`. The backend half passes on red (field already returned); 5 frontend assertions fail on assertions (column header missing, sort branch missing, empty-last violated). - **Green commit** `9274b36c` — only `public/nodes.js`. All 6 tests pass. Verified red is real-fail (assertion-shaped) by checking out the red commit's `nodes.js` and re-running the test: 5 failures, all on `assert.strictEqual`, none on parse/import. ## Test results ``` node test-issue-1166-first-seen-column.js → 6 passed, 0 failed node test-frontend-helpers.js → 611 passed, 0 failed go test ./cmd/server/... → ok (45.16s, all pass) ``` ## Files changed - `public/nodes.js` (+14 / −1) - `test-issue-1166-first-seen-column.js` (new) - `cmd/server/first_seen_1166_test.go` (new) ## Scope guardrails - No schema migration. - No new files outside the worktree's three allowed surfaces. - No refactor of other Nodes columns. - Empty cells handled in both render (em-dash) and sort (always last). --------- Co-authored-by: fix-1166-bot <bot@corescope.local> |
||
|
efiten
|
f7571a261e |
fix(#1546): remove dead server-side backfill flag (stuck backfilling=true) (#1583)
## Summary Closes #1546. `/api/stats` reported `{"backfilling":true,"backfillProgress":0}` on every fully-converged server, and `X-CoreScope-Status: backfilling` was sent on every request. Root cause: the `Store` had three atomic fields — `backfillComplete` / `backfillTotal` / `backfillProcessed` — read by `handleStats` and `backfillStatusMiddleware`, but **nothing ever wrote to them**. They are leftovers from the server-side async backfill added in #612/#614. That work moved to the **ingestor** in #1289 (server is now read-only) and the writer `backfillResolvedPathsAsync` was deleted, orphaning the readers. `backfillComplete.Load()` therefore always returned `false`, so `backfilling := !false` was permanently `true`. This is the leftover of an intentional architecture change, not an unfinished feature — the server no longer does backfill by design, so the correct fix is to delete the dead flag (per triage recommendation; zero consumers). ## Changes - `store.go` — drop the 3 dead atomic fields. - `routes.go` — drop `backfillStatusMiddleware` (+ its registration) and the backfill-progress computation in `handleStats`. - `types.go` — drop `Backfilling` / `BackfillProgress` from `StatsResponse`. **API change:** `/api/stats` no longer emits `backfilling` / `backfillProgress`; the `X-CoreScope-Status` header is removed. Verified no frontend or other consumer reads them. - `resolved_index.go` — remove stale comment referencing the deleted `backfillResolvedPathsAsync`. ## Test Regression assertion added to `TestStatsEndpoint` (#1546): asserts the response no longer carries `backfilling` / `backfillProgress` and that `X-CoreScope-Status` is unset. Verified red→green — against pre-fix code all three assertions fail; with the fix they pass. Full `cmd/server` suite green locally. ## Out of scope If a real server-side backfill/migration status indicator is wanted, that's a new feature on top of the ingestor stats pipe — tracked separately, not by reviving these dead fields. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Kpa-clawbot
|
9465949e79 |
fix(#1558): mirror Load's resolved_path indexing into loadChunk (#1582)
## Summary Closes #1558. The background-backfill path (`loadChunk`) silently dropped the resolved-path indexing branch that `Load` performs per observation. Same SQL rows, two different post-conditions — a contract violation between the hot-startup load and the background chunk load. ## Root cause (the differential matters) The reporter's hypothesis — `indexByNode` not invoked on background-loaded transmissions — was 90% right but pointed at the wrong line. - `cmd/server/store.go:1116` already calls `s.indexByNode(tx)` inside the loadChunk per-batch merge lock for every backfilled tx. Decoded `pubKey` / `destPubKey` / `srcPubKey` ARE indexed. - `indexByNode` (store.go:1313 pre-patch) only reads three fields from `decoded_json`. It does NOT and cannot touch `resolved_path`. - `Load` (store.go:783-799) per-observation unmarshals `o.resolved_path`, extracts every relay-hop pubkey, and feeds them through `addToByNode` + `addResolvedPubkeysToPathHopIndex` + `addToResolvedPubkeyIndex`. - `loadChunk` (store.go:937-1023 pre-patch) selects `o.resolved_path` into `resolvedPathStr`… then never touches it. Result: after a container restart, every transmission older than `hotStartupHours` ends up present in `s.packets` / `s.byHash` / `s.byTxID` but missing from `s.byNode[relayPK]` for every relay pubkey. Home-page per-node `packetsToday` / `totalTransmissions` / `observers` / `avgHops` / `avgSnr` collapse for relay-heavy nodes (753 → 8 in the reporter's trace). Stats only self-heal as live ingest re-populates `byNode` through the ingest path (which DID call the full sequence inline). ## Fix shape 1. **Extract a shared `(s *PacketStore) indexResolvedPathHops(tx, pks, hopsSeen)` helper.** Owns the `addToByNode` + `addResolvedPubkeysToPathHopIndex` + `addToResolvedPubkeyIndex` sequence. Single point of truth so the "feed decode-window consumers for resolved-path pubkeys" invariant is structural, not duplicated. 2. **Re-point `Load` and both ingest sites at the helper.** Load's semantic behaviour is byte-identical with the prior inline block. 3. **Add the missing call in `loadChunk`.** Per AGENTS.md performance rule #0 ("no expensive work under locks"), unmarshal `resolved_path` and dedupe relay pubkeys per txID **outside** the merge critical section (`localResolvedPKsByTx`), then feed the pre-built slice through `indexResolvedPathHops` inside the existing per-batch lock alongside `indexByNode`. Mirrors `loadChunk`'s "build local, merge under lock" shape. ## TDD: red → green commits ``` |