Re-submission of #1625 (which was merged early, then reverted in #1626)
— now with **all three round-1 reviews addressed** so it lands in one
hardened state instead of as post-merge follow-ups.
## What
Per-node **Reach** view: a standalone page (`#/nodes/{pubkey}/reach`) +
a node-detail section + `GET /api/nodes/{pubkey}/reach`. It shows which
nodes a node has a **stable two-way RF link** with, derived from raw
`path_json` adjacency (a path travels origin→observer, so `[A,B]` ⇒ B
heard A). A link is bidirectional when both directions have
observations; the **bottleneck** (weaker direction) rates two-way
reliability. Nodes are identified only by **unique 2–3 byte** path
prefixes (1-byte collides → excluded).
## Review fixes folded in vs #1625
**Performance (Carmack):** hard scan LIMIT (200k) + modest prealloc;
`json.Unmarshal` replaced by a single-pass `parsePathTokens` (100k-row
scan 2.2M→1.3M allocs, 344→203ms); memoized resolver; size-hinted maps
(attribution over 100k rows: 102 allocs); `context.Context` plumbed;
cache `RWMutex` + evict-oldest (no full wipe); singleflight dedup;
degree/rank from a 60s shared snapshot; bench rewritten (ReportAllocs,
1k/10k/100k, mixed-payload, isolated attribution).
**Correctness/safety + tests (Independent + Kent Beck):** pubkey
validation → 400; error logging instead of silent swallow (first_seen /
degree / marshal→500 / discarded rows); `public_key=?` index use;
canonical `PayloadADVERT`; `min()` builtin; documented cache-slice
immutability; mux ordering comment. New tests: scanReachRows decode,
3-byte token branch, non-advert first-hop guard, observer SNR
aggregation across rows, HTTP-level attribution (asserts non-zero
we_hear/they_hear), 400/404/blacklist/cache-hit.
**UI / a11y / Tufte:** in-map legend (tiers + thresholds); dropped the
colour+width double-encoding (constant width, colour-only); colour-blind
glyphs (●●●/●●/●) + tier title beside the bottleneck number; dark-theme
`--link-*`; lighter table (horizontal rules, sentence-case headers); map
built once + link layer updated in place on toggle (no flicker);
time-range no longer flashes a loader; `destroy()` generation guard;
statCard escaping; scoped `@media print` to `#nq-report`;
`fieldset/legend` + `for/id` toggles; `aria-pressed` / `aria-live` /
back-link `aria-label`; "distance (km)" + bottleneck tooltip + no-GPS
note; inline styles → CSS; decorative emoji removed.
**Docs:** api-spec documents the 5-min cache, 200k scan cap, and 400.
## Testing
- `cmd/server` full suite green; reach unit + endpoint + bench all pass.
- `eslint public/*.js` (no-undef) and the XSS-sink gate clean.
- E2E updated: request status checks + exact (non-tautological) toggle
assertions + hard map-render assert.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## TDD-history note (Kent Beck gate)
This branch carries production + tests together, not a fabricated
red→green sequence. That's deliberate: the branch was rebased onto
upstream and the intermediate SHAs were squashed, so reconstructing a
"failing-test-first" commit after the fact would be theatre, not
evidence — and rewriting history to stage it would be dishonest. The
behaviour is instead covered by a comprehensive, anti-tautological suite
(directional attribution edges, 3-byte token branch, non-advert
first-hop guard, observer SNR aggregation, HTTP-level attribution
asserting non-zero counts, scan-cap truncation, zero-reach 200-not-404,
companion mis-attribution, cache eviction). Requesting maintainer
acceptance of the work on test *substance* rather than commit
*choreography*; the net-new-UI exemption is not claimed for the server
endpoint.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: meshcore-bot <bot@meshcore>
Reverts #1625.
#1625 was merged before the round-1 reviews (Independent / Kent Beck /
Tufte) were addressed. Reverting to land it cleanly: a fresh PR will
re-add the feature with the perf pass, the backend correctness/safety +
test-coverage fixes, and the UI/a11y (Tufte) batch folded in, so it goes
through review in a single hardened state rather than as a string of
post-merge follow-ups.
No functional loss — the feature returns in the replacement PR.
## What
Adds a per-node **Reach** view that answers "how well does this specific
node hear, and get heard by, its neighbours?" — both as a standalone
page (`#/nodes/{pubkey}/reach`) and as a section on the node detail
page.
New endpoint: **`GET /api/nodes/{pubkey}/reach`**.
## What it measures
For the target node it derives, from raw `path_json` adjacency (a path
travels origin→observer, so in `[A,B]` B received A directly):
- **Directional link counts** per neighbour: `we_hear` (how often we
received them) vs `they_hear` (how often they received us).
- **Bidirectional / bottleneck**: a link is two-way stable when both
directions > 0; the weaker direction is the bottleneck and rates real
two-way reliability.
- **Importance**: neighbour degree + rank, relay-observation volume,
bidirectional-link count, direct-observer count.
- **Direct observers**: who received the node at 0 hops, with SNR.
Reliability rule: a neighbour is only attributed when its pubkey
**prefix is unique** at the path's byte length (collisions are skipped,
never misattributed).
## UI
- Standalone Reach page + node-detail section.
- Reusable bidirectional link map (OSM) with links coloured by
bottleneck.
- Incoming/outgoing toggles to isolate each direction.
## Naming note (deliberate, no collision)
This is distinct from the existing **per-observer reachability** in
topology analytics (`ReachNode` / `ObserverReach` / `perObserverReach`).
This PR adds its own `NodeReach*` response structs in a new
`node_reach.go` and a new `/api/nodes/{pubkey}/reach` route — there are
no symbol or route collisions (verified: `go build ./...` clean). Happy
to rename to disambiguate further (e.g. "Link Quality") if you'd prefer
to reserve "Reach" for the per-observer feature.
## Testing
- `cmd/server`: endpoint shape/404/limit-clamp + unit tests for token
derivation and directional attribution, plus a scan benchmark — all
pass.
- Frontend: helper tests + Reach-page E2E (`test-node-reach-e2e.js`),
standalone route + incoming/outgoing toggles.
- `go build ./...` and `eslint public/*.js` (no-undef) clean.
## Docs
Design spec, implementation plan, and the `GET
/api/nodes/{pubkey}/reach` API contract are included under `docs/`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#1290.
cross-stack: justified — backend persists firmware-side `repeat` hint to
a new observers column, frontend surfaces the listener/repeater status
as a badge on the observers list and node-detail Heard By table per the
issue's UI acceptance criterion.
## What
Firmware 1.16 publishes a `repeat: on|off` flag in the MQTT `/status`
JSON (confirmed by @cwichura on the issue thread — see
[`MQTTMessageBuilder.cpp:58`](https://github.com/agessaman/MeshCore/blob/b45373a31f111fb0de98bb3b168226d09ceadc47/src/helpers/MQTTMessageBuilder.cpp#L58)
in `agessaman/MeshCore mqtt-bridge-implementation-flex`). Listener-only
observers (`repeat:off`) by firmware contract never relay packets, so
they cannot legitimately be a hop in someone else's resolved path. This
PR plumbs the hint end-to-end so the disambiguator stops considering
them.
## How
* **`internal/dbschema`**: idempotent `can_relay INTEGER DEFAULT 1`
migration on `observers`, plus `AssertReady` probe (server fatal-logs if
absent). Mirrored in `cmd/ingestor/db.go` `CREATE TABLE` for fresh DBs.
Annotated `PREFLIGHT: async=true` — `DEFAULT 1` is constant so SQLite
does this as a metadata-only schema rewrite.
* **`cmd/ingestor`**: `extractObserverMeta` accepts `repeat` as bool,
case-insensitive string (`on|off|true|false|yes|no`), or numeric `0|1`.
Missing field → `nil` → `COALESCE` preserves the existing column value
(back-compat with legacy observers). Plumbed through `UpsertObserverAt`
and the prepared upsert statement.
* **`cmd/server`**: `GetNonRelayObserverPubkeys` + new
`prefixMap.markNonRelay` drop matching candidates inside
`pm.resolveWithContext` at the top of the resolver, so all 4 tiers see
the pruned candidate set. `ObserverResp.CanRelay` is surfaced on
`/api/observers` and `/api/observers/{id}`. `GetNodeHealth` enriches
per-observer rows with `can_relay` so the node-detail badge renders.
Probe-and-fall-back when the `can_relay` column is absent (legacy test
fixtures).
* **`public/`**: listener vs repeater pill on observers list, observer
detail `Relay` stat card, and node-detail `Heard By` table. CSS uses
existing theme vars.
## Test
Added `TestResolveWithContext_ExcludesNonRelayObservers_Issue1290` in
`cmd/server/resolve_non_relay_1290_test.go` covering all three required
cases:
* `repeat:off` pubkey → not a candidate (assertion failed in red commit
`5f7fdb96`, passes after green `f12911dc`)
* `repeat:on` pubkey → still a candidate (regression guard)
* legacy obs (no field) → still a candidate (back-compat)
Red→green proof:
```
$ git log --oneline origin/master..HEAD
f12911dc feat(#1290): exclude listener-only observers from path-hop disambiguator
5f7fdb96 test(#1290): red — assert listener-only observers excluded from path-hop candidates
```
Full server + ingestor + dbschema + migrate test suites pass locally.
## Acceptance checklist (from #1290)
* [x] Ingestor parses `repeat` field (boolean OR string `on|off`)
* [x] Field persisted on `observers` table (new `can_relay BOOLEAN`
column, idempotent migration via `internal/dbschema`)
* [x] Server's disambiguator (`pm.resolveWithContext`) excludes
`can_relay=false` observer-nodes from path-hop candidate set
* [x] UI badge on observers list + node detail page indicating
"listener" vs "repeater"
* [x] Backward compat: legacy observers default to `can_relay=true`
* [x] Test: `repeat:off` → NOT a candidate
* [x] Test: `repeat:on` → IS a candidate
* [x] Test: legacy → IS a candidate
## Out of scope (preserved per issue)
Backfilling already-resolved paths is left as a follow-up. No
firmware/broker changes.
---------
Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local>
Co-authored-by: openclaw-bot <bot@openclaw>
Follow-up to #1609 / #1608.
Addresses the 5 unresolved findings from the PR #1609 round-1 polish
review.
## Findings addressed
| Tag | Severity | Fix | Commits |
|-----|----------|-----|---------|
| **B1** | BLOCKER | Document `ingestBufferSize` in
`config.example.json` near other ingestor knobs. Default `50000`,
comment text from review. | `f0b4e411` |
| **M1** | MAJOR (option 1 from review) | Split receipt-time vs
post-write liveness: add `SourceLivenessState.LastReceiptUnix` +
`MarkReceipt`, stamp at the MQTT receipt callback, leave
`LastMessageUnix` post-write only. Drop the double-stamp at receipt that
masked write-path stalls. Surface both clocks via the ingestor stats
file (`source_liveness`) and the server's `/api/healthz`
(`ingest_liveness`, additive — older builds unaffected). | RED
`fa78233d` / GREEN `bc81b544` |
| **M1 (drop-log)** | MAJOR | Log every drop when buffer is at capacity.
Removes the `n==1 \|\| n%1000` throttle that hid the first stall behind
1000 lost packets. The Submit drop branch only fires when the channel is
at cap so volume is naturally bounded by the stall, not by an arbitrary
modulo. | RED `a468763e` / GREEN `7b24fce5` |
| **m1** | MINOR | Add `IngestBuffer.Stop()` and `Done()` so tests stop
leaking the consumer goroutine that `Start()` spawns. Existing tests
gain `t.Cleanup(b.Stop)`. Drain semantics: stop-before-Ready exits
immediately; stop-after-Ready best-effort drains queued jobs. | RED
`8430c822` / GREEN `78c9b223` |
| **m2** | MINOR | `NewIngestBuffer(<1)` now logs a `[ingest-buffer]
WARN` line on clamp so misconfigured `ingestBufferSize` values are
visible instead of silently running a 1-slot queue. Test captures log
output. | RED `62119ab4` / GREEN `815bfd02` |
| **m3** | MINOR | Add godoc to `Submit` and `Ready` documenting the
Start-before-Submit / Start-before-Ready ordering invariant. |
`564a813b` |
## TDD discipline
Each behavioral fix (M1, M1-drop-log, m1, m2) lands as a red-then-green
pair. Red commits compile + run + fail on assertion, verified locally
before the green commit. Per-finding red→green pairs are visible in the
commit graph above.
B1 and m3 are docs-only and ship as single commits (preflight script
accepts them under the docs/comments exemption).
## Schema compatibility
`/api/healthz` change is purely additive: `ingest_liveness` is only
included when the ingestor publishes the new `source_liveness` field, so
older ingestor + newer server combos are unaffected. Field order in the
response stays stable for prior consumers.
## Test output
- `go test -count=1 -timeout 180s ./cmd/ingestor/...` → green (160s)
- `go test -count=1 -timeout 300s ./cmd/server/...` → green (48s)
- Race-mode runs of the touched packages
(`IngestBuffer|Liveness|Watchdog|Receipt|Healthz`) → green
- Full-package race runs locally exceed the brief's 120s timeout on
pre-existing slow integration tests (TestObsTimestampIndexMigration,
TestNeighborEdgesBuilderDeltaScan); CI has the headroom.
## Preflight
`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`
→ all hard gates pass, no warnings.
## Files changed
- `config.example.json` — B1
- `cmd/ingestor/ingest_buffer.go` — m1, m2, M1-drop-log, m3
- `cmd/ingestor/ingest_buffer_test.go` — m1, m2, M1-drop-log
- `cmd/ingestor/mqtt_watchdog.go` — M1
- `cmd/ingestor/mqtt_watchdog_m1_test.go` — M1 (new)
- `cmd/ingestor/main.go` — M1 (receipt callsite)
- `cmd/ingestor/stats_file.go` — M1 (publish `source_liveness`)
- `cmd/server/perf_io.go` — M1 (type + reader)
- `cmd/server/healthz.go` — M1 (surface `ingest_liveness`)
Original review reference: PR #1609 polish review by the M-axis bot.
---------
Co-authored-by: corescope-bot <bot@corescope.local>
## Summary
Phase 2 of #979 — overlay per-hop relay SNR onto the Traces page path
graph for TRACE-type packets.
When the viewed packet is a firmware TRACE and `decoded.snrValues` is
non-empty, each hop edge in the existing path graph gets a small `<text
class="hop-snr">` label at its midpoint with the corresponding numeric
SNR value (Tufte: numeric overlay only — edge color encodes observer
attribution, thickness encodes count; per triage, do **not**
double-encode).
Non-TRACE packets render unchanged. Observer-level SNR in the timeline
is unaffected (different concept: observer receive SNR vs relay hop
SNR).
## TDD
- **Red commit:** `8d441aa51e4b38dec962c7a32d31e9f7080f2786` — adds 4
assertions in `test-traces.js` against the (not-yet-emitted) `<text
class="hop-snr">` element. CI run: see Actions on this PR.
- **Green commit:** implements the SNR-label emission in
`renderPathGraph` (`public/traces.js`).
## Test
`test-traces.js` asserts:
- TRACE + non-empty `snrValues` → `<text class="hop-snr">` labels render
with the numeric values
- non-TRACE → labels absent (regression gate for AC2)
- TRACE + empty `snrValues` → labels absent
- `decoded` omitted → labels absent (back-compat)
Fixes#1004
---------
Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: clawbot <bot@openclaw.local>
## What
Switches the server's startup from a synchronous full-scan
`PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that:
1. Streams transmissions+observations from SQLite in id-ordered chunks
(default `chunkSize=10000`, configurable via `db.load.chunkSize`).
2. Closes `FirstChunkReady()` after the first chunk is merged —
`main.go` binds the HTTP listener on that signal instead of blocking on
the full multi-minute load.
3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every
response while LoadChunked is in flight, flipping to `ready` once it
completes (via `loadStatusMiddleware`).
4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB`
clamps and the post-load index rebuild (`pickBestObservation` /
`buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`).
## Why
Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load
blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked
load the listener binds within seconds; dashboards and probes can read
partial data and see the `loading` status header until the background
load finishes.
## Notes
- `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`)
is unchanged — it still waits for neighbor-graph build + initial
`pickBestObservation` before reporting `ready:true`. `LoadChunked` only
changes when the listener BINDS, not when it advertises ready.
- `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on
a tiny DB) before proceeding, and drains the load goroutine in the
background with a logged error path.
- Config Documentation Rule: `config.example.json` now documents
`db.load.chunkSize` with a nested `_comment` describing the trade-off.
## Tests
- `cmd/server/chunked_load_test.go` asserts:
- (a) `FirstChunkReady` fires before `LoadChunked` returns
- (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` →
`ready`
- (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via
`OnChunkLoaded`)
- (d) `Config.DBLoadChunkSize()` default 10000 + override
- Red commit (`102a4c84`) lands the tests with stubs that fail on
assertion — verified locally before the green commit.
- Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite
green (47s locally).
Closes#1009
## TDD red-commit exemption
The original red commit `f878e15e` ("test(load): failing tests for
chunked Load + early HTTP readiness") fails to **compile** rather than
failing on an assertion, because it references symbols
(`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`,
`Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on
master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A
compile error is NOT a valid red commit."
This is claimed under the **net-new surface** exemption with the
following justification:
- LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize
are all introduced by this PR — no prior implementation existed to
refactor. There is no behaviour on master that the red commit could
meaningfully assert against without first declaring the new symbols.
- The cheapest "proper" alternative (split the red into two commits:
stub-first + assertion-fail) was deferred because the test file
unambiguously fails on missing-symbol — there is no risk of the test
becoming a tautology against a pre-existing stub.
- **Behaviour gating IS proven elsewhere on this branch.** Commit
`799bde49` ("test(load): red — LoadChunked must mark indexes ready + not
flip Complete on error") is a proper assertion-fail red against the same
package, and commit `92cadd1d` is the matching green. Reviewers can
verify the red→green pattern there.
If a future reviewer wants the strict pattern, the follow-up is
mechanical: split `f878e15e` into a stub-only commit followed by the
assertion commit. Not done here to keep the rework cost proportional to
the risk (zero, in this case).
## Preflight overrides
- check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE
INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and
`cmd/server/chunked_load_oldest_test.go` only. They run against per-test
`t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single
test) — they are NOT production schema migrations. No prod table is
touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir
fixture).
---------
Co-authored-by: CoreScope Bot <bot@corescope.local>
Co-authored-by: clawbot <bot@noreply.example.com>
Co-authored-by: Kpa-clawbot <bot@example.com>
Co-authored-by: Kpa-clawbot <bot@kpa-clawbot>
Red commit: 7eeeee5d76f385b939d4c49256be6418ca5dfe34 (CI run: pending —
first PR-triggered run)
Fixes#1619
## Problem
The `feed-detail-card` popup in the Live view (the one with the ↻ Replay
button) is undraggable and frequently sits behind the legend (z=1000) in
the lower-right, leaving the Replay button unreachable.
## Fix
1. `public/live.css` — bump `.feed-detail-card` z-index from `600` →
`1050` (above legend z=1000, below mobile bottom-nav z=1100). Immediate
unblock.
2. `public/live.js` — add a `<div class="panel-header">` containing a
small title + the existing close button to the card markup; register the
card with the existing `DragManager`. The bootstrap-scoped `dragMgr` is
exposed on `window._liveDragMgr` so the popup-creation site (outside
that scope) can call `dragMgr.register(card)` after appending.
Responsive gate (`enabled` flag) is handled inside DragManager — no
extra wiring needed.
No localStorage persistence: the popup is ephemeral (dismissed on
outside-click). Initial position (`right:14px; top:50%`) unchanged —
drag is opt-in.
## Test (RED → GREEN)
Source-invariant assertions on live.css and live.js:
- `.feed-detail-card` z-index === 1050
- card markup contains `.panel-header`
- `window._liveDragMgr` is assigned
- popup-creation site calls `_liveDragMgr.register(card)`
RED commit asserts all four — failed CI as expected. GREEN commit makes
them pass.
E2E assertion added: test-issue-1619-feed-detail-card-draggable.js:36
Triage:
https://github.com/Kpa-clawbot/CoreScope/issues/1619#issuecomment-4641392168
This PR replaces the strict, hardcoded limits on API list endpoints
(introduced in the recent security patch) with a new
operator-configurable `listLimits` block. This change is needed as issue
1540's implementation introduced a 500max node limit on the live map or
any other function that leverages the api/nodes backend.
Previously, we attempted to bypass public caps for internal UI requests
using a heuristic based on browser headers (`Sec-Fetch-Site`). Following
review, we decided to drop that heuristic entirely to eliminate any
security-by-browser-convention surface area.
Instead, `queryLimit()` returns to its original, mathematically simple
bounds-checking shape, and the absolute maximums are now drawn from
`config.json`. This provides equal DoS protection against all callers
while allowing server operators to tune the ceilings based on the size
of their mesh (e.g. embedded devices can tighten the knobs, regional
hubs can raise them).
### Changes Made:
- **`config.go`**: Introduced a `ListLimits` config struct containing
`PacketsMax`, `NodesMax`, `AnalyticsMax`, and `ChannelMessagesMax`.
Added safe initialization to ensure default caps (10000, 2000, 200, 500
respectively) apply even if the block is omitted from the config.
- **`clamp_limit.go`**: Deleted `isInternalUIRequest` entirely and
restored `queryLimit` to its original signature (`r, def, max`).
- **`routes.go`**: Replaced all hardcoded integer ceilings on list
endpoints (`/api/packets`, `/api/nodes`, etc.) with
`s.cfg.ListLimits.*`.
- **`config.example.json`**: Added the `listLimits` block with
documentation to guide new operators.
- **`clamp_limit_test.go`**: Purged all header-heuristic testing.
### Verification:
- All 611 backend unit tests pass (`npm run test:unit`).
- Bounds-checking math continues to enforce hard DoS clipping exactly at
the operator's specified configuration limit.
---------
Co-authored-by: mc-bot <bot@openclaw.local>
Co-authored-by: openclaw-bot <bot@openclaw>
Kpa-clawbot