Follow-up to #1986, and the second half of the same problem.
`ScopeAuditForwarding` drops rows whose `scope_name` is the empty string
with a bare `continue`. That empty string is the ingestor's
"transport-scoped, but no configured region key matched `code1`" state
(`scopeNameForDB`), so those packets name no region and can never
satisfy a declared one. The consequence is on the page: **a repeater
forwarding a region this instance holds no `hashRegions` key for is
reported exactly like a repeater forwarding nothing at all.** The audit
presents a gap in the reader's own configuration as a finding about
someone else's hardware.
This counts them per target, exposes the count as
`observedUnmatchedPackets`, and renders it as a caveat chip beside the
scope chips.
## Why it is not a rare edge
Measured on a live instance before this landed: of 613 `notObserved`
entries across 205 repeaters, **260 named a region that never appeared
under any name in the whole 7-day window**. Two of them (`behss`,
`fm-112`) were hash-verified as genuinely forwarded traffic the instance
simply could not name: packet `0a065d41d51f1f77` decodes to
`code1=9209`, which is exactly the code `#fm-112` derives over that
packet's own payload.
That instance had 16 region keys configured against 124 distinct region
names its repeaters declare. A stock install has fewer.
## What the counter is not
It is deliberately **not** folded into `unscopedPackets`. The two are
opposites:
| | meaning | what governs it |
|---|---|---|
| `unscopedPackets` | the packet carried no scope at all (`scope_name`
SQL NULL) | the `*` wildcard |
| `observedUnmatchedPackets` | the packet IS scoped, this instance holds
no key for that region | nothing the repeater declares |
For the same reason the new count never feeds `wildcardContradiction`,
which counts only plain unscoped floods. `scopeNameForDB` in the
ingestor is the source of truth for that three-state encoding, and the
comments point there rather than restating it.
It is also distinct from `ambiguousHops`, and the distinction is the
point of the chip: that one is a pubkey-prefix collision between two
repeaters and is nobody's fault, this one is a missing entry in the
reader's own configuration and they can act on it. Saying which is which
is what stops someone investigating an innocent repeater.
## Frontend
The chip reuses the muted dashed treatment of `.sa-chip-ambiguous` on
purpose: both are caveats on the row's finding rather than findings
themselves, and neither may compete visually with the red/green scope
chips beside them.
It renders nothing for a non-numeric count. The value is
server-supplied, and a truthiness check would put the literal string
`NaN forwarded packets` on the page if that ever stopped holding.
## Docs
`docs/api-spec.md` had **no entry for `GET /api/scope-audit` at all**,
so this adds one: query parameter, full response shape, and the notes a
client needs (the three traps the per-node endpoint documents apply here
identically, `*` is never a scope, and "never asked" is not "declared
nothing"). The new field is documented there rather than in isolation.
## Tests
- the counter on a last-hop and on a mid-path hop
- an unmatched packet enters neither `agg.scopes` nor `unscopedPackets`,
which is the confusion this field exists to prevent
- the field on the API row
- six frontend cases: zero renders nothing, a missing field renders
nothing (older server), the chip carries its count and class, singular
and plural are both grammatical, the title names the cause and the fix,
and a non-numeric count renders nothing rather than `NaN`
`cd cmd/server && go test ./...` passes, frontend 712 assertions pass,
`go vet` and `gofmt -l` clean.
Rule 0: the counter is one increment on a branch that already existed as
a `continue`, inside a loop this PR does not change. No new query, no
new pass over the data.
Part 1 of #1856. Part 2 (the hash migration reporting false success) is
#1958.
## It has never worked
`handlePostPacket` writes to the server's DB handle, and that handle is
read-only. `cmd/server/db.go:106`:
```go
dsn := fmt.Sprintf("file:%s?mode=ro&_journal_mode=WAL&_busy_timeout=5000", path)
```
Every call answered `500 attempt to write a readonly database`.
**This is the second report.** #1196 raised it on 2026-06-13, a fix was
merged that corrected v2 column names to v3, and the issue was closed.
That fix could not have worked, because the column names were never why
the write failed. Its comment is still sitting at `routes.go:1288`, next
to code that has never executed successfully in production.
## Why remove rather than build a handoff
**It cannot break a caller.** An endpoint that has only ever returned
500 has no working consumer. This is not a breaking API change, it is
documentation catching up with reality. Nothing in `public/` calls it.
**It was actively misleading.** `openapi.go` advertised it as "Ingest a
packet" and it sits behind `requireAPIKey`, which reads as a live,
protected write endpoint.
**Its test hid the breakage.** `TestPostPacketPersistsV3Schema` asserted
the observation row is written and passed for four months, because the
test DB is opened read-write while production is not. That is how #1196
came to be closed as fixed.
**Ingest is MQTT-only by design since #1283.** Re-adding an HTTP write
path re-opens the invariant that change established. If manual injection
is wanted later for testing or replay, it belongs on the ingestor side
and deserves its own issue. The repository already has the handoff shape
for that: the server writes `request-<id>.json` and the ingestor
consumes it (`cmd/ingestor/prune_geofilter.go`).
## What went
The route, `handlePostPacket` (103 lines), the now-unused
`PacketIngestResponse` type, the `openapi.go` entry, the round-trip
test, and the section plus table-of-contents line in `docs/api-spec.md`.
The `packetpath` import in `routes.go` became unused and went with it.
`+4/-225` across 6 files.
## The auth tests
The four `requireAPIKey` tests used `"/api/packets"` only as a request
path while building their own handler with `s.requireAPIKey(...)`, so
they never touched the route.
I checked that by **running them**, not by reading the code:
```
--- PASS: TestRequireAPIKey_RejectsWeakKey
--- PASS: TestRequireAPIKey_AcceptsStrongKey
--- PASS: TestRequireAPIKey_EmptyKeyDisablesEndpoints
--- PASS: TestRequireAPIKey_WrongKeyUnauthorized
```
Their paths now point at `/api/admin/prune-geo-filter`, which still
exists, so they no longer name a removed endpoint. Re-ran after that
change: still 4 of 4.
`/api/packets/observations` is a different endpoint and is untouched.
Verified: `gofmt` clean, `go vet` clean, `cmd/server` suite ok in 62.9s.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
v3.10.0 was tagged and then withdrawn. **Nothing was ever available
under that number**: no container image and no release asset was ever
published, so no user could have pulled it.
This renames the notes and the CHANGELOG section. No product code
changes.
## Why it had to be renumbered
Three things, in the order they bit.
**1. The image never built.** `release-fast-path.yml` re-tags `:edge` to
`:vX.Y.Z` when the `:edge` revision label matches the tagged commit, and
dispatches `deploy.yml` when it does not. The tagged commit was
documentation-only, so the `paths-ignore` from #1949 meant no `:edge`
existed for it and the fallback ran. That part behaved correctly. The
fallback then published nothing, because every GHCR step was gated on
`github.event_name == 'push'` and a dispatch is not a push. It built
locally, reported `success`, and pushed nothing.
Fixed in #1951, but that fix is not in the `v3.10.0` tag, and a
`workflow_dispatch` runs the workflow file **from the ref it targets**.
So the existing tag could not be made to publish.
**2. The assets never uploaded.** I created the GitHub release by hand
before the workflow reached it, and `action-gh-release` cannot update an
immutable release. The correct procedure is to push the tag and let the
workflow create the release.
**3. The tag name cannot be reused.** GitHub's immutable releases keep a
tag name reserved even after the release is deleted:
```
remote: - Cannot create ref due to creations being restricted.
```
I established that only after deleting the release, which is the wrong
order. The lesson, written into the commit message so it survives: check
whether a tag can be rewritten before removing anything that depends on
it.
## What is in v3.10.1
The same 111 commits, plus the three CI fixes that landed after the
v3.10.0 tag (#1949, #1950, #1951). Those are listed in their own section
in the notes. **No product code differs** from what was tagged as
v3.10.0.
All 69 SHA references in the notes were re-verified after the rename.
## Procedure for this tag
Push the tag and stop. The workflow creates the release and attaches the
assets. Do not create it by hand.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Release notes for the first tag since `v3.9.2` on 2026-06-13. **111
commits**, and no auto-generated coverage bumps fall in this range, so
all 111 are substantive.
Nothing here changes behaviour. It is `docs/release-notes/v3.10.0.md`
plus a `CHANGELOG.md` section.
## Verification
The header promises that every bullet ends with a SHA you can `git
show`. That is checked mechanically rather than trusted: all **69**
references were confirmed to point at a commit that exists and whose
subject line contains the issue or PR number cited beside it. Zero
mismatches.
## Two things operators need, and both are silent failures
The urgency line leads with the first one on purpose.
1. **CARTO requires an API key** on its raster basemaps since 2026-08.
Without one every tile is served watermarked with HTTP 200. Nothing
errors, no healthcheck fires, and the only way to notice is to look at a
tile. Anyone upgrading needs to set `map.tiles.providers.carto.key`.
2. **`pathTrust.minHashBytesForMapping` ships at 1**, which is the
existing behaviour, so an upgrade changes nothing on its own. The note
states what raising it to 2 would actually cost, with numbers from a
live instance (56% of path-hop observations are 1-byte, 41% of repeaters
use a 1-byte hash), because there is no UI to undo it.
The relay `last_seen` fix is quantified the same way rather than
described as "improved": for repeaters that relayed within the last
hour, the gap between `last_relayed` and `last_seen` drops from a median
of 12,062 s to 193 s, and the share more than five minutes behind falls
from 96% to 39%.
## A theme worth naming
Three of the highlights are the same defect in three places: something
is operable before its own setup has finished. The Live view toggles are
inert for about 100 ms after paint, the colour picker's deferred focus
undid arrow-key navigation so Enter assigned the wrong colour, and an
analytics theme-refresh discarded the filter you had just applied. All
three were first written off as flaky tests, twice by me. Each is now
fixed with a regression test that fails on the previous commit.
## Sequencing
This should land, and the `v3.10.0` tag be cut, **before** the Go 1.27
upgrade in #1946. A toolchain bump changes the compiler, the runtime and
`gofmt` for everything at once; landing it on top of 111 unpublished
commits means a later regression cannot be separated from the toolchain.
#1946 itself says no 1.27-only features are being adopted, so there is
no cost to waiting one release. A tag first also gives a known-good
bisect point.
## Not done
The CHANGELOG has no `3.9.2` section and did not have one before this
change. I left that gap alone rather than reconstructing it
retroactively.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- `docs/DEPLOYMENT.md` and `docs/deployment.md` were both tracked in
git, colliding into a single file on case-insensitive filesystems
(default on macOS/Windows) and causing `git status` to report spurious
modifications.
- `docs/deployment.md` is the actively maintained guide (linked from
`README.md` and `docs/deployment-behind-cdn.md`); `docs/DEPLOYMENT.md`
was a stale duplicate untouched since the MeshCore → CoreScope rename.
- Removed `docs/DEPLOYMENT.md` from the index, keeping
`docs/deployment.md`.
## Test plan
- [x] `git status` is clean on a case-insensitive checkout with no
spurious modification
- [x] Confirmed no remaining references to `docs/DEPLOYMENT.md` in the
repo
> **Stacked on #1905.** This branch contains #1905's commits; review and
merge that one first. The diff unique to this PR is the
`client_rf_samples` table, its handler, the delta query and its
retention.
## What
Everything CoreDrive RX records today is anchored to a *packet*. But a
drive also passes through RF conditions that exist whether or not a
packet arrives: the noise floor, how busy the channel is, how many
receptions fail CRC. The radio measures all three and was never asked.
This samples the companion's own counters along the GPS track and stores
them, so the server can render a noise-floor map, a channel-utilisation
map and a CRC-error-rate map. A fixed observer cannot produce those — it
measures one point forever.
**Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the
attached radio. Nothing is transmitted.
## Design points worth knowing
- **Absolutes are stored; deltas are derived at query time.** A lost or
reordered sample then costs one interval rather than corrupting a
running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs`
fails to increase — that is the reboot and counter-wrap detector.
- **Absent is not zero, end to end.** Firmware predating the
`recv_errors` field cannot count CRC errors at all, and a stored `0`
would read downstream as "a perfectly clean channel" — the opposite of
"we don't know". Presence/absence is preserved through the app parser,
the wire payload, a nullable column, and the delta view, which returns
`nil` rather than `0` when either endpoint is unknown. Each of those
five layers has its own test.
- **`sampled_at` is millisecond precision, and it is load-bearing.**
SQLite compares these strings lexicographically and `.` (0x2E) sorts
before `Z` (0x5A), so a second-resolution retention cutoff would delete
rows *inside* the window. The prune formats its cutoff with the same
layout.
## Performance justification (touches the ingest hot path)
- One INSERT per sample, gated behind an opt-in flag that defaults off.
Sample rate is 15 s while moving and 5 min while parked, so roughly 240
rows per hour per active driver.
- The delta query is a single `LAG(...) OVER` pass with no nested query
inside the loop, so it cannot deadlock the single writer connection.
Window functions are already used elsewhere in this codebase.
- Retention has its own key and index (`sampled_at`); without it the
table would grow unbounded, so `config.example.json` documents it
inline.
## Safety for existing deployments
Opt-in and default off on both sides (`clientRfSamples.enabled`, and
`rfSampler` in the app). The coverage path is untouched —
`Publisher.buildPayload` is byte-identical and a record with no `kind`
field still routes to `/packets` unchanged.
The MQTT dispatch was reshaped so that **anything on `meshcore/client/…`
returns from that branch in every config state**, with the enable-gates
inside rather than in the topic match. Previously a disabled gate let
the message fall through to the observer path, where `parts[1]` — the
literal string `client` — was read as a region and the phone's pubkey
registered as an observer. The blacklist check now also runs ahead of
the sub-topic switch, so it covers every present and future client
sub-topic.
## Testing
Full ingestor suite green. Notable coverage: a `/rf` message with the
gate off writes nothing anywhere and does not fall through; a sample
missing `uptime_secs` is rejected rather than stored as an unusable row;
two samples 40 ms apart remain two rows; and the retention test seeds a
row with a non-zero millisecond component inside the cutoff second,
which is the only row that distinguishes a correct cutoff from an
RFC3339 one.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
A CoreDrive RX drive already carries far more RF information than
reaches CoreScope, and it was being discarded twice: once in the mobile
app (every packet it could not attribute to a directly-heard node was
dropped before queueing) and once here (the ingestor decodes the
*complete* packet, then keeps only
`heard_key`/`snr`/`rssi`/`lat`/`lon`).
This captures what was being thrown away, at **zero extra airtime** —
nothing new is transmitted.
- **`transmissions.code1` / `code2`** — the transport codes were decoded
on every packet and used only to derive `scope_name`, then dropped.
Storing them turns "which repeater forwards which scope" from a re-parse
into a query.
- **An async backfill** re-parses the `raw_hex` already on disk, so
months of scope history become queryable with no new data collection.
- **`client_rx_observations`** — a new diagnostic table holding every
decodable packet a phone heard, with route type, transport codes, scope
name, path-hash size, the full forwarder chain and the forwarder.
## Why it is safe for existing deployments
Both halves are **opt-in and default off**
(`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an
existing deployment sees no behaviour change and no volume change on
upgrade.
The coverage invariant is untouched: `client_receptions` keeps its rule
— 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an
unattributable packet writes **zero** coverage rows. `deriveHeardKey`,
`buildClientReception` and `InsertClientReception` are unmodified except
for one guard described below.
## Performance justification (touches the ingest hot path)
- **Backfill:** keyset-paginated by `id` in 5000-row batches, a single
forward scan, `rows.Close()` before `Begin()` so it never deadlocks
against `SetMaxOpenConns(1)`, and commits per batch so live ingest
interleaves. Termination is driven by rows *scanned*, not rows decoded —
an earlier count-based loop would have stopped at the first batch
containing an undecodable row and then written its completion guard,
permanently stranding the rest.
- **Guard row is written if and only if the loop ran to genuine
exhaustion.** Every error path leaves it unwritten so the next startup
retries.
- **Per-packet cost:** one extra INSERT on the client topic when
enabled, gated behind an opt-in flag. No new work on the observer path.
- **New indexes** cover the prune (`rx_at`), the flood-grouping
(`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the
scope query (`scope_name, rx_at`). Retention has its own shorter window
— this table is diagnostic, not archival.
## Two firmware-derived correctness points
- **`pkt_hash` is `ComputeContentHash()`**, byte-identical to
`transmissions.hash`, so dark-traffic queries are a plain equality join
rather than a translation layer.
- **TRACE packets are refused.** TRACE repurposes the header path bytes
as per-hop SNR values, so deriving a `heard_key` from them invents a
node that never existed. `packetpath.PathBytesAreHops` existed but was
never wired into the client path; it became reachable only because the
app half now publishes packets it previously dropped locally.
## Testing
Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes
zero coverage rows and NULL `forwarder`; a `direction: "tx"` message
writes no observation; a DIRECT route never sets `forwarder`; two
forwarder copies of one flood remain two rows; the backfill's
multi-batch path is exercised with an undecodable row in the first page;
and a forced error asserts the migration guard stays unwritten.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Continues #1916 by @nullrouten0. The commit is theirs, authorship
unchanged; I rebased it onto master and resolved the fallout from #1919.
Opening it here rather than force-pushing to someone else's branch.
## Why this is needed after #1919#1919 shipped the Carto key, but only for the five `BASE_STYLES` entries
in `map-tile-providers.js`. Four map surfaces still request unkeyed
tiles and get them back stamped `API KEY REQUIRED` at HTTP 200:
- `public/roles.js` — `getTileUrl()` returns the bare `TILE_LIGHT`
constant in light mode and never consults the registry (the dark branch
has consulted it since #1461). Affects `analytics.js:2200` and
`nodes.js:94`, so the analytics map and the node-detail map stay
watermarked in light theme even with a key configured.
- `public/customize-v2.js:1838` and `:2047` — the two geo-filter maps in
the customizer.
- `public/geofilter-builder.html` — standalone page, outside the SPA, so
it fetches `/api/config/client` itself.
@nullrouten0 had already found and fixed all four, plus written the
docs, before #1919 was merged. Their diagnosis of the light-mode branch
is in the PR verbatim.
## What I changed while rebasing
1. **`carto.token` → `carto.key`.** #1919 shipped `key` and operators
already have it in their configs; renaming now would break them
silently. All of #1916's code, tests and docs follow suit.
2. **Dropped the duplicate key getter.** `_getCartoKeyParam` did the
same job as master's `_getCartoKey`; kept master's.
3. **Merged the two test suites.** Master's five cases plus four of
#1916's that master does not cover: the `api_key`-is-ignored assertion,
lazy resolution after async config, and both `MC_tileUrlById` cases. 42
passed, 0 failed.
4. **Corrected one factual claim in the docs.** #1916 stated CARTO
offers no referrer or origin restriction. CARTO does ask for a domain
when the key is issued. Whether that is enforced per request is not
something I verified, so the note now says exactly that rather than
asserting either way.
5. Merged the two config comments, keeping the operationally useful
part: unkeyed tiles return **200** with a watermark, so nothing errors
and no healthcheck fires. Verify by looking at a tile, not at a status
code.
## Verification
Rebased onto `7aa60c03`. No regressions:
| | base 7aa60c03 | this branch |
|---|---|---|
| `test-issue-1420-tile-providers.js` | 38 passed, 0 failed | **42
passed, 0 failed** |
| `test-issue-1614-tile-url-function.js` | 3 passed, 0 failed | 3
passed, 0 failed |
| `test-issue-1470-node-tile-helper.js` | 3 passed, 3 failed | 3 passed,
3 failed (pre-existing) |
| `test-frontend-helpers.js` | 625 passed, 2 failed | 625 passed, 2
failed (pre-existing) |
Also verified end to end on a live instance running this change with a
real key: the tile that came back carried the `API KEY REQUIRED`
watermark before and came back clean after.
@nullrouten0 — this is your work and I would rather you had the credit
and the merge. Say the word and I will close this and hand the rebase
back to you, or push it to your branch instead if you prefer #1916 to
stay the vehicle.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE
Co-authored-by: nullrouten <nullrouten@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
Replaces the three drifted per-surface payload-type label vocabularies
with a single canonical map keyed by firmware enum name.
Per the locked triage comment on #1799
([comment-4823975431](https://github.com/Kpa-clawbot/CoreScope/issues/1799#issuecomment-4823975431)):
> Create `public/payload-labels.js` exporting `{GRP_DATA: {short:'Group
Data', long:'Group data packet', enumId:6}, ...}`. Migrate `packets.js
typeMap`, `packet-filter.js FW_PAYLOAD_TYPES`, `live.js TYPE_COLORS
legend` to consume it. E2E that scrapes each surface and asserts label
equality.
## Changes
- **`public/payload-labels.js`** (new) — canonical map exposed as
`window.PayloadLabels` and `window.PayloadLabelsApi`. Keys are firmware
enum names; values carry `{short, long, enumId}` plus derived
`SHORT_BY_ID` / `FW_PAYLOAD_TYPES` / `TYPE_ALIASES` for legacy callers.
- **`public/packets.js`** — `TYPE_NAMES` + `typeMap` now read from
`PayloadLabelsApi.SHORT_BY_ID`. Literal kept only as a defensive
fallback for the case where the script tag fails to load.
- **`public/packet-filter.js`** — `FW_PAYLOAD_TYPES` + `TYPE_ALIASES`
now sourced from `PayloadLabelsApi`. Literal fallback retained so `node
test-packet-filter.js` still works headlessly.
- **`public/live.js`** — legend `<li>` rows are now generated from
`window.PayloadLabels` in stable order, killing the third-vocabulary
`Message — Group text` / `Direct — Direct message` drift the #1797
review surfaced.
- **`public/index.html`** — `<script src="payload-labels.js">` loaded
before `roles.js` / `packet-filter.js` / `packets.js`.
- **`test-issue-1799-label-vocab-e2e.js`** (new) — Playwright E2E.
Scrapes `#liveLegend` rows and the `/packets` type-filter checklist,
asserts each label matches `window.PayloadLabels[ENUM].short` for
`TXT_MSG`, `GRP_TXT`, `GRP_DATA`. Also verifies `window.PacketFilter`
still recognises the enum names.
- **`.github/workflows/deploy.yml`** — wired the new E2E into the
existing Playwright block.
## TDD trail
- Red commit `eb392d4` — adds the failing E2E only (asserts
`window.PayloadLabels` exists and labels match; both fail).
- Green commit `44e902a` — introduces the canonical map and migrates the
three surfaces.
## Verification
- `node test-packet-filter.js` — 92/92 pass with the new fallback
wiring.
- Preflight: `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh
origin/master` — clean.
Browser verified: E2E `test-issue-1799-label-vocab-e2e.js` exercises
`/live` legend + `/packets` type filter against a Playwright headless
Chromium; CI's Playwright block runs it on every push.
E2E assertion added: `test-issue-1799-label-vocab-e2e.js:139` —
`assert(fromLegend === canon, ...)` and `assert(fromPackets === canon,
...)` per enum.
Fixes#1799
---------
Co-authored-by: mc-bot <bot@corescope>
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: clawbot <clawbot@kpa.com>
Co-authored-by: clawbot <bot@clawbot.local>
Closes#1780.
## What
Adds an opt-in **"Multibyte only"** toggle to the live map controls.
When ON, packets whose path hash size is `< 2` bytes (single-byte, or
unresolvable) are excluded from the entire live view — feed, map
polylines/rain, and the packet counter — in both LIVE and REPLAY modes.
- **Default OFF** — no behavior change for existing users.
- Persisted in `localStorage` under `live-multibyte-only`.
- Distinct from the existing global "hide 1-byte path hops" toggle: that
filters individual hops within a path at every render site; this filters
whole packets, on the live view only. They share no state.
## How
- **`public/hop-filter.js`** — new pure, dependency-free classifier
`MC_packetHashSize(rawHex, routeType)` returning `1|2|3`, or `0` when
unresolvable. Reads the path-length byte from `raw_hex` (`(pathByte >>
6) + 1`), offset `5` for transport routes (route_type 0/3) else `1` —
mirroring the existing `getPathLenOffset`/`computeBreakdownRanges` logic
in `app.js`. Lives next to the existing `hopByteLen`/`MC_*` family;
`app.js` is untouched (no duplication of the byte math).
- **`public/live.js`** — `groupIsMultibyte(packets)` consumes that
helper; applied at two render-time sites: the top of `renderPacketTree`
(above the counter increment, so the counter reflects multibyte-only)
and inside the `rebuildFeedList` group loop (so toggling re-filters the
buffered feed). Toggle markup + change handler mirror the existing
`liveFavoritesToggle` pattern.
## Why read from `raw_hex` and not the path hops
The hash size is a property of the whole packet and is present even for
zero-hop packets (where there are no hops to inspect), so reading the
path-length byte is correct in all cases. Unresolvable size is treated
as single-byte (excluded when ON) — we only show packets we can
positively confirm are multibyte.
## Performance (hot path)
The filter runs in the packet-render hot path, so: classification is
**O(1) per packet group** — it reads the first resolvable observation's
`raw_hex` (a short hex string, single `parseInt` of one byte) and
short-circuits. No per-packet API calls, no allocation in the loop, no
added O(n²). When the toggle is OFF (default) the check is a single
boolean guard and does nothing else. The buffered-feed re-filter reuses
the existing `rebuildFeedList` pass — no extra traversal.
## Tests
- **Unit** (`test-live-multibyte-filter.js`, 9 cases):
single/2-byte/3-byte classification, transport-route offset,
missing/short/garbage `raw_hex` → 0, whitespace tolerance.
- **E2E** (`test-live-multibyte-only-e2e.js`, Playwright): toggle
present and defaults OFF; ON hides a single-byte packet while a
multibyte one renders; OFF restores it; setting persists across reload.
Registered in the CI live-E2E block in `deploy.yml`.
## Docs
User-guide entry added in `docs/user-guide/live.md`.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Implements #1727.
## What this adds
**Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage
feature. A roaming MeshCore **companion** radio (driven by the
open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA,
GPLv3) reports which nodes it heard directly, tagged with the phone's
GPS and the packet's SNR/RSSI. CoreScope ingests these into a new
`client_receptions` table and renders per-node **hex coverage** on the
Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`)
with a top-mobile-observers leaderboard.
Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only
node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by
the companion app for friendly names.
## Opt-in — default OFF (zero impact on existing deployments)
The whole feature is gated behind one config flag, **disabled by
default**:
```jsonc
"clientRxCoverage": { "enabled": false }
```
When disabled (the default): the ingestor writes **no**
`client_receptions`; the three coverage endpoints return a clean
**404**; the UI hides the Coverage nav link, the `#/rx-coverage` route,
and the Reach-page toggle. `/api/nodes/resolve` is always available (not
coverage-specific).
## How it works
```
companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets
│
ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key)
│
server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard
```
- **Direct-only capture:** records only what the companion heard itself
and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder)
for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded.
- **No new deps:** hexbins are a pure-Go pointy-top grid over Web
Mercator (`cmd/server/hexgrid.go`) computed at query time
(`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the
existing Leaflet.
- **Trust:** companion pubkey = identity; an EMQX ACL binds each client
to publish only to its own `meshcore/client/{pubkey}/packets` topic.
Payload contract in `docs/client-rx-coverage.md`.
## How to enable / try it
1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and
restart server + ingestor.
2. Point an EMQX (or any broker) listener so a client can publish to
`meshcore/client/<pubkey>/packets`; the ingestor already subscribes
under `meshcore/#`.
3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on
an Android phone paired (BLE) to a MeshCore companion — it captures
heard nodes + GPS and publishes.
4. View results: per-node Reach page → toggle **coverage**, or the
**Coverage** dashboard at `#/rx-coverage`.
## What's where
- **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go`
(`client_receptions` + `client_observers` schema), `main.go` (gated
dispatch), `config.go` (flag).
- **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go`
(endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid),
`node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go`
(wiring + flag + `/api/config/client` field).
- **Frontend:** `public/rx-coverage.js` (dashboard),
`node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach
toggle, flag-gated), `roles.js` (reads the flag, hides nav when off).
- **Docs:** `docs/client-rx-coverage.md`.
## Testing
- Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test
./...` — green, including new gate tests (`coverage_gate_test.go` in
both: off → no rows / 404, on → works) and the rx-coverage / resolve /
hexgrid suites.
- JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js`
(wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is
wired into the e2e job and **skips when `clientRxCoverage` is
disabled**, so it's safe under the default-off config.
## Notes for reviewers
- The four new routes are registered in
`cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness
ratchet), matching how other not-yet-spec'd routes are tracked. Happy to
write full OpenAPI spec entries instead if you prefer.
- Commits are split per layer (ingestor / server endpoints / resolve /
frontend / CI) for review.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Erwin Fiten <e.fiten@opteco.be>
## What
Adds `docs/agents/` — an onboarding pack for external contributors using
their own AI coding agent (Claude Code, Codex, Cursor, Aider, OpenClaw,
etc.).
## Why
Maintainers run an agent-driven workflow against this repo. External
contributors using agents benefit from the same discipline (TDD
red→green, PII preflight, parallel persona polish, three-axis merge
readiness) but had nothing portable to point at. This documents the
**process** and the **reusable building blocks** in an agent-agnostic
way.
## Contents
```
docs/agents/
README.md
WORKFLOW.md # pipeline + planning + PII preflight + force-push + worktrees
RULES.md # 36 hard-won discipline rules
TDD.md # red→green requirement, exemptions
SUBAGENT-BRIEF-TEMPLATE.md
skills/ # 14 task playbooks (intake, fix, polish, merge-gate, release, ops...)
personas/ # 14 review voices (carmack, dijkstra, torvalds, meshcore, taleb, ...)
```
## Scope
Docs-only. No code changes. Existing `AGENTS.md` is unchanged. All
committed text uses sanitized placeholders (`<workspace>`, `<repo>`,
`YOUR_NAME`, `YOUR_HANDLE`, etc.) — no personal names, phones, IPs,
keys, or absolute home/root paths.
## Verification
- PII preflight grep on staged diff: only matches are the literal
placeholders inside the documented sanitized example
(`YOUR_NAME|YOUR_HANDLE|...|api[_-]?key|...`).
- Off-topic skill grep on `docs/agents/`: clean (zero hits for the
wrong-language/off-topic skill names that were scrubbed from the prior
attempt).
---------
Co-authored-by: meshcore-bot <bot@meshcore.local>
Co-authored-by: Kpa-clawbot <bot@openclaw.local>
Co-authored-by: efiten <erwin.fiten@gmail.com>
Release notes for v3.8.4 — the "Phosphor migration" release. Six PRs
(#1649–#1654, tracking #1648) plus three followup fixes
(#1659/#1660/#1665) replaced all decorative emoji in the UI with
Phosphor sprites and added a lint gate to prevent regression.
## Verification summary
Test plan: `workspace-meshcore/test-plans/v3.8.4-cdp-test-plan.md` (93
tests, 16 sections).
- Initial run (pre-#1665): 56 pass / 22 partial / 5 fail / 14 skipped.
Two BLOCKER lint-gate breaches in observers and analytics Channels.
- Final run (post-#1665, hot-patched to staging): both blockers ✅ —
v384-1.2 (11 chips, 11 sprites, 0 emoji), v384-12.18 (315 lock sprites,
0 🔒 emoji).
- 22 partials are plan selector drift, not code regressions; deferred to
v3.8.5.
## Tagging
Per the notes file, this is ready for `git tag -a v3.8.4 037dc8c4 -m
"v3.8.4"` after merge — **not executed by this PR**.
## Review
Draft for user review. Will be marked ready / merged before tag.
---------
Co-authored-by: meshcore-bot <bot@meshcore.dev>
Re-submission of #1625 (which was merged early, then reverted in #1626)
— now with **all three round-1 reviews addressed** so it lands in one
hardened state instead of as post-merge follow-ups.
## What
Per-node **Reach** view: a standalone page (`#/nodes/{pubkey}/reach`) +
a node-detail section + `GET /api/nodes/{pubkey}/reach`. It shows which
nodes a node has a **stable two-way RF link** with, derived from raw
`path_json` adjacency (a path travels origin→observer, so `[A,B]` ⇒ B
heard A). A link is bidirectional when both directions have
observations; the **bottleneck** (weaker direction) rates two-way
reliability. Nodes are identified only by **unique 2–3 byte** path
prefixes (1-byte collides → excluded).
## Review fixes folded in vs #1625
**Performance (Carmack):** hard scan LIMIT (200k) + modest prealloc;
`json.Unmarshal` replaced by a single-pass `parsePathTokens` (100k-row
scan 2.2M→1.3M allocs, 344→203ms); memoized resolver; size-hinted maps
(attribution over 100k rows: 102 allocs); `context.Context` plumbed;
cache `RWMutex` + evict-oldest (no full wipe); singleflight dedup;
degree/rank from a 60s shared snapshot; bench rewritten (ReportAllocs,
1k/10k/100k, mixed-payload, isolated attribution).
**Correctness/safety + tests (Independent + Kent Beck):** pubkey
validation → 400; error logging instead of silent swallow (first_seen /
degree / marshal→500 / discarded rows); `public_key=?` index use;
canonical `PayloadADVERT`; `min()` builtin; documented cache-slice
immutability; mux ordering comment. New tests: scanReachRows decode,
3-byte token branch, non-advert first-hop guard, observer SNR
aggregation across rows, HTTP-level attribution (asserts non-zero
we_hear/they_hear), 400/404/blacklist/cache-hit.
**UI / a11y / Tufte:** in-map legend (tiers + thresholds); dropped the
colour+width double-encoding (constant width, colour-only); colour-blind
glyphs (●●●/●●/●) + tier title beside the bottleneck number; dark-theme
`--link-*`; lighter table (horizontal rules, sentence-case headers); map
built once + link layer updated in place on toggle (no flicker);
time-range no longer flashes a loader; `destroy()` generation guard;
statCard escaping; scoped `@media print` to `#nq-report`;
`fieldset/legend` + `for/id` toggles; `aria-pressed` / `aria-live` /
back-link `aria-label`; "distance (km)" + bottleneck tooltip + no-GPS
note; inline styles → CSS; decorative emoji removed.
**Docs:** api-spec documents the 5-min cache, 200k scan cap, and 400.
## Testing
- `cmd/server` full suite green; reach unit + endpoint + bench all pass.
- `eslint public/*.js` (no-undef) and the XSS-sink gate clean.
- E2E updated: request status checks + exact (non-tautological) toggle
assertions + hard map-render assert.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## TDD-history note (Kent Beck gate)
This branch carries production + tests together, not a fabricated
red→green sequence. That's deliberate: the branch was rebased onto
upstream and the intermediate SHAs were squashed, so reconstructing a
"failing-test-first" commit after the fact would be theatre, not
evidence — and rewriting history to stage it would be dishonest. The
behaviour is instead covered by a comprehensive, anti-tautological suite
(directional attribution edges, 3-byte token branch, non-advert
first-hop guard, observer SNR aggregation, HTTP-level attribution
asserting non-zero counts, scan-cap truncation, zero-reach 200-not-404,
companion mis-attribution, cache eviction). Requesting maintainer
acceptance of the work on test *substance* rather than commit
*choreography*; the net-new-UI exemption is not claimed for the server
endpoint.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: meshcore-bot <bot@meshcore>
Reverts #1625.
#1625 was merged before the round-1 reviews (Independent / Kent Beck /
Tufte) were addressed. Reverting to land it cleanly: a fresh PR will
re-add the feature with the perf pass, the backend correctness/safety +
test-coverage fixes, and the UI/a11y (Tufte) batch folded in, so it goes
through review in a single hardened state rather than as a string of
post-merge follow-ups.
No functional loss — the feature returns in the replacement PR.
## What
Adds a per-node **Reach** view that answers "how well does this specific
node hear, and get heard by, its neighbours?" — both as a standalone
page (`#/nodes/{pubkey}/reach`) and as a section on the node detail
page.
New endpoint: **`GET /api/nodes/{pubkey}/reach`**.
## What it measures
For the target node it derives, from raw `path_json` adjacency (a path
travels origin→observer, so in `[A,B]` B received A directly):
- **Directional link counts** per neighbour: `we_hear` (how often we
received them) vs `they_hear` (how often they received us).
- **Bidirectional / bottleneck**: a link is two-way stable when both
directions > 0; the weaker direction is the bottleneck and rates real
two-way reliability.
- **Importance**: neighbour degree + rank, relay-observation volume,
bidirectional-link count, direct-observer count.
- **Direct observers**: who received the node at 0 hops, with SNR.
Reliability rule: a neighbour is only attributed when its pubkey
**prefix is unique** at the path's byte length (collisions are skipped,
never misattributed).
## UI
- Standalone Reach page + node-detail section.
- Reusable bidirectional link map (OSM) with links coloured by
bottleneck.
- Incoming/outgoing toggles to isolate each direction.
## Naming note (deliberate, no collision)
This is distinct from the existing **per-observer reachability** in
topology analytics (`ReachNode` / `ObserverReach` / `perObserverReach`).
This PR adds its own `NodeReach*` response structs in a new
`node_reach.go` and a new `/api/nodes/{pubkey}/reach` route — there are
no symbol or route collisions (verified: `go build ./...` clean). Happy
to rename to disambiguate further (e.g. "Link Quality") if you'd prefer
to reserve "Reach" for the per-observer feature.
## Testing
- `cmd/server`: endpoint shape/404/limit-clamp + unit tests for token
derivation and directional attribution, plus a scan benchmark — all
pass.
- Frontend: helper tests + Reach-page E2E (`test-node-reach-e2e.js`),
standalone route + incoming/outgoing toggles.
- `go build ./...` and `eslint public/*.js` (no-undef) clean.
## Docs
Design spec, implementation plan, and the `GET
/api/nodes/{pubkey}/reach` API contract are included under `docs/`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
List of changes too long to describe, so I'll hit high level.
- Config now supports the json map tiles that were suggested by
@Kpa-clawbot.
- Leaflet map layer button appears in the top right of live.js and
map.js (because all the work was already done on live.js... Added bonus)
- Allows users to enter creds for OSM and Stamen to get enterprise
related perks, in the config file
- Added a default light map under customizer. Still suggest removing
them all together and relying on the config
- You can enable OSM and Stamen in the config without a license, but at
your own risk!!!
- Config comment explains where to register and the providers for osm,
as well as the general limits per X interval
- Updated tests (28) to address the changes made to the maps
### TDD Exemption
**Reason**: Net-new UI surfaces (per `AGENTS.md`)
This PR introduces a net-new UI surface (the multi-provider map tile
selector). Under the `AGENTS.md` exemption for net-new UI surfaces, the
absence of an initial failing (red) commit is permitted, as the UI was
built first. However, the underlying public APIs are fully covered.
The following tests serve as the first assertions for these new APIs:
- `window.MC_createLayerControl`: Asserted in `MC_createLayerControl
handles Auto mode and explicit layers correctly`
- `window.MC_setDarkTileProvider` & `window.MC_getDarkTileProvider`:
Asserted in `MC_setDarkTileProvider persists to localStorage...`
- `window.MC_setLightTileProvider` & `window.MC_getLightTileProvider`:
Asserted in `MC_setLightTileProvider persists to localStorage...`
- `window.MC_initTileRegistry`: Asserted in `MC_initTileRegistry(true)
dispatches mc-tile-provider-changed`
- `applyTileFilter`: Asserted in `applyTileFilter sets invert CSS for
inverted dark provider...`
- Cross-tab synchronization: Asserted in `Cross-tab storage event
re-dispatches mc-tile-provider-changed`
Closes#1561. Follow-up to #1551.
## Why
#1551 added `Cache-Control: no-store` to all `/api/*` responses. That's
sufficient for CDNs that honour origin headers (Varnish, nginx). It is
**not** sufficient for Cloudflare zones where Cache Rules / Page Rules
override origin Cache-Control.
Field evidence from the meshat.se diagnosis (2026-06-04): observers
behind Cloudflare were returning `cf-cache-status: HIT` with `age` up to
~6 hours despite the origin emitting `no-store`. The CDN was caching per
zone policy and ignoring the upstream directive — exactly the failure
mode #1551 cannot reach. The application has no way to inject CDN rules;
the only durable fix is operator-side.
This PR makes that operator step discoverable and verifiable.
## What
### Server-side detection (log-only)
`cmd/server/cdn_detection.go` adds a middleware wired into the `/api/*`
chain after `noStoreAPIMiddleware`. On the **first** request bearing any
CDN-typical header (`CF-Connecting-IP`, `CF-Ray`, `X-Forwarded-For`,
`X-Real-IP`, `Fastly-Client-IP`, `True-Client-IP`) it logs:
```
[security] WARNING: detected request via CDN (CF-Ray header present).
Ensure /api/* is bypassed in your CDN config — see docs/deployment-behind-cdn.md.
Cached API responses cause observer-flap and incorrect dashboards.
```
`sync.Once` guarantees the warning fires at most once per process boot.
The middleware never blocks, never modifies the response, never adds
headers. Detection is observational only — operators who run behind a
CDN without bypass have a real bug; the warning is appropriate.
### Operator documentation
`docs/deployment.md` gains a new **"Behind a CDN (Cloudflare, Fastly)"**
section covering:
1. Curl verification command + healthy vs unhealthy output examples
2. Cloudflare Cache Rule creation (URI Path starts-with `/api/` → Bypass
cache)
3. Legacy Page Rules equivalent
4. Fastly note
5. Re-verification
6. Meaning of the startup log warning
7. Why we can't fix this server-side
`docs/deployment-behind-cdn.md` is the canonical path the log message
references — it's a short TL;DR that links back to the full section.
### Healthcheck script
`scripts/check-cdn-bypass.sh` — POSIX sh, no dependencies beyond curl +
grep + awk. Operators run:
```sh
scripts/check-cdn-bypass.sh https://your-domain.example.com
```
Exits `0` with `OK: no CDN caching detected ...` or `1` with a precise
diagnostic naming the offending header (`cf-cache-status: HIT` or stale
`age`).
## TDD
- **Red commit `e90ccaba`** (`test(security): RED ...`) —
`cmd/server/cdn_detection_test.go` (4 Go tests + 6 subtests for each
header) and `scripts/test-check-cdn-bypass.sh` (3 shell harness cases).
Middleware stub returns `next` unchanged so tests compile and fail on
assertions, not build errors.
- **Green commit `5e6a60b5`** (`feat(security): GREEN ...`) — real
middleware, wiring in `routes.go`, healthcheck script, doc.
## Deliverables
| File | Status | Purpose |
|------|--------|---------|
| `cmd/server/cdn_detection.go` | new | middleware + sync.Once warning |
| `cmd/server/cdn_detection_test.go` | new | 4 Go tests (1 stand-alone +
1 silence + 1 once + 1 table-driven over 6 headers) |
| `cmd/server/routes.go` | modified | `r.Use(cdnDetectionMiddleware)`
after no-store |
| `docs/deployment.md` | modified | TOC entry + "Behind a CDN" section |
| `docs/deployment-behind-cdn.md` | new | canonical path referenced by
log message + script output |
| `scripts/check-cdn-bypass.sh` | new | operator-runnable healthcheck |
| `scripts/test-check-cdn-bypass.sh` | new | shell harness with fake
curl |
## What this PR explicitly does NOT do
- Does not block requests based on CDN detection (log-only).
- Does not enforce CDN bypass (impossible — operator-controlled).
- Does not spoof, strip or modify CDN headers.
- Does not add CSP / HSTS / other security headers (out of scope).
- Warning is not configurable — operators behind a CDN without bypass
have a real bug, surfacing it is correct.
## Verification
- `go test ./...` in `cmd/server/` — full suite green.
- `sh scripts/test-check-cdn-bypass.sh` — 3/3 pass.
- Preflight checklist — all 11 gates clean (PII, branch scope, red
commit, CSS vars, CSS self-fallback, LIKE-on-JSON, sync migration,
async-migration annotation, XSS sinks, img/SVG ratio, themed-img/SVG,
fixture coverage).
---------
Co-authored-by: openclaw-bot <bot@openclaw.local>
Co-authored-by: clawbot <bot@clawbot.invalid>
## Summary
- Adds configurable GPS polygon areas to `config.json`; nodes are
attributed to an area if their last-known position falls inside the
polygon
- New `Area: …` dropdown filter (matching the existing region filter
style) appears on all analytics, nodes, packets, map, and live screens
when areas are configured
- Backend resolves area membership with a 30s TTL cache; area filter
bypasses the 500-node cap on `/api/bulk-health` so all area nodes are
always returned
- Includes a polygon builder tool (`/area-map.html`) for drawing and
exporting area boundaries
## Changes
**Backend**
- `AreaEntry` type + `Areas` config field
- `GetNodePubkeysInArea` DB query + `resolveAreaNodes` (30s TTL,
`areaNodeMu` RWMutex)
- `PacketQuery.Area` + `filterPackets` polygon check
- `?area=` param propagated through all analytics, topology,
clock-health, and bulk-health routes
- `/api/config/areas` endpoint
**Frontend**
- `area-filter.js`: single-select dropdown, persists to localStorage,
cleans up stale keys on load
- Wired into analytics, nodes, packets, channels, map, and live pages
- Live map clears node markers on area change
**Docs & tools**
- `docs/user-guide/area-filter.md` — configuration and usage guide
- `docs/api-spec.md` — updated with new endpoint and `?area=` param
table
- `tools/area-map.html` — polygon builder for defining area boundaries
- Demo areas added to `config.example.json`
## Test plan
- [x] No areas configured → filter dropdown does not appear on any page
- [x] Areas configured → dropdown appears, "All" selected by default
- [x] Selecting an area filters nodes/packets/topology/map correctly
- [x] Selecting "All" restores unfiltered view
- [x] Selection persists across page reloads (localStorage)
- [x] Stale localStorage key (area removed from config) is cleared on
load
- [x] `/api/bulk-health?area=X` returns all nodes in area (no 500-node
cap)
- [x] `/api/config/areas` returns correct list
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
@
## What this PR does
Implements region-scoped transport-route packet tracking with two
sub-features:
### Feature 1 — Scope statistics (`scope_name`)
- At ingest, transport-route packets (route_type 0/3) with Code1 !=
`0000` are HMAC-matched against configured `hashRegions` keys (mirroring
the `hashChannels` pattern). Matched region name (or `""` for unknown)
stored in new `transmissions.scope_name` column via migration
`scope_name_v1`.
- New `GET /api/scope-stats?window=` endpoint (1h/24h/7d, 30s
server-side TTL) returning transport totals, scoped/unscoped counts,
per-region breakdown, and time-series.
- New **Scopes** tab in Analytics with summary cards, per-region table,
and two-line SVG chart. Auto-refreshes every 60s.
### Feature 2 — Node default scope (`default_scope`)
- Per-node `default_scope` column on `nodes`/`inactive_nodes` (migration
`nodes_default_scope_v1`) tracks the most recently matched region for
each node, derived from transport-scoped ADVERT packets.
- `GET /api/nodes` response includes `default_scope` field when column
is present.
- Node detail panel displays the default scope badge.
- Async startup backfill (`BackfillDefaultScopeAsync`) populates the
column for nodes with pre-existing ADVERT data.
### Config
Add `hashRegions` to `config.json` (see `config.example.json`). One
entry per region name (with or without leading `#`).
@
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
## Summary
- **Backend**: adds `relayTimes` in-memory index (sorted unix-millis per
repeater pubkey), maintained in lockstep with `byPathHop`. Populated at
startup from all packet observations (not just best), updated on
ingest/evict/backfill. Exposes `relay_count_1h`, `relay_count_24h`,
`last_relayed` in both `/api/nodes` (for repeaters) and
`/api/nodes/{pubkey}/health`.
- **Frontend**: `getNodeStatus` extended to three-state (`relaying` /
`active` / `stale`) for repeaters based on relay_count_24h.
`getStatusInfo` is the single source of truth for status label,
explanation, and relay stats. Detail pane shows relay counts and last
relayed time. Nodes list gets a status emoji column with hover tooltip
showing relay info.
- **Correctness fixes**: relay index scans all observations per packet
(not just best); backfill now updates relay index after resolving paths;
pubkeys lowercased consistently throughout index.
## Changes
### `cmd/server/store.go`
- `relayTimes map[string][]int64` field added to `PacketStore`
- `addTxToRelayTimeIndex` / `removeFromRelayTimeIndex`: scan all
observations, idempotent sorted insert, lowercase keys
- `relayMetrics(times, nowMs)`: returns `(count1h, count24h,
lastRelayed)`
- `buildPathHopIndex`: populates `relayTimes` at startup
- `pollAndMerge`: updates relay index on ingest and eviction; new `else`
branch for path-unchanged observations
- `addTxToPathHopIndex` / `removeTxFromPathHopIndex`: lowercase resolved
pubkeys (fixes casing mismatch with lookup)
### `cmd/server/routes.go`
- `GetBulkHealth` / `GetNodeHealth`: include relay stats for repeater
nodes
- `handleNodes`: enriches repeater nodes with relay stats from
`relayTimes` so list view has same data as detail pane
### `cmd/server/neighbor_persist.go`
- `backfillResolvedPathsAsync`: calls `addTxToRelayTimeIndex` after
`pickBestObservation` to capture newly resolved pubkeys
### `public/roles.js`
- `getNodeStatus(role, lastSeenMs, relayCount24h)`: three-state logic
for repeaters
- `getStatusInfo(n)`: single source of truth returning status, label,
explanation, relay counts, last relayed
### `public/nodes.js`
- Detail pane: `n.stats` populated from health endpoint before
`getStatusInfo` call
- Nodes list: status emoji column with relay hover tooltip; status
filter uses `getStatusInfo`
### Tests
- `relay_liveness_test.go`: index functions, relay metrics, wiring
integration, bulk/single health endpoints
- `test-repeater-liveness.js`: three-state frontend logic, backward
compat
## Test plan
- [x] Repeater with recent relay traffic shows green relaying emoji in
list and detail pane
- [x] Repeater with no relay traffic in 24h shows yellow idle in both
views
- [x] Repeater not heard recently shows grey stale in both views
- [x] Non-repeater nodes unaffected (no relay stats, no status change)
- [x] Hover tooltip on list emoji shows relay count and last relayed
time
- [x] `go test ./...` passes
- [x] `node test-repeater-liveness.js` passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
## Summary
- Adds `public/geofilter-docs.html` — a self-contained, app-served
documentation page for the geofilter feature, matching the builder's
dark theme
- Updates the GeoFilter Builder's help-bar "Documentation" link from
GitHub markdown URL to the local `/geofilter-docs.html`
## Docs coverage
Polygon syntax, coordinate ordering (`[lat, lon]` — not GeoJSON `[lon,
lat]`), multi-polygon clarification (single polygon only), examples
(Belgium rectangle + irregular shape), legacy bounding box format, prune
script usage.
## Test plan
- [x] Open `/geofilter-docs.html` — dark theme renders, all sections
visible
- [x] Open `/geofilter-builder.html` → click "Documentation" → navigates
to `/geofilter-docs.html` in same tab
- [x] Click "← GeoFilter Builder" on docs page → navigates back to
`/geofilter-builder.html`
Closes#820
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Problem
`Load()` loaded all transmissions from the DB regardless of
`retentionHours`, so `buildSubpathIndex()` processed the full DB history
on every startup. On a DB with ~280K paths this produces ~13.5M subpath
index entries, OOM-killing the process before it ever starts listening —
causing a supervisord crash loop with no useful error message.
## Fix
Apply the same `retentionHours` cutoff to `Load()`'s SQL that
`EvictStale()` already uses at runtime. Both conditions
(`retentionHours` window and `maxPackets` cap) are combined with AND so
neither safety limit is bypassed.
Startup now builds indexes only over the retention window, making
startup time and memory proportional to recent activity rather than
total DB history.
## Docs
- `config.example.json`: adds `retentionHours` to the `packetStore`
block with recommended value `168` (7 days) and a warning about `0` on
large DBs
- `docs/user-guide/configuration.md`: documents the field and adds an
explicit OOM warning
## Test plan
- [x] `cd cmd/server && go test ./... -run TestRetentionLoad` — covers
the retention-filtered load: verifies packets outside the window are
excluded, and that `retentionHours: 0` still loads everything
- [x] Deploy on an instance with a large DB (>100K paths) and
`retentionHours: 168` — server reaches "listening" in seconds instead of
OOM-crashing
- [x] Verify `config.example.json` has `retentionHours: 168` in the
`packetStore` block
- [x] Verify `docs/user-guide/configuration.md` documents the field and
warning
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Closes#919
## Summary
Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.
## Changes
### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this
### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor
### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility
### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM
### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides
## Migration path for existing databases
1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration
## Test results
```
ok github.com/corescope/server 19.448s
ok github.com/corescope/ingestor 30.682s
```
---------
Co-authored-by: you <you@example.com>
## Summary
- Add missing `geo_filter` block to `config.example.json` with polygon
example, `bufferKm`, and inline `_comment`
- Add `docs/user-guide/geofilter.md`: full operator guide covering
config schema, GeoFilter Builder workflow, and prune script as one-time
migration tool
- Add Geographic filtering section to `docs/user-guide/configuration.md`
with link to the full guide
Closes#669 (M1: documentation)
## Test plan
- [x] `config.example.json` parses cleanly (no JSON errors)
- [x] `docs/user-guide/geofilter.md` renders correctly in GitHub preview
- [x] Link from `configuration.md` to `geofilter.md` resolves
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Part of #669 — M2: Link the builder from the app.
- **`public/geofilter-builder.html`** — the existing
`tools/geofilter-builder.html` is now served by the static file server
at `/geofilter-builder.html`. Additions vs the original: a `← CoreScope`
back-link in the header, inline code comments explaining the output
format, and a help bar below the output panel with paste instructions
and a link to the documentation.
- **`public/customize-v2.js`** — adds a "Tools" section at the bottom of
the Export tab with a `🗺️ GeoFilter Builder →` link and a one-line
description.
- **`docs/user-guide/customization.md`** — documents the new GeoFilter
Builder entry in the Export tab.
> **Note:** `tools/geofilter-builder.html` is kept as-is for
local/offline use. The `public/` copy is what the server serves.
> **Depends on:** #734 (M1 docs) for `docs/user-guide/geofilter.md` —
the link in the help bar references that file. Can be merged
independently; the link still works once M1 lands.
## Test plan
- [x] Open the app, go to Customizer → Export tab — "Tools" section
appears with GeoFilter Builder link
- [x] Click the link — opens `/geofilter-builder.html` in a new tab
- [x] Builder loads the Leaflet map, draw 3+ points — JSON output
appears
- [x] Copy button works, output is valid `{ "geo_filter": { ... } }`
JSON
- [x] `← CoreScope` back-link navigates to `/`
- [x] Help bar shows paste instructions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Fixes#702 — `.env` file `DISABLE_MOSQUITTO`/`DISABLE_CADDY` ignored
when using `docker run`.
## Changes
### Entrypoint sources `/app/data/.env`
The entrypoint now sources `/app/data/.env` (if present) before the
`DISABLE_*` checks. This works regardless of how the container is
started — `docker run`, compose, or `manage.sh`.
```bash
if [ -f /app/data/.env ]; then
set -a
. /app/data/.env
set +a
fi
```
### `DISABLE_CADDY` added to compose files
Both `docker-compose.yml` and `docker-compose.staging.yml` now forward
`DISABLE_CADDY` to the container environment (was missing — only
`DISABLE_MOSQUITTO` was wired).
### Deployment docs updated
- `docs/deployment.md`: bare `docker run` is now the primary/recommended
approach with a full parameter reference table
- Documents the `/app/data/.env` convenience feature
- Compose and `manage.sh` marked as legacy alternatives
- `DISABLE_CADDY` added to the environment variable reference
### README quick start updated
Shows the full `docker run` command with `--restart`, ports, and
volumes. Includes HTTPS variant. Documents `-e` flags and `.env` file.
### v3.5.0 release notes
Updated the env var documentation to mention the `.env` file support.
## Testing
- All Go server tests pass
- All Go ingestor tests pass
- No logic changes to Go code — entrypoint shell script + docs only
---------
Co-authored-by: you <you@example.com>
- Release notes for 95 commits since v3.4.1
- OpenAPI/Swagger docs: /api/spec and /api/docs called out everywhere
- Deployment guide: new API Documentation section
- README: API docs link added
- FAQ: 'Where is the API documentation?' entry
- Test plans for v3.4.2 validation
## Summary
- **nodes.js**: `#/nodes?tab=repeater` and `#/nodes?search=foo` — role
tab and search query are now URL-addressable; state resets to defaults
on re-navigation
- **packets.js**: `#/packets?timeWindow=60` and
`#/packets?region=US-SFO` — time window and region filter survive
refresh and are shareable
- **channels.js**: `#/channels/{hash}?node=Name` — node detail panel is
URL-addressable; auto-opens on load, URL updates on open/close
- **region-filter.js**: adds `RegionFilter.setSelected(codesArray)` to
public API (needed for URL-driven init)
All changes use `history.replaceState` (not `pushState`) to avoid
polluting browser history. URL params override localStorage on load;
localStorage remains fallback.
## Implementation notes
- Router strips query string before computing `routeParam`, so all pages
read URL params directly from `location.hash`
- `buildNodesQuery(tab, searchStr)` and `buildPacketsUrl(timeWindowMin,
regionParam)` are pure functions exposed on `window` for testability
- Region URL param is applied after `RegionFilter.init()` via a
`_pendingUrlRegion` module-level var to keep ordering explicit
- `showNodeDetail` captures `selectedHash` before the async `lookupNode`
call to avoid stale URL construction
## Test plan
- [x] `node test-frontend-helpers.js` — 459 passed, 0 failed (includes 6
`buildNodesQuery` + 5 `buildPacketsUrl` unit tests)
- [x] Navigate to `#/nodes?tab=repeater` — Repeaters tab active on load
- [x] Click a tab, verify URL updates to `#/nodes?tab=room`
- [x] Navigate to `#/packets?timeWindow=60` — time window dropdown shows
60 min
- [x] Change time window, verify URL updates
- [x] Navigate to `#/channels/{hash}` and click a sender name — URL
updates to `?node=Name`
- [x] Reload that URL — node panel re-opens
Closes#536🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Zero-Config Defaults + Deployment Docs
Make CoreScope start with zero configuration — no `config.json`
required. The ingestor falls back to sensible defaults (local MQTT
broker, standard topics, default DB path) when no config file exists.
### What changed
**`cmd/ingestor/config.go`** — `LoadConfig` no longer errors on missing
config file. Instead it logs a message and uses defaults. If no MQTT
sources are configured (from file or env), defaults to
`mqtt://localhost:1883` with `meshcore/#` topic.
**`cmd/ingestor/main.go`** — Removed redundant "no MQTT sources" fatal
(now handled in config layer). Improved the "no connections established"
fatal with actionable hints.
**`README.md`** — Replaced "Docker (Recommended)" section with a
one-command quickstart using the pre-built image. No build step, no
config file, just `docker run`.
**`docs/deployment.md`** — New comprehensive deployment guide covering
Docker, Compose, config reference, MQTT setup, TLS/HTTPS, monitoring,
backup, and troubleshooting.
### Zero-config flow
```
docker run -d -p 80:80 -v corescope-data:/app/data ghcr.io/kpa-clawbot/corescope:latest
```
1. No config.json found → defaults used, log message printed
2. No MQTT sources → defaults to `mqtt://localhost:1883`
3. Internal Mosquitto broker already running in container → connection
succeeds
4. Dashboard shows empty, ready for packets
### Review fixes (commit 13b89bb)
- Removed `DISABLE_CADDY` references from all docs — this env var was
never implemented in the entrypoint
- Fixed `/api/stats` example in deployment guide — showed nonexistent
fields (`mqttConnected`, `uptimeSeconds`, `activeNodes`)
- Improved MQTT connection failure message with actionable
troubleshooting hints
Closes#610
---------
Co-authored-by: you <you@example.com>
## Summary
Addresses user feedback on #600 — two improvements to RF Health detail
panel charts:
### 1. Auto-scale airtime Y-axis
Previously fixed 0-100% which made low-activity nodes unreadable (e.g.
0.1% TX barely visible). Now auto-scales to the actual data range with
20% headroom (minimum 1%), matching how the noise floor chart already
works.
### 2. Hover tooltips on all chart data points
Invisible SVG `<circle>` elements with native `<title>` tooltips on
every data point across all 4 charts:
- **Noise floor**: `NF: -112.3 dBm` + UTC timestamp
- **Airtime**: `TX: 2.1%` or `RX: 8.3%` + UTC timestamp
- **Error rate**: `Err: 0.05%` + UTC timestamp
- **Battery**: `Batt: 3.85V` + UTC timestamp
Uses native browser SVG tooltips — zero dependencies, accessible, no JS
event handlers.
### Design rationale (Tufte)
- Auto-scaling increases data-ink ratio by eliminating wasted vertical
space
- Tooltips provide detail-on-demand without cluttering the chart with
labels on every point
### Spec update
Added M2 feedback improvements section to
`docs/specs/rf-health-dashboard.md`.
---------
Co-authored-by: you <you@example.com>
efiten