mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 12:37:33 +00:00
d083f9ade91dbcbc7aa577bbf7b2248ce21ebf09
107
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 n30nexandn30nex
|
c211ef7643 |
feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option. The Last Relayed row shows green **actively relaying** only when `last_certain_relay` falls within the existing `healthThresholds.relayActiveHours` window. Missing, invalid, future or expired evidence adds no idle/alive claim. Existing `relay_active`, attributed timestamps and counts retain their meanings. The optional node list/detail field uses the existing certain-relay path: an eligible observed frame, matching hop, unique prefix, no retired namesake and existing signature checks. Originating an advert does not count. Forwarding another node's advert can count, even with zero non-advert attributions. Adverts with an unknown origin cannot establish forwarding. Evidence uses transmission `first_seen`, not late observation receipt time, and follows node-membership eviction. Snapshots take one read lock over requested keys, O(page size), with no packet scan or new database query. Work per qualifying hop stays bounded. No new dependency or setting; configuration documentation and OpenAPI explain the distinction. The test-first history includes assertion-failing RED commits and GREEN fixes. The initial RED's nil-return compile stub is explicitly permitted by `docs/agents/TDD.md`. Local native server/ingestor suites, focused race checks, 809 frontend helper assertions and 70 browser cases pass. Browser coverage includes both themes and full/side/tablet detail views. [Review regression RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339) proves both malformed-origin failures. All three independent review roles pass at `d1fe79cb`. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728) passes on that exact commit, including all three browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
efitenandClaude Opus 5.5
|
d29957fbdc |
feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated observer still is not safe from the shared login: the hook answers `ignore` for the shared login, so EMQX's built-in database lets it publish under any observer's topic, including one whose owner has moved to a personal login. This PR lets an operator close that per migrated observer, before the shared login is removed altogether. ## Changes **users.db v8.** `observer_bindings.confirmed_at`: the moment the owner's personal login first published under that pubkey. - An auto-bind (the owner's first publish) is confirmed at once. - An admin assignment stays unconfirmed until the owner's login publishes under that pubkey; that publish sets it. - Reassigning to another user or unbinding clears it; reassigning to the same owner keeps it. - Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin` rows stay NULL. **Config.** `userManagement.observerLogins.sharedUsernames` (default empty): the broker's shared login names. A name starting with `obs_` is refused at startup. **Authz hook.** For a login in `sharedUsernames`, a publish under `meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is confirmed, whatever the number of topic segments (the ingestor attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey). Logged as `publish denied: user=<name> observer=<8 hex>: confirmed owner`. Every other request from a shared login keeps `ignore`, so the built-in database decides as before. Personal logins are unchanged. An orphaned binding (owner account deleted) does not count as confirmed. **Admin.** The observers table of the Observer logins tab gets a Confirmed column; the bindings JSON gets `confirmedAt`. **Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out of migrated observers" (what it denies, how to see `confirmedAt`, the risk below). `config.example.json` has the key. ## Risk for operators An observer that is confirmed but still runs a second uploader on the shared login loses that second uploader once its name is in `sharedUsernames`. The deny log line names the observer. ## Tests - `internal/users`: v7 to v8 migration on a real v7 database, auto confirm, admin assignment then owner confirm, non-owner and repeated confirm, reassign clears, orphan reassign. - `cmd/server`: the decision table (confirmed, unconfirmed, unbound, non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty and `client` iata, lowercase key; owner publish confirms, confirm failure still allows and retries; reassign, unbind and user removal clear the flag; the index load against a real store; config validation; the hook and admin endpoints end to end. - `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an auto-bound observer. - Local: `internal/users`, `cmd/server` (two Windows-only tests skipped, as on master) and `cmd/ingestor` green. The E2E assertion was not run locally; CI runs it. ## Not in this PR - Using `confirmed_at` for verified `/neighbors` reports: a separate PR, built on #2300. - Removing the shared login: still an operator step after migration. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
e149568c6c |
feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on
|
||
|
efitenandClaude Opus 5.5
|
2bb000f2aa |
feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge that first; until then this diff also shows its commit. ## Adaptive offline threshold `userManagement.notifications.adaptiveOffline` — **off by default**: ```json "notifications": { "enabled": true, "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 } } ``` The fixed `infraSilentHours` (72 h) treats every repeater the same. With this on, a watched node's `node.offline` window is learned from its own **certain** traffic (the #2261 rule): 1. Certain packets per hour over the week **before its last certain packet**. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online". 2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning. 3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert. 4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8). 5. Clamped to [`floorMinutes`, the role's silent window]. Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than `minObservers` observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes. For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone). ## Certain node analytics - Each `activityTimeline` bucket carries `certain`: the part that can only be this node (`certainFor`, the packet-level form of the `certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other endpoints are unchanged. - Availability and longest silence use certain buckets only; the longest silence now includes one **still running** (an outage in progress is the silence worth seeing). - The chart stacks certain (`--accent`) and uncertain (`--text-muted`) — existing theme variables, so no new customizer entry. Descriptions updated. ## Performance `CertainActivity` walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken **per node**: ``` BenchmarkCertainActivity-16 12.1 ms/op (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold) BenchmarkNoteCertainRelays-16 121 ns/op 0 allocs ``` Node analytics adds one `certainFor` per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets). ## Tests - `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed. - `CertainActivity` ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now. - Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune. - `certainFor`, `longestSilence` (incl. running silence), config defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket. - Frontend: `NodeAnalyticsTimeline.series` (new `tests/unit/test-node-analytics-certain-timeline.js`). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork. ## Validation Server on the freshened e2e fixture: analytics buckets carry `certain` (e.g. 89 of 90 packets certain for one repeater). **No browser screenshot**: Playwright isn't installed on this machine. ## Later (rule 8) `floorMinutes` / `minObservers` are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
0e3ce48cff |
feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210)
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into this branch (`47ce8264`), so the diff against master is this PR's own change: `ea4d1130` plus the fix in `5cbbe5d5`. ## What it does Two nodes hear each other only when SF and bandwidth match and their frequencies are close enough. `/api/rf-params` now also returns `deviations`: nodes whose setting differs from a majority of the nodes around them on the same band. - **Peers:** nodes with GPS and RF params within `rfParams.compareRadiusKm` (default 50) whose frequency is within 30 MHz. - **Same channel:** frequencies within a quarter of the bandwidth count as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a transmitter/receiver offset of up to 25% of BW without sensitivity loss, for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the [S62F module datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf), an SX1262 module; I did not find the figure in a Semtech document itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10, which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the tighter bound there. - **Flagged when:** a setting the node cannot hear is shared by more than half of the group, the node itself counted, with at least 3 peers and no other incompatible setting at the same support. So 5 against 5 flags nobody, and 2 of 4 does not count as a majority. Each deviation carries the majority, how many peers are compatible with it, the peer count, and which of freq/sf/bw differ. - **Coding rate is not compared.** RadioLib starts SX126x radios with an explicit LoRa header (`SX126x::begin` sets `RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the firmware pins in `platformio.ini`), and nothing in `firmware/src` switches to implicit. The header carries the CR, so a CR mismatch still decodes. I checked SX126x only, not SX127x or LR11x0. UI: - Nodes list: a badge such as `LoRa SF7` next to the name of a deviating node (the majority in its tooltip), and a `LoRa: differs from nearby (N)` filter kept in the URL as `?lora=differs`. - Node detail: one line, for example "Differs from 12 of 14 nearby nodes on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz". - Badge and filter only exist while the server has map data. ## Why same-band only On our instance (1,110 nodes with params and GPS, measured with the first version of the rule), a plain neighbourhood majority flagged exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The same-band comparison flags none there. On our network this PR shows nothing today. The flags are for instances where a node really is set to a different SF, bandwidth or channel than the nodes around it. ## Cost - Points are bucketed on a radius-sized grid (rows of radius/110.57 degrees, longitude columns widened by 1/cos(lat)) and compared with a squared equirectangular distance. - `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the same box): 388 ms, 1.3 MB. Both on the first version. - After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms, 10k 821 to 852 ms, 1.8 MB. On that machine the first version measured 7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable time. It is one pass over the distinct settings around a node, and those are few (one per preset in use). - Inside the radius the work is close to O(n²): 10,000 nodes in one dense box means about 2,000 neighbours per node, and a request after the 5 minute TTL waits for that computation. - The result is cached per map snapshot for 5 minutes; concurrent requests wait for one computation. - On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26 ms max (8 requests). ## Tests - Go: deviation cases (SF, freq within band, BW, CR ignored, other band ignored, no GPS, no params, far away), split vote and too few peers, radius, a peer in another longitude column at 60°N, the cache (same snapshot within TTL, new snapshot, after TTL), the route with four inserted nodes, plus the two benchmarks. - Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a 15 kHz offset are not deviations while 32 kHz is, and offset peers add up into one majority; the BW/4 boundary; peers two grid columns away at 60°N; a peer at the north edge of the radius. Each fails on the first version. The two-column test also fails with a ±1 column search, which the existing longitude test did not catch. - Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`. - Frontend unit: 6 new assertions for `deviationBadge` and the detail line (13 in the file). - E2E: 4 new checks: badge only on the deviating node, filter count and result, `?lora=differs` written and restored, detail line. The disabled check now also asserts no filter and no badges. 8/8 locally. ## Validation On staging with live map data, with the first version: 1,115 nodes served, `deviations: {}`, and the filter shows `LoRa: differs from nearby (0)` with no badges. Because no real deviation exists there, the badge, filter and detail line were checked by serving one simulated deviation through Playwright route interception against the staging page. ## Not done - No map marker for deviating nodes. - `compareRadiusKm` is config-only; the customizer is a later step. The other values are hardcoded and belong in that step too: at least 3 peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute TTL. - Not re-run on staging after `5cbbe5d5`, and the E2E suite was not re-run locally. The fix changes only the server's choice of nodes, not the UI. - The detail line says the peers "use" the majority setting. With the BW/4 tolerance, some of them may be a few kHz off it. - Companions and room servers are compared too, not only repeaters. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
dc98886d92 |
feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239)
Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - **Validation.** - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - **Retention.** `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - **Compatibility.** An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - **Which name wins.** The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - **Only repeaters heard directly.** A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - **Filters.** The blacklist and the hidden name prefixes apply to both sources. - **Missing table.** A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - **Gated** like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - **Which receptions count.** Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - **How the estimate is made.** - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - **Header search** adds matching self-reported repeaters, tagged "self-reported". - **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - **`lookupPrefixes`** adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - **`/api/nodes/reported`** makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - **The map does not wait for it.** It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - **Not benchmarked.** On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - **Map filters.** The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - **Name characters.** Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - **Not done:** the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - **New Go tests:** - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - **New frontend unit suites:** `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - **`test-all.sh`** passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - **E2E**, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - **Real data, API only.** On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - **Not done:** a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
efd2aeabb4 |
perf(neighbors): rebuild the neighbor graph only in the background (#2221) (#2222)
Fixes #2221. ## Problem `getNeighborGraph` rebuilt `Server.neighborGraph` on the request goroutine once it passed its 5 min TTL. It held `neighborMu` for the whole build, which takes 11.1-11.4 s on a production store. The 5 min recomputer went through the same staleness check. A tick that landed just before the TTL skipped the rebuild and left it to the next request. As a result `/api/nodes/{pubkey}/neighbors` ran at p95 10.4 s and max 11.0 s (57 requests). ## Change - **`getNeighborGraph`** serves the current graph. It builds only when there is none yet, which happens at startup. - **`rebuildNeighborGraph`** builds from the packet store whatever the age of the current graph, outside `neighborMu`, and then swaps the pointer. - `recomputeNeighborGraphCache` calls it first on every tick. - Without a store it keeps the current graph, so tests that inject a graph are unaffected. - **`neighborBuildMu`** serializes builds, so a startup request and the first tick do not both build. Freshness is now bounded by the recompute interval (`neighborGraph.cacheRecomputeIntervalSeconds`, default 5 min) plus one build. Before this change it was the 5 min TTL plus one build, whatever the interval. So the bound is the same only at the default 300: - with a longer interval (for example 3600), `/api/nodes/{pk}/neighbors` can serve a graph up to that old, where before it was about 5 min; - with a shorter interval (for example 30), every tick does a full build, where before there was at most one build per 5 min. The config comment and `config.example.json` now say this (commit `de7ae981`). `IsStale`/`neighborGraphTTL` stay, because `path_inspect.go` still uses them for `store.graph`. ## Perf - **Request path:** after the first build it does no build at all. It takes one mutex and reads one pointer. - **Background:** the recompute tick now always builds, once every 5 min. Before, it built only on the ticks that found the graph stale, and a request built in the other cases. - **Total:** the number of builds stays about the same. Before: `/api/perf` on production shows p95 10404 ms. After: the follow-up comment gives client-side curl timings from staging (40 requests, p95 0.204 s), not `/api/perf`. Staging builds the graph in 1.03 s against 11 s on production, so the two are different instances. ## Tests `neighbor_graph_rebuild_test.go`: - a stale graph is served without a rebuild; - a graph is built when there is none yet; - a recompute tick replaces a graph that is not yet stale; - a recompute without a store keeps the injected graph. The first and third tests fail on master. `go vet` and `go test .` pass in `cmd/server`, and `-race` passes on the neighbor-graph tests. No frontend change. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7c00a42947 |
perf(reach): answer /nodes/{pk}/reach from a rolling hop index, not a path_json LIKE scan (#2214)
Refs #2146 (reach item: "an indexed hop table would remove it"). Follows #2155. ## What changes Reach finds the observations that relayed through a node with `path_json LIKE '%"XX"%'` over every observation in the window. The cost follows the window, not the node: a quiet repeater costs about as much as a busy one. When the database does not fit in the page cache this outlives the 60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146 reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan reads the whole window, so cold it is far slower (below), and it holds one of the 4 read-pool connections the whole time. This PR adds an inverted index and uses it only when it is complete for the requested window: - `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A token lookup is a seek and the window bound is a range on the same b-tree. - Triggers on `observations` maintain it (insert, delete, `path_json` update), so prune and path-repair need no changes. - `observation_hops_coverage.since` marks how far back the index is complete. If the window starts before `since`, or the table is missing (older DB, backfill unfinished), reach runs the existing `LIKE` scan unchanged. - The ingestor owns the schema, a resumable newest-first backfill, and a daily trim to `retention.hopIndexDays` (default 14). The server stays read-only and `AssertReady` does not require the table. The backfill starts after the ingest gate opens rather than inside `OpenStore`. My first version did the latter and held live ingest in the buffer for about 15 minutes on a 13.7 GB database (nothing dropped). Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7 s per commit on EBS. ## Results ### Real data: vanilla vs this branch One snapshot of a real instance (11.5 GB, 10.7M observations over 14 days) served by both builds on the same machine (8 cores, 30 GB RAM, NVMe, warm cache). One cold request per cell, seconds, repeaters from busiest to quietest by 24 h relay count. | node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d branch | |---|---|---|---|---| | 7,314 | 24.5 | 4.5 | 8.3 | 9.6 | | 7,307 | 1.7 | 2.4 | 1.3 | 3.1 | | 7,047 | 3.3 | 2.2 | 2.0 | 2.3 | | 2,151 | 20.0 | 0.44 | 29.6 | 2.0 | | 673 | 23.7 | 0.18 | 28.5 | 0.62 | | 240 | 21.8 | 0.01 | 27.8 | 0.04 | | 52 | 21.6 | 0.00 | 28.4 | 0.00 | - Nodes with few relays drop from 20 to 30 s to under a second. - The busiest relays are about the same, sometimes slower: both builds spend their time loading tens of thousands of matching rows by primary key, which the index does not speed up. - Single runs, and page-cache state moves them a lot: an earlier run, with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s for 7 d, close to #2155's warm figure. Read orders of magnitude, not exact figures. Nothing failed on this fast machine. **Same answers.** Unit tests check indexed rows equal `LIKE` rows, fallback on partial coverage, and trigger upkeep. On real data, 21 node/window pairs (3 to 14 days) requested from both builds at the same moment were all identical once links were compared by public key. `LIKE` ignores case and the index does not; 0 of 10.1M observations in the window have a lower-case hop. Above `reachScanRowLimit` both paths truncate, and they can keep different subsets. ### Production instance t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations per 7 days, so the cache held a fraction of it. - Before: `days=7` on a busy node returned 500 (`context canceled`), and `days=1` took 58.5 s. - With the index (38 probes of four nodes over 18 hours): `days=7` on that node returned 200 in 23 s. The busiest probe node had a median of 0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The quietest had a median of 0.04 s. - On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache holds the database. Same index afterwards (493 probes): hottest node median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From 2026-10-03 the instance also ran a local change, not in this PR, that lets reach requests continue in the background, which only affects requests over 45 s.) So on a small cold host the index turned "always fails" into "usually works" but did not make the hottest relays reliably fast. That needs cheaper row loading (a compact side table measured 4 to 10x on cold reads; kept out of this PR) or background computation. ### Benchmarks (included) ``` go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x # 600k observations, 11 hops avg, 7 d window in 14 d, warm cache hot_token/like 816 ms rare_token/like 486 ms hot_token/indexed 293 ms rare_token/indexed 1.7 ms go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x # one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops without_index 23 us/op with_index 127 us/op ``` The triggers add about 100 us per observation, roughly 0.1% of a core at the production instance's 8 observations per second. They cap one writer at about 8k observations per second instead of 43k. The benchmark ran in a temp directory, so it shows CPU and b-tree cost, not disk sync cost. Over 13 days on the production instance, 0 observations were dropped and newest-observation age was p99 4 s. Four of 623 samples were 20 s or more: 91 s and 20 s came within six minutes of a container restart, 31 s came 24 hours after one (when the ingestor's daily tickers fire), and 23 s had no restart nearby. Index coverage stayed between 14.0 and 15.2 days. **Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M observations in the window, about 196 bytes each, next to 6.81 GB for `observations`. It scales with hops per observation and `hopIndexDays`. Backfill of 77M rows took 15.5 min on the production instance. **Perf justification (AGENTS.md).** Reach goes from a scan over the whole window to a seek over the matching rows. Everything new is bounded: the index by `hopIndexDays` and the daily trim, the backfill by chunk size and a cancellable context, and the server adds no in-memory state. The ingest hot path pays the trigger cost above. ## Design notes and open questions - Why not the in-memory store: it defaults to 7 days (`retentionHours: 168`) and reach offers up to 30, so it could not answer the longer windows. The other option listed in #2146, driving the query from `observations.timestamp`, is what #2155 already did. - SQLite only: the triggers use `json_each`. The optional PostgreSQL backend in #2145 would need its own equivalent. - No off switch: the table and triggers are always installed and `hopIndexDays` only sets the window. If you would rather have this opt-in, or want a smaller default for small disks, I'll change it. ## Operator notes - Upgrade: the ingestor installs the table and triggers on start and backfills. Until a window is covered, reach behaves as today. - Rollback: an older binary keeps indexing new rows with no trim. To remove it with the ingestor stopped: `DROP TRIGGER trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE observation_hops_coverage;` - New key `retention.hopIndexDays`, documented in `config.example.json`. ## Tests `go test ./...` passes in `cmd/server`, `cmd/ingestor` and `internal/dbschema`. New: `node_reach_hops_test.go`, `observation_hops_test.go`, `observation_hops_backfill_test.go`, and the two benchmarks. ## Update after review (commit `2291bea2`) The text above is what was reviewed at `95e17976`. Where they differ, this section is current: - **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the value in `config.example.json`) installs nothing. A value above 0 installs the table and triggers and backfills that many days. This replaces "default 14", the "No off switch" note and the Upgrade note above. - **Changing the setting** applies at the next ingestor start: raising it backfills the gap, lowering it trims, and 0 removes the triggers and both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at startup before MQTT connects. - **Case.** Hops are stored and matched upper-cased, so the index ignores ASCII case the way LIKE does. This replaces the sentence on case under "Same answers". - **Rollback.** Set it to 0 and start the ingestor once, then roll back. The manual `DROP` list above still works with the ingestor stopped. - **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M observations, about 224 bytes each, in 13 min 37 s. - Backfill and trim are serialized, and the chunk SQL lives in `dbschema` next to the triggers. ## Update after review, round 2 (commit `95690e8c`) Where this differs from the two sections above, this one is current: - **Startup order.** The index is installed or removed in the startup block, after the MQTT subscription is up and before the ingest gate opens, so packets arriving during a removal are buffered (`ingestBufferSize`), not lost. This replaces "at startup before MQTT connects" in the first update. - **Mark moves cover their range.** Each backfill and trim transaction indexes or deletes everything between the old mark and the new one, so a row arriving between the chunk-picking read and the transaction is accounted for either way. - **Negative `hopIndexDays`** is a config error at startup; the ingestor refuses to run. This replaces "−5 is off" in the first update. - **Server.** If the index disappears between the coverage check and the scan, the request is answered from the `LIKE` scan instead of a 500. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
cab4f71c64 |
feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209)
Fixes #1268. Frequency, spreading factor, bandwidth and coding rate are per-device settings that never travel in a packet. The public MeshCore map is the only source. This PR adds them to the node detail views, opt-in, and answers the open policy questions from the triage comments. ## Policy decisions | Question (triage) | Answer in this PR | |---|---| | Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl` empty (default) = nothing fetched, `/api/rf-params` returns `enabled: false`. Same model as `knownChannelsUrl`. | | 33 MB every 6 h, thundering herd | Random 0-60 s delay before the first fetch, then every `refreshHours` (default 6). The map sends gzip (17 MB on the wire). It sends no ETag or Last-Modified, so conditional requests are not possible. | | Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding it fails the fetch instead of parsing a truncated body. | | Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF 5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows are dropped. A body that is not a well-formed array fails the fetch. | | Keep stale cache on anomaly | Any failure keeps the previous snapshot; a failed presets fetch keeps the previous presets. `fetched_at` is returned so an outage is visible rather than hidden. | | Freshness | Only adverts within `maxAgeDays` (default 7) are kept at fetch time and re-checked at serve time. | | TLS pinning | Not done: Go's certificate validation plus the field validation above. | ## What changes - `cmd/server/rf_params_cache.go`: stream decoder, validation, cap reader, preset parser (presets with identical values are merged, titles joined), background refresh. In memory only, no DB writes, so the read-only server invariant holds. - `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`; `nodes` holds only nodes this instance knows, keyed by lower-case pubkey. `/api/nodes` is unchanged, so its payload does not grow for every consumer and the 500-row paging is unaffected. - Node detail (side panel and full page): `LoRa` row with the four values, the preset name when all four match one, and the age of the map advert. Shown only when the feature is on and the server has fetched the map at least once. - No column in the Nodes table, on purpose: on our network such a column showed one preset title in 814 of 1,366 rows and nothing in 443. Showing only the nodes that differ from their region is the follow-up PR. - `config.example.json` documents the block with the two URLs to use. ## Cost - Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52 s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days). Runs in a background goroutine every 6 h. - The endpoint walks the cached node list once with map lookups: O(nodes). - Frontend: one bulk fetch of `/api/rf-params` when a node detail opens (client TTL 5 min), not one per node. ## Tests - Go (`rf_params_cache_test.go`): parse with every rejection reason, malformed bodies (not an array, truncated, wrong type, empty), preset merge and invalid presets, cap reader (one byte over refused, exactly at the cap accepted), fail-soft against an `httptest` upstream (503, HTML body, presets-only failure), config defaults, route (disabled shape, only known nodes, entry aged past max age dropped at serve time). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). - Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions on the real file in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`). It checks the real disabled state on the fixture server (no detail row), then serves `/api/rf-params` through `page.route`: no table column, full detail page with preset, side panel with raw values, and the hint for a node that is not on the map. 4/4 locally, plus the existing node table and node detail E2E files. ## Validation On a staging instance with `rfParams` set to the map and preset URLs: - Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s after start. - `/api/rf-params` returned 1,115 of our nodes: 988 on 869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on SF7. - In the browser the detail row shows in the side panel and on the full page, and the Nodes table keeps its original columns. ## Not done - Highlighting nodes whose settings differ from their region's majority (a badge in the Nodes list plus a filter) is the follow-up (separate PR). In that follow-up a CR-only difference stays informational until it is confirmed whether MeshCore radios use the explicit LoRa header (then CR travels in the header and a mismatch still decodes). - `maxAgeDays` is config-only; the customizer is a later step. - No data is written to the DB, so the RF parameters are gone after a restart until the next fetch (at most 60 s plus the fetch time). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Adam
|
6d56f55168 |
fix(nodes): place nodes in regions by zero-hop home regions (#1879) (#2190)
Red commit: |
||
|
efitenandClaude Opus 5.5
|
1ae4c74913 |
feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208)
Fixes #661 (the narrowed scope from the 2026-10-14 comment). ## What changes - `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and `rx_airtime_pct_1h` per observer: the airtime share over the hour before now, whatever `window` is, from the cumulative `tx_air_secs` / `rx_air_secs` counters in `observer_metrics`. - A counter that goes down marks a reboot; only the stretch after the last reset counts. - Under 15 minutes of samples (one sample, or a reboot late in the hour) returns `null`. - Clamped to 0..100, rounded to 2 decimals. - The response also carries `duty_cycle_warn_pct`, from the new config key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit). Added to `config.example.json`. - RF Health tiles show `TX 1h: x%`; the observer list gets a sortable numeric `TX 1h` column. Both use the shared `public/observer-airtime.js` helper and flag values at or over the threshold with `.airtime-over` (`var(--status-red-text)`). - The handler now returns a named `MetricsSummaryResponse` struct instead of `map[string]interface{}` (one occurrence fewer). ## Why the warning matters Repeater and room-server firmware default to `airtime_factor = 1.0` (`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26` turns into a 50% TX budget per 1 h window. The device therefore does not keep itself under the EU 10% limit; this view is where an operator can see it. ## Cost - Server: one extra query per summary request, `SELECT observer_id, timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`, then one O(rows) pass. On our instance that is 35 observers x about 12 samples (5 min interval). - Observer list: one extra bulk call `summary?window=1h` per load (client TTL 30 s, failure ignored), not a call per observer. ## Tests - Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample, span under 15 min, reboot leaving under 15 min, nil counters, clamp, rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h ignored inside a 24 h window, single-sample observer gets null), `TestMetricsSummaryDutyCycleWarnPct` (default 10, config override, `observers` is `[]` not `null`). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows on clean master too). - Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8 assertions, real file in `vm`), added to `test-all.sh`. `test-observers-headings.js` updated for the new column. Full `test-all.sh` passes locally. - E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in `deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no `observer_metrics` table, so the summary is served through `page.route` with known values for the fixture's own observer ids. It checks the column header, over/under/unknown cells, sort order and the RF Health tile. 4/4 locally. It caught a real bug during development: `loadRFHealthData` declares a local `const window`, so the tile calls the helper as the global `ObserverAirtime`. ## Validation Deployed to a staging instance with live MQTT data (35 observers with metrics) and checked in the browser: - The observer list shows the column for 42 rows (33 with a value, 9 without enough samples). Sorting puts the highest share (5.3%) first. - RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line wrapped inside items on narrow tiles; the stats row now wraps between items only. - No observer was over 10% at the time, so the red state was checked by rendering one cell client-side at 12.5% (computed colour `rgb(185, 28, 28)` in light mode), and by the E2E test. ## Not done - The threshold is config-only; exposing it in the customizer is a follow-up. - No airtime comparison across observers (that is #671) and no per-packet airtime (Relay Airtime Share covers it). - Dark mode was not checked in the browser; `--status-red-text` is defined for both themes. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
CoderNemesisandCoderNemesis
|
d9342bbebd |
(feat): UserManagement, Add Support for Postal Email Delivery Provider (#2154)
This PR extends email support to add another email delivery provider called Postal for user management. It is a, open source, self hosted email delivery and reputation management service. It adds 2 new additional configuration settings to the userManagement.mail object which are "postalBaseUrl" and "postalApiKey". It also adds "postal" as a supported value for the "provider" setting. All existing functionality including email delivery notification is supported by Postal. Included test cases to support the new functionality and all pipelines pass. --------- Co-authored-by: CoderNemesis <rfedor@comchan.net> |
||
|
Alex BandClaude Opus 5.5
|
4e91c5cf0b |
perf(analytics): opt-in pauseWhenIdle skips recomputes nobody reads (#2164)
## Summary The analytics recomputers (#1240) recompute eleven snapshots (topology, RF, distance, channels, hash collisions, hash sizes, roles, two clock-skew views, retransmissions, direct-heard) on a fixed interval, 300 s by default, whether or not anyone reads them. That is the right trade for a busy instance: reads always hit a warm cache. On a quiet one it means most of the server's allocation goes on results nobody looks at. Measured on one production instance (2 vCPU / 2 GB): - topology, RF and hash-sizes were each **read 23 times in 45 hours** while being recomputed every 5 minutes; - `analyticsRecomputer.runOnce` accounted for **30% of all bytes allocated** (121 GB of 391 GB in 45 h); - garbage collection was 87% of the process's CPU (30 s CPU profile: `gcDrain` 39.9 s of 45.9 s sampled), with an 805 MB live heap against a 1 GiB `GOMEMLIMIT`, so allocation rate translates directly into collector time. This PR adds an **opt-in** `analytics.pauseWhenIdle`: - a recomputer's tick is skipped when nothing has read its snapshot since the previous compute, and the recomputer is marked paused; - the next read still returns immediately with the snapshot it holds (`Load` stays non-blocking, as #1240 requires), and makes a non-blocking send that wakes the loop to recompute; reads after that get the fresh snapshot and the ticker resumes from there. **Default is off**, so behaviour is unchanged unless an operator turns it on. The trade-off for operators who do: the first read after a quiet spell can show a snapshot older than the interval (refreshed seconds later). It is documented in `config.example.json`. No existing issue covers this. ## Config example Off by default. To turn it on, add one key to the existing `analytics` block in `config.json`: ```json "analytics": { "defaultIntervalSeconds": 300, "pauseWhenIdle": true } ``` Nothing else changes: per-endpoint `recomputeIntervalSeconds` still apply while an endpoint is being read. There is no UI change, so no screenshot. ## Performance Staging A/B: the same 30 minutes of real production API traffic (5,640 requests from a day's nginx log, replayed at 2x against a copy of the production database with the live MQTT feed), on `master` and on this branch with `pauseWhenIdle: true`, each after a restart and warm-up. Allocation is the delta of `/debug/pprof/allocs` across the replay: | during the replay | master | this PR | |---|---|---| | allocated by `analyticsRecomputer.runOnce` | 459.7 MB | **53.4 MB (-88%)** | | total allocated by the process | 3,704 MB | 3,372 MB (-9%) | | GC cycles | 52 | 41 | | requests: p50 / p95 / errors | 9.9 ms / 57.0 ms / 0 | 10.4 ms / 60.1 ms / 0 | The replayed traffic included real analytics reads, so the paused recomputers were woken by them; the remaining 53 MB is that. Over 15 minutes each recomputer only ticks about three times, so on a longer quiet period the saving grows with the idle time. GC count and CPU on this shared host vary between runs on their own, so the allocation figures are the claim, not the GC count. There is no Go benchmark for this change: the behaviour is "how many computes happen in a period of no reads", which the tests below assert directly (16 computes on master vs at most 2 here over the same idle period). ## Tests - Red commit `24056c57`: config field, accessor stub and an unused recomputer flag, plus tests. Three fail on the stub: - `TestAnalyticsRecomputer_PauseWhenIdleSkipsUnreadTicks`: an unread recomputer with a 20 ms interval computes at most twice in 300 ms (16 times on the stub); - `TestConfig_AnalyticsPauseWhenIdle`: `pauseWhenIdle: true` enables it, absent or nil config leaves it off; - `TestStartAnalyticsRecomputers_AppliesPauseWhenIdle`: all eleven recomputers receive the flag. - Also added, passing on both commits on purpose: a read on a paused recomputer returns the existing snapshot and triggers a refresh; a recomputer that is being read keeps ticking; the default (off) keeps ticking unread. - Green commit `19de3636`: the behaviour, `config.example.json` entry with `_comment_pauseWhenIdle`, and the `Load` doc comment. - `go test -race` on the recomputer, warm-up (#1659) and config tests, three runs: pass. `make test`: 16 of 16 modules pass. The frontend tests that read `config.example.json` pass. ## Review notes - `roles` reads the nodes-clock-skew snapshot inside its own compute, which counts as a read, so that dependency keeps its source fresh while roles is in use. - The #1659 warm-up gate is untouched: a skipped tick never marks a first pass done. - Config documented per WORKFLOW.md section 7. Customizer (AGENTS.md rule 8): it is an operator setting, not a display value; no customizer entry proposed. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - No issues on the added lines. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
352a0ccd19 |
feat(region-filter): configurable region quick picks (#2170)
## Summary **Independent of every other open PR.** Choosing a pick saves the selection, redraws the control and notifies the page's listeners itself, exactly as a click in the control does, so it re-queries straight away whether or not #2168 (`setSelected` notifies) is merged; with both in, nothing fires twice. Region quick picks are named groups of region (IATA) codes, offered as one-tap choices inside the region filter, e.g. "Northern California" for a dozen airport codes instead of ticking each box. One deployment has been injecting these into the page through an nginx `sub_filter` and a script that wraps `window.RegionFilter`; this makes them a configured feature. ```jsonc "regionQuickPicks": [ { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] } ] ``` - **Server:** `GET /api/config/region-quick-picks` returns the groups cleaned up: names and codes trimmed, codes upper-cased and de-duplicated in their configured order, groups without a name or codes dropped, at most 20 groups of 200 codes, always a list. `/api/config/regions` is unchanged. - **Region filter:** loads the groups with the regions and offers every group with at least one observer today, inside the region control: - dropdown layout: rows of the same kind as **All** (checkbox and bold name), directly under All with a divider before the single regions; the row is ticked while that pick is the selection, ticking it selects the pick and closes the menu, unticking goes back to all; - pill layout: the picks lead the same bar, set off from the single regions by a thin divider. - A tap selects the group's codes that have an observer and re-queries straight away. - The selector then reads the group's name ("South Bay ▾") instead of "2 Regions", and the button shows as pressed. Selecting the same codes by hand does the same. - Tapping the active group again goes back to all regions. - A failed fetch, or no configured groups, leaves the filter exactly as before. - **Mobile:** the rows take the menu's 44 px row height under `(pointer: coarse)`; the pill-bar picks reuse `.region-pill` and its touch target. Inside `.filter-bar` (Packets) the existing touch rule for inputs made every region checkbox a 44 px square; the checkbox is now 20 px and the 44 px row is the tap target. - **Semantics:** like every region filter, a group means "heard by observers in these regions", not "located there"; `config.example.json` says so. No existing issue covers this. ## Screenshot Nodes page with two picks configured (example config below), after tapping **South Bay**: the pick and its codes (SJC, MRY) are selected, and the list re-queried for them. Taken against `test-fixtures/e2e-fixture.db`.  ## Config example ```json "regionQuickPicks": [ { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] }, { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] } ] ``` Each pick needs a `name` and at least one code; `description` is optional (shown as the tooltip). Codes are trimmed, upper-cased and de-duplicated; a pick with no code that has an observer today is not shown. At most 20 picks of 200 codes. ## Browser validation The branch's server was run locally against a migrated copy of `test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless Chromium: | check | desktop 1280 px | phone 390 px (touch) | |---|---|---| | groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) | same | | tap South Bay selects | SJC, MRY (SCZ has no observer) | same | | next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` | | selector label | "South Bay ▾" | "South Bay ▾" | | second tap | back to all regions | same | | button height | 24 px | 44 px | | page wider than viewport | no | no | | page errors | none | none | Checked on Packets (dropdown layout) and Nodes (pill layout). Three layout problems found this way on the phone were fixed before opening: a sideways-scrolling row clipped at the screen edge, the selector squeezed into a narrow column (the filter group is a non-wrapping flex row), and the two wrapped rows spread apart on Nodes (`align-content`). **Revision 1:** the first version put the picks in a separate row beside the selector; they moved inside the dropdown. Re-checked with a 14-region list and three picks on Packets: desktop 1280 px dark and phone 390 px light both keep the menu on screen (menu 340 px wide, page not wider than the viewport), the pick closes the menu and the trigger names the pick. **Revision 2:** picks are checkbox rows like **All** instead of chips. Re-checked on Packets at 1280 px dark and 390 px light: rows render bold under All, the divider separates the single regions, ticking a pick ticks its codes and names the trigger, no sideways scroll. Unrelated and not changed: on Nodes at 390 px the region row (existing pills included) starts flush with the screen edge. ## Tests - Red commit `81ea67d6`: the config type, a stub normaliser and endpoint returning nothing, and tests. - Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups), caps, nil config, and the endpoint's JSON. - Frontend, `tests/e2e/test-region-quick-picks.js` (moved from `tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job like `test-table-sort.js`): the real `region-filter.js` in **jsdom** against faked config endpoints. Nine cases: rendering and hiding groups with no observer, selecting and re-querying, the selector label, toggling back, a hand-made selection marked active, the pill layout, no groups, a failed fetch, HTML escaping. Seven fail on the stub, all on assertions. - Green commit `0e33210e`: the implementation, styles, `config.example.json` (`regionQuickPicks` + `_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route list (`TestOpenAPICompleteness` requires it). The jsdom test also had a cross-realm comparison bug fixed in this commit (`Array.from` on arrays coming out of the jsdom window). - `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. - Red commit `f991060d` (revision): picks live inside the dropdown menu before the checkboxes, choosing one closes the menu, and in the pill layout they share the region bar; two fail on assertions against the old layout. Green commit `1f720e35`. Frontend 195 of 196 (same environment-only failure). - Red `134f5256` / green `da0adb50` (second revision): picks are checkbox rows between All and the single regions with a divider, no chips in the menu; ticking and unticking a pick row. The green commit also repairs `style.css`: `1f720e35` had prepended its block to the top of the file (its edit matched an earlier `.region-pill {`) and left the previous rules in place. The file now differs from `master` by 13 lines. ## Review notes - Groups are offered only when at least one of their codes appears in `/api/config/regions`, and only those codes are selected. The stored selection is pruned to known codes on load anyway, so selecting absent codes would not survive a reload. - Customizer (AGENTS.md rule 8): groups are operator configuration; a customizer editor could come later. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `b84ace4e`: - `normalizeQuickPickCodes` split out of the config normaliser (S3776, 17 > 15); - a comment on why a failed quick-pick fetch is ignored (S2486); - `for…of` in `activePick` (S4138); - in the tests: `node:` specifiers, `includes()`, and an explicit string comparator for sorted selections (S7772, S7765, S2871). - **Kept on purpose:** - S2486 is still reported on that `catch`. Swallowing the error is the intended behaviour: the filter works as before, just without picks. - S1523 on the `vm` test loader, as above. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
0737f10a5d |
feat(notify): optional external node alerts feed and the node.external event (#2177) (#2178)
Closes #2177. ## What Node notifications can mail alerts computed outside CoreScope. With `userManagement.notifications.externalAlerts` configured, users can choose a new event, `node.external`, which is off by default and only offered when configured: ```json "notifications": { "enabled": true, "externalAlerts": { "url": "http://coverage-job/alerts.json", "maxAgeHours": 48, "label": "Coverage" } } ``` The feed: ```json { "generatedAt": "2026-10-11T01:35:12Z", "alerts": [ { "pubkey": "<64 hex>", "key": "SE", "text": "never heard to the south-east", "url": "https://example.org/coverage" } ] } ``` Mail lines: `<name>: Coverage: never heard to the south-east, <time>` when an alert appears, `<name>: Coverage resolved (SE), <time>` when it is gone from a fresh feed. ## How - `cmd/server/notify_external.go`: feed types, `parseExternalFeed` (RFC 3339 `generatedAt`, age check, per-alert validation), `fetchExternalFeed` (200 only, 1 MB cap, 10 s timeout). - `notify_eval.go`: `evalExternal`. Subjects are `<pubkey>/<key>`; a `<pubkey>/*` row is the per-node baseline, so the first fresh feed after watching stores the current alerts without a mail (like `foreign.new`), and later alerts mail at once. `External == nil` (not configured, nobody chose it, or stale) leaves every `node.external` state as it is. - `notify_service.go`: the feed is fetched only when configured and at least one user chose the event. Each new reason the feed is unusable is logged once, and so is the recovery. Mail text and link: the alert's `url`, otherwise the node page. - `notify_handlers.go`: `availableEvents` and the new `externalLabel` include the event only when configured; PUT answers 400 for `node.external` otherwise. - `internal/users/notify.go`: event constant, `OptionalNodeNotifyEvents` (canonical order: node, optional, admin), `RemoveWatch` also deletes the node's `<pubkey>/...` rows. - Account page label, OpenAPI descriptions, `config.example.json` comment, `docs/user-guide/accounts.md` section. Decisions on the three open questions in #2177: the generic name `node.external`; the resolved mail names the key instead of repeating the text, so there is **no `users.db` schema change** (a bump would also make older binaries refuse the file); one feed. ## Invariants - Read/write separation (#1283): the server only reads the feed over HTTP. State rows go to `users.db` through `internal/users`, like the other events. - A dead, garbled or old feed never mails "resolved": `TestEvaluateExternalStaleFeedChangesNothing` and `TestNotifierMailsExternalAlertsAndSkipsAStaleFeed` (HTTP 502 and a 72 h old feed in a row, then a fresh empty feed resolves). ## Performance The evaluator adds one pass over the stored states (already read every tick) plus, per user with the event on and per watched node, the node's current alerts. Bounded by `maxWatchesPerUser` (50) times the alerts per node. The feed is fetched once per tick (default 5 minutes) only when someone chose the event, capped at 1 MB. No packet-store lock is taken for it. Nothing changes for instances without `externalAlerts`. ## Tests - `cmd/server`: `go test` green locally (Windows; `TestSaveGeoFilterPreservesFileMode` skipped as it needs Unix mode bits). New: `notify_external_test.go` (feed parsing and limits, HTTP fetch 200/404/oversized, config resolution, evaluator baseline/new/resolved/returning/stale/not chosen/notifications off, notifier end to end with a fake feed, account routes). - `internal/users`: green, 2 new tests (optional event and canonical order, `RemoveWatch` deletes external subjects only for that node). - Frontend: `test-all.sh` green, 1 new test in `test-notifications-ui.js` (label from `externalLabel`, escaped). - `scripts/check-xss-sinks.sh --file public/notifications.js`: clean. ## Not done - No E2E test: the account page change is one label, covered by the unit test. - Not run against a live instance yet. The first feed in use is the coverage job of on8ar.eu (`alerts.json`, 29 alerts at the moment). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
c2a590b741 |
feat(companions): linked-only ingest, account Companions and My coverage (#2176)
Part 3 of 3 for #2173 (companion linking): the optional linked-only ingest filter, the account page and the "My coverage" toggle, plus the docs. Builds on #2174 and #2175 (both merged); `upstream/master` is merged in, so the diff shows only this part. ## Changes in this PR **Ingestor: linked-only ingest** (`cmd/ingestor/linked_companions.go`) - With `clientRxCoverage.requireLinkedCompanion` on (default off) and user management enabled, every `meshcore/client/<pubkey>/...` message whose pubkey is not in `companion_links` is dropped before any client handler runs, and counted (`client_unlinked_dropped` in the stats file). - The ingestor opens `users.db` read-only (`mode=ro`, raw SQL, no `internal/users` import), like the approved-channels reader, so the #1283 write invariant holds. `TestLinkedCompanionSetIsReadOnly` pins it. - The set refreshes every 60 s. A miss re-reads at most once per 5 s, so a companion linked a moment ago is accepted within seconds while a flood of unknown pubkeys costs one read per 5 s. - It passes everything until its first successful read, and when `users.db` has no `companion_links` table yet (a server older than this). After the first read it keeps the last set on a read error. It is a filter, not a security boundary: anyone with the shared broker credential can still publish under a linked pubkey. **Frontend** - Account page: device tokens show under Devices as "CoreDrive RX" with their label; a new Companions section lists linked companions (name, short key, linked since, last seen) with an Unlink button. - Coverage page (`#/rx-coverage`): a "My coverage" toggle for logged-in users (`mine=1`). Only the newest coverage reply is drawn, and a refused `mine=1` turns the toggle off. Under "My coverage" the page does not ask for the "nothing received" cells from #2172, and their legend entry is hidden. **Docs**: user guide (accounts: devices, companions), `docs/client-rx-coverage.md` (linking, linked-only ingest, the CORS exception), `config.example.json`, and the spec status. ## Performance The filter runs on every client message. Benchmarks with 1000 linked companions (12-core dev box, `go test -bench`, 3 runs each): | Path | Time | Allocations | |---|---|---| | linked pubkey (lock-free map lookup) | 54 ns/op | 0 | | linked pubkey, parallel | 11-14 ns/op | 0 | | unlinked pubkey inside the 5 s miss gap | 85-95 ns/op | 0 | | one `users.db` re-read | 0.38 ms | 3030 | The unlinked benchmark asserts that no extra `users.db` read happens. A re-read happens at most once per 5 s on a miss and once a minute from the refresh loop. ## Verification - `cmd/ingestor`: 15 new tests (filter off changes nothing, linked passes, unlinked dropped before every handler, miss refresh accepts a new link, the 5 s cap holds under a flood, unlink honoured after the periodic refresh, missing table and missing file, fail-closed only after the first read, read-only open, startup, stats counter) and 4 benchmarks. Full ingestor suite green. - Frontend unit tests for the account page (device tokens, companions list escaping, empty state, Unlink confirm and cancel, load errors) and `tests/unit/test-rx-coverage-mine.js`. - E2E `tests/e2e/test-user-management-e2e.js` gains a companion step (device token under Devices, a companion linked with a real Ed25519 signature under Companions, Unlink). Run locally against the e2etest build: 28/28. The coverage suites (`test-rx-coverage-gaps-e2e.js`, `-noise-`, `-viewport-`) still pass. - Frontend unit suite (`test-all.sh`) green. ## Not in this PR - Narrowing the "nothing received" cells to the caller's companions under "My coverage" (the track query takes one companion today). - Per-user broker credentials or ACLs, which would turn the filter into an access control (non-goal in the spec). --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
b2cda21a27 |
feat(server): companion linking routes, device tokens and bearer auth (#2175)
Part 2 of 3 for #2173 (companion linking): the server routes, bearer auth and the link handshake. Builds on #2174 (merged); `upstream/master` is merged in, so the diff shows only this part. Everything here is behind `userManagement.enabled` and answers 404 with it off. ## Changes in this PR **Device tokens and bearer auth** - `POST /api/auth/device-token`: email + password returns a bearer token, stored as a `kind = 'device'` session with a label. Same failure answers and the same rate limit as login. - Server-enforced scope for a bearer token: `/api/auth/me`, `/api/auth/logout`, `/api/account/companions*`, `/api/account/settings`. Everything else answers 403. - A bearer header only counts when there is no session cookie, so an auth proxy that adds its own `Authorization` header does not hide a logged-in browser. Cookies never accept a device token. - The sessions list on the account API carries `kind` and `label`; logout with a bearer token revokes that device. **Linking a companion** - `POST /api/account/companions/challenge` `{pubkey}` returns `{challenge, expiresAt, host}`: 32 random bytes, single use, 5 minutes, bound to the caller and the pubkey. - `POST /api/account/companions` `{pubkey, challenge, signature, name}`: the signature is the companion's Ed25519 signature over `corescope-link:<host>:<challenge>` (`internal/sigvalidate.VerifyMessage`). The challenge is consumed in every case. - A companion linked to another account moves to the caller (the newest proof wins). Both users get a `companion.transfer` audit row and the previous owner a mail. That mail is a security notice: it is sent whatever the node notification settings are. - `GET /api/account/companions` (with `lastSeenAt` from `client_receptions`) and `DELETE /api/account/companions/{pubkey}`. - Linking does not touch the synced `meshcore-my-nodes`: a companion is not a node to monitor. - Admin user detail lists linked companions; the account data export includes companions and device sessions. **Coverage and config** - `GET /api/rx-coverage?mine=1`: only coverage of the caller's linked companions (401 when logged out, 403 for a device token). With `gaps=1` (#2172) the reply carries no `gaps` member under `mine=1`, because the track query takes one companion and cannot narrow to a set yet (`TestRxCoverageMineOmitsGaps`). - `clientRxCoverage.requireLinkedCompanion` (default off) is read and reported in `/api/config/client`, which also announces `userManagement.companionLinking`, so CoreDrive RX shows its login only where linking exists. The ingestor side is part 3. - CORS: for origins in `corsAllowedOrigins`, the device-token routes alone also allow POST/PUT/DELETE and the `Authorization` and `Content-Type` headers. No credentialed CORS. Documented in `config.example.json`. - OpenAPI and `docs/api-spec.md` cover the new routes. ## Verification - 26 new tests in `cmd/server` and `internal/sigvalidate`, among them: `TestCompanionLinkFlow`, `TestCompanionLinkRejections` (signature by another key, malformed signature, challenge bound to another pubkey, unknown challenge; reuse and expiry are covered by the store tests in #2174), `TestCompanionTransfer`, `TestCompanionTransferMailIgnoresNotificationSettings`, `TestBearerScopedRoutes`, `TestBearerIgnoredWithCookie`, `TestDeviceTokenSharesLoginRateLimit`, `TestCORSBearerRoutes`, `TestCompanionRoutesAbsentWhenOff`, `TestRxCoverageMine`, `TestRxCoverageMineOmitsGaps` (fails without the `mine == nil` guard), `TestVerifyMessage`. - `cmd/server`, `internal/sigvalidate` and `internal/users`: `go vet` and `go test ./...` green, `gofmt` clean. - The linking flow runs on analyzer.on8ar.eu since 9 October 2026 with CoreDrive RX v1.19.0 (a real companion linked through the signed challenge). ## Not in this PR - The ingestor's linked-only filter, the account page (Devices, Companions), the "My coverage" toggle and the user docs: part 3. - Per-user broker credentials and per-user API keys (non-goals in the spec). --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
2456762179 |
feat: account data export and users.db backup (#2141)
A small follow-up to parts A to E of #2128: users can download all data the instance holds about them, and the server backs up `users.db`. Stacked on #2140. Review the commits after that branch. ## The situation - An instance with accounts holds personal data, but a user has no way to get a copy (GDPR articles 15 and 20). Self-delete already exists. - `users.db` is the only file the server writes, and nothing backs it up: `GET /api/backup` snapshots the analyzer database only. ## What this PR adds **Data export.** `GET /api/account/export` and a "Download my data" button on the account page give one JSON file with: - the profile, including activation and a pending email change; - sessions (times and user agent); - synced settings, own proposals, notification prefs and watches; - audit rows where the user is actor or target, with other accounts as an id only; - the mail log. Password hash, session and link tokens and the unsubscribe token are left out: they are credentials. Each export is audited as `user.export`. **users.db backup** - Daily `VACUUM INTO` snapshots through `internal/users`, on by default when user management is on. They go to `backups/` next to `users.db`, the 7 newest are kept, the directory is 0700 and the files 0600. Rotation only touches `users-<timestamp>.db` files and never the snapshot it just wrote. Config: `userManagement.backup {enabled, dir, keep}`. - `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to keep off the server; audited as `user.backup`. - `docs/user-guide/accounts.md` gains a restore procedure. It covers what a restore brings back (accounts deleted after the snapshot, old passwords, revoked sessions) and how to clean that up afterwards. Spec: [`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md). ## Verification - `internal/users` with `-race` and the `cmd/server` suite pass locally, 30 new Go tests. The file-mode tests skip on Windows. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 27 of 27 steps locally. - On our production instance since 8 October 2026; the first snapshot there was 143 kB. ## Not in this PR - Importing an export into another instance. - Off-site upload of the snapshots. The admin download covers that by hand. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
de364a8bb6 |
feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a watched node goes offline, comes back, or reports a low battery. Admins can add instance events: a new foreign node, and an observer going offline or back. This covers the per-user mail part of #775. Stacked on #2139 (part D, which builds on #2138). Review the commits after that branch. ## The situation A sysop learns that a repeater went silent, or that its battery is running down, only by opening CoreScope. #775 asks for notifications. Accounts (part A) now give a verified address to mail and a mailer with delivery status. ## What this PR adds **Events.** They use the instance thresholds, so a mail does not disagree with the node page: - `node.offline`: silent for the role's `healthThresholds` window. Last heard comes from the packet store, and repeaters and rooms count relayed traffic (#1598). - `node.battery`: advert telemetry below `batteryThresholds.lowMv`, recovered at `lowMv + 100`. - Admins only: `foreign.new` (once per node) and `observer.offline`. **Store.** Schema v5 with `notification_prefs`, `notification_watches` and `notification_state`. **Server** (only with `userManagement.notifications.enabled`) - A notifier next to the janitor, every `intervalMinutes` (5). The evaluator is a pure function of watches, prefs, stored states and node snapshots. - No burst after a restart or a new watch: a subject's first evaluation stores its state without mailing. The loop waits for the startup load. While ingest is stale (newest packet older than 30 minutes) the offline checks pause, and after recovery they wait one silent window. - One mail per user per check. Limits: 20 per user and 100 per instance per rolling 24 hours, 50 watches per user. A change over a limit is recorded and never mailed later. - Every mail has a one-click unsubscribe (`List-Unsubscribe` and `List-Unsubscribe-Post`). The GET only redirects to a confirm page, so link scanners cannot unsubscribe anyone. Node names are cleaned of control and bidi characters before they go into a mail. - Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE /api/account/notifications/watches/{pubkey}`, `POST /api/account/notifications/watch-my-nodes` (copies the synced "my nodes" list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message` gains `Headers`. **Frontend.** A "Notify me" toggle on the node pages, a Notifications section on the account page (`#/account?section=notifications`), an unsubscribe page, and the notification mail count on the admin overview. Spec: [`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md). ## Performance Each check reads watches, prefs and states from `users.db` (one query each), the watched nodes from the analyzer DB in chunks of 500, and last-heard times from the packet store. | Measurement | Result | |---|---| | One check on our staging instance, 130,000 packets in memory, 1 watcher | 6.9 ms, of which 0.93 ms holding the store read lock | | Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes | 1.75 ms | No work is added to ingest, broadcast or any request path. ## Verification - `internal/users` and `internal/mailer` with `-race`; the `cmd/server` suite plus `-race` on the notifier tests; 74 new Go tests. The same Windows-only failure as noted in #2138 applies. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 26 of 26 steps locally. With notifications off the new steps fail, so they do test the flag. - On our staging and production instance since 7 October 2026. ## Not in this PR - Other channels (Discord, Telegram, webhooks). Detection is separate from delivery, so one can be added. - The topology, RF and anomaly alerts of #775. - If the whole server starts after a feed outage that already ended, the grace window is unknown and a watcher can get one wrong offline mail. The user guide says so. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
abbe6846ff |
feat: propose and approve hashtag channels (user management part D) (#2139)
Part D of #2128: logged-in users propose a hashtag channel, an admin approves, rejects or later revokes it, and the ingestor decrypts approved channels without a restart. It is the first consumer of a generic proposal table, and answers the request in #2092. Stacked on #2138 (part C). Review the commits after that branch; GitHub shows C's commits here until it is merged. ## The situation - An instance decrypts only the hashtag channels listed in `hashChannels`. Users who want a city or club channel shown have to ask the operator, who edits `config.json` and restarts the ingestor. - #2092 asked for a suggest-and-approve flow; a downstream fork (dborup/CoreScope#99) runs one with anonymous suggestions and a file queue. ## What this PR adds **Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in the browser: - at most 31 UTF-8 bytes including `#` (firmware `ChannelDetails.name[32]`), case preserved, `Public` refused in any case; - control, bidi, separator and format characters refused except ZWJ, plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800, non-ASCII spaces). Go and JS give the same verdict for every code point. **Store.** Schema v4, one generic `proposals` table (kind, subject, status pending/approved/rejected/revoked, proposer, reviewer, note). One row per (kind, subject); limits are checked in the same transaction as the change. **Server** (only with `userManagement.channelProposals.enabled`) - `POST /api/proposals`, `GET /api/account/proposals`, `GET /api/admin/proposals`, `POST /api/admin/proposals/{id}/approve|reject|revoke`. - 400 for an invalid name. 409 for a duplicate, a rejected name, or a name already in `hashChannels`/`channelKeys` ("this channel is already decrypted on this instance"). 429 over a limit (`maxPending` 100, `maxApproved` 128, `perUserPerDay` 5). - `GET /api/channels` gains `approvedChannels`, so approved channels are listed before they have traffic. With the feature off the response is byte-identical. **Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds, re-validates the names, and swaps its key map through an `atomic.Pointer`. Configured channels always win, and a failed read keeps the last good set. The ingestor never writes `users.db`; AGENTS.md now says so. **Frontend.** "Propose for everyone" in the add-channel dialog, an "approved" marker in the channel list, "My proposals" on the account page, and an admin "Proposals" tab with a warning on approve. The packet detail pane now escapes the channel name: until now only the operator chose those names. Spec: [`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md). ## Performance Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`). Benchmark of that worst case: | Keys | Time per packet | |---|---| | 320 configured | 204 to 216 µs | | 320 configured + 128 approved | 272 to 320 µs | That is about +33%, linear in the key count and capped by `maxApproved`. With the feature off the decoder gets the identical map and no ticker runs. ## Verification - `internal/channel`, `internal/users` (with `-race`), `cmd/server` and `cmd/ingestor` pass locally, 53 new Go tests. The ingestor's `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on Windows and fails on `master` too. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 21 of 21 steps locally (propose, approve, the channel appears, revoke removes it). - On our staging and production instance since 7 October 2026. ## Not in this PR - Private (PSK) channels: their keys stay in the browser (#725). - Anonymous proposals. - Names that need ZWNJ (some Persian and Urdu spellings) and subdivision flags are refused, because ZWNJ and tag characters are format characters. Allowing them later only widens the rule, so no approved name gets stranded. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
d232072f85 |
feat: optional user accounts (part A: foundation) (#2129)
Part A of #2128: optional, off-by-default user accounts. With the feature off nothing changes; with it on, visitors can register and log in, and admins manage users and use the operator actions without the API key. PR #2130 (settings sync) builds on this one. The two are meant to be merged together. ## The situation - Operator actions (geofilter save and prune, backup, perf reset) need the shared `apiKey`. There is no per-person right. - Nothing in CoreScope knows who a visitor is, so the requests in #2128 that need that (#1835, #2092, #1508, #730) have nothing to build on. ## What this PR adds **Two new Go modules** - `internal/users`: a separate `users.db` (SQLite through `modernc.org/sqlite`) with users, sessions, single-use tokens, an audit log and a mail log. Passwords use argon2id. - `internal/mailer`: a `Mailer` interface with a Brevo client (send, delivery events, webhook parsing) and an in-memory fake for tests. **Server (`cmd/server`)**, active only with `userManagement.enabled` - 24 routes, all documented in OpenAPI under the `users` tag ([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)): - auth: register, activate, login, logout, me, forgot, reset; - account: profile, password, email change with confirmation, sessions, self-delete; - admin: list, detail, disable, enable, delete, role, resend activation, manual activation, mail status refresh; - a Brevo webhook, registered only when `mail.webhookSecret` is set. - `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites: the API key **or** an admin session. With the feature off it is the old API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`). - `/api/config/client` gets `userManagement: {enabled: true}` only when the service started; with the feature off the response is byte-identical. **Frontend** - `auth.js` (header account control, request helper that adds the CSRF header), `account.js` (login, register, activate, forgot, reset, confirm email, my account), `admin-users.js` (`#/admin/users`, deep-linked filters), `account.css` (theme tokens only). - On phones the top-bar control is hidden, so a conditional entry goes into the bottom-nav "More" sheet and the nav drawer. - The customizer geofilter tab and the Perf "Reset stats" button use the admin session when there is one. **Config.** A `userManagement` block (`config.example.json`, [`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)). The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server refuses to start when the block is enabled but incomplete. ## Security choices - Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when `publicBaseUrl` is https. Every cookie-authenticated state change needs the `X-CS-CSRF` header and a matching Origin. - Activation needs the token **and** the account password. Without the password, an attacker who keeps re-registering a known address could get the owner to activate an account that carries the attacker's password. - Register, forgot and email change answer identically for known and unknown addresses. A password reset ends all sessions, a password change ends all other sessions, and both end outstanding email-change links. - Rate limits: login 10 per 15 minutes, register and forgot 5 per hour, per IP and per address. The bucket count is capped. `trustedProxies` makes the per-IP limits see real client IPs behind a proxy. - Server logs carry `#<user id>`, never addresses, tokens or passwords; mail-provider error texts are redacted before logging. ## Performance No change to an existing hot path with the feature off. With it on: - One `users.db` lookup per authenticated request (session by token hash). - The admin user table rebuilds its `tbody` on each filter change. `users.List` caps the result at 1000 rows (`internal/users/users.go`), which bounds the rebuild. - `map[string]interface{}` in `openapi.go`: 79 before, 78 after. ## Verification - `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go test -race` pass locally. 121 new Go tests. - `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass. - `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67 passing (vm, real modules). - `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally against an `e2etest` build with the fake mailer and against a feature-off build. CI builds the `e2etest` binary and runs the suite on a second server (`deploy.yml`). - On a staging instance with a real Brevo key: register, activation mail delivered, activate, admin table, "Refresh status" showing sent, deferred, delivered, opened and clicked. ## Not in this PR - Settings sync (#2130), the admin dashboard, approval flows and notifications (parts B to E of #2128). - A `requireReadAuth` mode (#1835). Sessions from this PR are what such a mode would accept. - Binary size and build time with `modernc.org/sqlite` linked next to `mattn/go-sqlite3` were not measured. Their driver names do not collide. #1992 discusses the driver choice. - No Brevo webhook was configured on staging; delivery status there came from "Refresh status". --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efiten
|
e01565737a |
feat(ingestor): store CoreDrive RX region answers, with position, clock and retention (#2047)
The Scope Audit page said a repeater's declared region list can come from CoreDrive RX while nothing the app sends ever reached it: the client-topic switch handled packets and rf only, so /regions was dropped without a log line, and node_declared_regions is read by region_keys.go and config.go but created by nothing in this tree. #2044 found that gap and proved it against a live instance. This lands the implementation that has been carrying the feature in production on the ON8AR fork since 2026-09-06, the instance CoreDrive RX publishes to. Measured there: 1840 answers about 275 repeaters from 51 collectors, 2026-08-18 to 2026-09-19. Beyond storing the answer it keeps three things the first version did not: position (lat, lon, pos_acc_m, filled on 1263 of 1840 rows, with acc_m dropped when the fix it qualifies was rejected), repeater_clock (filled on all 1840, so a wrong repeater clock cannot make an answer look newer than it is), and retention with per-collector history (pruneOldClientDeclaredRegionsAt bounds by age instead of keeping one row per target). On that dataset 138 of 275 repeaters have answers from more than one collector and 40 have collectors that disagree about the region list, which is the signal the Scope Audit exists to surface and which only survives while more than one answer does. It gates on its own clientRegions block rather than riding on clientRxCoverage, so region answers can be accepted without GPS-tagged reception uploads, and AES block padding is trimmed from region names on ingest. Taken from #2044 with the author credited as co-author: declaredRegionsTablePresent() and its test, a real bug this version lacked (supervisord starts both processes together, so a server that probes first ignores every answer until its next restart), and the docs/client-rx-coverage.md section. Ported by cherry-picking the fork's nine commits rather than retyping, so this is the code that has been running. CI run 35434151026 is green: server ok 80.177s, ingestor ok 99.413s, race detector ok 112.406s, no --- FAIL lines. Merged by the interim maintainer without a second human reviewer: CI and the production figures above are the independent checks. |
||
|
efitenandClaude Opus 5
|
5efe61eef2 |
fix(ingestor): set an explicit MQTT ClientID per source (#2013) (#2016)
## What `buildMQTTOpts` (`cmd/ingestor/main.go:591`) never called `SetClientID`, so with paho.mqtt.golang v1.5.0 every ingestor connected with a zero-length ClientID and `CleanSession=true`. The session identity then depended on the broker. This PR: - adds an optional `clientId` per `mqttSources` entry (`cmd/ingestor/config.go:29`) - when unset, uses `corescope-<name>-<6 hex chars>` (`cmd/ingestor/main.go:653`). The name is reduced to `[0-9A-Za-z-]`, with the broker host as fallback when the name is empty. The suffix comes from `crypto/rand` and changes on every ingestor start. - sets the ID once per source in `buildMQTTOpts` (`cmd/ingestor/main.go:616`). paho copies the options into the client and reuses them for every reconnect, and the watchdog force-reconnect reuses the same client, so the ID is stable for the life of the process. - logs the ID on connect: `MQTT [tag] connected to <broker> as client <id>` (`cmd/ingestor/main.go:150`) - documents the key in `config.example.json:210` as a `_comment_clientId` entry rather than a value, because `docker/entrypoint.sh:6` copies that file as a live config and a literal value would give every default deployment the same ID. Also listed in `cmd/ingestor/README.md:94`. ## paho behaviour - No client-side length limit. `SetClientID` only stores the value; the 65535 check in `packets/connect.go:156` is in `Validate()`, which the client never calls. - The default ID is longer than the MQTT 3.1 limit of 23 characters for most source names. paho falls back to MQTT 3.1 after any refused CONNACK when no protocol version is set (`client.go:412`), so on a broker that refuses the first 3.1.1 attempt, the retry may hit that limit. I did not cap the length because paho does not require it and the 3.1.1 path accepts it (see below). ## Tests `cmd/ingestor/mqtt_opts_test.go:53-107`: - default ID is non-empty, has the sanitized name prefix, and contains only `[0-9A-Za-z-]` - broker host is used when the name is empty - configured `clientId` is used verbatim - two unconfigured sources with the same name get different IDs - the client built from the options reports the same ID Mutation checks: removing the random bytes fails the "different IDs" test; removing sanitization fails the prefix and character-set tests. `go test ./...` in `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails locally on Windows for a symlink privilege reason. `gofmt` and `go vet` are clean. ## Validation against a real broker On a staging instance (build `e84d2da6`) connecting to a Mosquitto bridge: ``` MQTT [lincomatic] connection attempt #1 to tcp://mosquitto-bridge:1883 MQTT [lincomatic] connected to tcp://mosquitto-bridge:1883 as client corescope-lincomatic-71a6eb MQTT [lincomatic] subscribed to meshcore/# ``` The 27-character default was accepted on the first attempt and packets kept arriving afterwards. ## Not verified - Only one broker type (Mosquitto) was tried. - `-race` was not run locally (no cgo toolchain on the test machine). - The case where both the source name and the broker host are empty (ID becomes `corescope-<hex>`) has no test. Fixes #2013 ## Review follow-up (commit `a1d6709e`) An independent review found no bug in the ID handling, but the tests covered less than their names said. Changed, tests only (`cmd/ingestor/mqtt_opts_test.go`): - `TestBuildMQTTOpts_ClientIDSurvivesReconnects` replaces the old stability test, which only checked that paho copies the options. Against a loopback fake broker built on paho's `packets` codec, the first CONNECT, paho's auto-reconnect after the broker drops the socket, and the watchdog force-reconnect (`buildForceReconnectFn`) must all carry the same non-empty ID. It runs in about 0.01 s and passed `-count=30 -cpu 1,2,8`. - `TestBuildMQTTOpts_ClientIDDefaultShape` asserts full IDs: `^corescope-local-feed-1-[0-9a-f]{6}$`, `^corescope-mqtt-example-com-[0-9a-f]{6}$` for the broker host fallback (no port), and `^corescope-[0-9a-f]{6}$` with neither a name nor a host. - Mutations now caught: `SetClientID` removed, `u.Host` instead of `u.Hostname()`, a 1-byte suffix, the name guard dropped, sanitization removed. The "as client" log line has no test because it is logged from a closure inside `main()`. Corrections to the description: - **Fallback to MQTT 3.1.** paho falls back after any failed handshake once the socket is open, not only after a refused CONNACK: also a read error or timeout before any CONNACK, or a first packet that is not a CONNACK (`client.go:401-416`, `net.go:83-97`). A failed dial does not trigger it (`client.go:387-391`), and after the first successful connect the protocol version is locked in (`client.go:422-424`). Without this PR the 3.1 retry sent an empty ID, which MQTT 3.1 forbids as well, so nothing gets worse. - **Broker side.** On the EMQX broker we run, authorization has per-username and all-client rules and no client-ID rules (checked through its REST API). Two per-username rules use `${clientid}` in a topic, but both are publish rules and the ingestor only subscribes, so no rule can match it. A broker that caps IDs at 23 characters but accepted the empty ID before would now reject the default ID for source names of 7 or more characters; I have no evidence such a broker is in use. - The "Not verified" item about the empty name and host case no longer applies. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efiten
|
0e607c1d01 |
feat(ingestor): derive region keys from what nodes declare, opt-in (#1989)
Follow-up to #1988, which made an ambiguous match deterministic. This adds the second tier of keys that ambiguity rule was needed for. A transport-scoped packet can only be named by a region key this instance holds. `hashRegions` is a hand-maintained list, so every region a node forwards that nobody typed into the config is stored unmatched, and everything downstream reports that region as **absent** rather than as **unnameable**. The instance already knows the names, though: `nodes.configured_scope` holds what the observer `/neighbors` ingestion (#1865) confirmed each node is configured for. This derives keys from those names, on top of the explicit list rather than instead of it. **Default off.** An absent config block leaves behaviour byte-for-byte unchanged, asserted by tests rather than argued. ## Sources Two, mirroring what the server's `AllCurrentDeclaredRegions` already merges, so the derived tier sees exactly what the Scope Audit sees: | source | availability | |---|---| | `nodes.configured_scope` | always — the column is part of the schema, written by the `/neighbors` path | | `node_declared_regions` | optional, where a deployment fills it by other means | The optional table is probed via `sqlite_master` before it is read. A stock install does not have it, and its absence must not abort a refresh the first source could answer on its own. ## The two spellings The sources spell the same region differently, and both are accepted: - `configured_scope` carries the leading `#` that `normalizeScopeList` adds, because every other stored scope value has one - an OTA answer in the optional table carries the bare name - `loadRegionKeys` already prefixes a missing `#` before hashing So `regionNameAcceptable` canonicalises before judging, and hands back the bare name the caller re-prefixes. What it rejects is what cannot be a region name at all: `*` (the flood wildcard, not a region), a comma (it would split the name on the next round-trip through a comma-separated column), a second `#`, whitespace, non-ASCII, NUL padding from a stale client, and anything past 32 characters. The rules are deliberately structural rather than about meaning. A real declared set contains entries that look like junk, but a blocklist on string values is unmaintainable, and the cost of one bad name is a single slot out of the cap plus a 1-in-65536 collision chance. `*` is skipped in **two** places on purpose: in the filter, so no key is ever derived for it, and in the source count, because nearly every node declares it and counting it would overstate both the cap arithmetic and the refresh log on every deployment. ## Rule 0 Each derived key costs one HMAC per transport-scoped packet and raises the random 2-byte collision rate by 1/65536. So: - the tier is **capped** (default 256) - over the cap, names are kept by how many distinct nodes declare them, so a one-off local name is dropped before a region half the network uses - benchmarked linear at ~0.65µs per key: at 314 keys that is 217µs per packet, 0.0008% of one core at the 0.037 transport-scoped packets/s this network produces There is no indexable shortcut to reach for, for the same reason as in #1988: `code1` is an HMAC over the payload, so nothing is payload-independent. The set is an `atomic.Pointer` to an immutable snapshot. A refresh builds the replacement off to the side and swaps the pointer, so the ingest path never blocks on a rebuild. `refreshDerived` is a load-then-store rather than a CAS loop, which is safe only because exactly one goroutine calls it: the refresh ticker, plus one synchronous call at startup. The comment says so, rather than leaving the type looking as though it tolerates concurrent refreshers. ## Tier 2, and the counters When several keys match one packet and exactly one of them is explicit operator config, the explicit one wins: an operator who typed a region into `hashRegions` outranks a name overheard on the air. Ambiguity between two equally-sourced keys still stores unmatched, unchanged from #1988. Counters tally how each packet was decided (unique / explicit-over-derived / ambiguous / none) and are logged on the refresh tick. They exist to answer one question with data rather than estimation: whether a third tier that breaks ties on path evidence is worth building at all. Measured on a live instance at 159 keys over 41.7 hours and 147,535 packets: **0.253% ambiguous**, with explicit-over-derived at zero because that instance has exactly one derived key. ## Rule 8 `maxDerived` and `refreshMinutes` are configurable values and belong in the customizer eventually. Documented in `config.example.json` for now, flagged here so it is tracked rather than forgotten. ## Tests - default off, and a refresh that is a no-op while disabled - the name filter across both spellings, the wildcard, and each structural rejection - ranking by declarer count, then recency, then name, so the result is deterministic rather than churning between refreshes - `configured_scope` as a source, including that a node counts once per name and the newest answer wins - both sources merged, neither dropped - the explicit-over-derived tie-break - the derived tier replaced rather than merged on refresh, so a region that stops being declared leaves the key set and the cap keeps meaning something `cd cmd/ingestor && go test ./...` passes apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs `SeCreateSymbolicLinkPrivilege` and fails on Windows on master too. `go vet` and `gofmt -l` clean, `config.example.json` still parses. |
||
|
efitenandClaude Opus 5
|
1ffaad8eb1 |
feat(#1794): per-IP limits and a deny list on the /ws upgrade (#1974)
Closes #1794. Follow-up to #1793, decided **before** the upgrade because the handshake is the resource being protected. - Deny list of addresses and CIDRs → 403 - Per-IP concurrent connection cap → 403 - Per-IP upgrade rate limit over a rolling minute → **429**, not 403: a temporary refusal should not read as "never come back" - Rejection counters split by cause in `/api/stats` under `websocket` ### The decision this feature lives or dies on Most CoreScope installs sit behind nginx, Caddy, Traefik or an ingress. `cdn_detection.go` says so in as many words: it deliberately excludes `X-Forwarded-For` from its CDN signals precisely because *every* reverse-proxied install sets it. For those deployments `r.RemoteAddr` is the proxy, `127.0.0.1` for every visitor on earth. A per-IP cap keyed on that address protects nobody and hands the sixth legitimate browser tab a 403. That is a self-inflicted outage wearing the costume of hardening. So: - **`X-Forwarded-For` is believed only from an address listed in `webSocket.trustedProxies`.** From anywhere else it is attacker-supplied, and trusting it would let anyone mint a fresh source IP per connection, which is strictly worse than having no limit at all. - **When the peer looks like a local reverse proxy and no `trustedProxies` is set, the per-IP limits are skipped**, and one warning names the setting that fixes it. Silently refusing real users is the worse failure. - **The deny list still applies there**, because it is the operator's explicit instruction rather than an inference. That is the answer to @mcode6726's question on the thread: it is neither "always the socket address" nor "always the header", and the operator decides which by naming their proxy. ### Two deliberate departures from the issue body **`maxConnsPerIP` ships as 0 (off), not 5.** Carrier-grade NAT puts thousands of unrelated mobile subscribers behind a single public IPv4. A cap of 5 refuses real visitors on phones while a scraper simply rents more addresses: all of the cost, none of the benefit. `upgradesPerMinPerIP` ships at **30 and on**, because that one *is* safe under CGNAT: a real client upgrades a handful of times per minute even while reconnecting, so 30 leaves ordinary traffic untouched while flattening a reconnect loop. A pointer type distinguishes "unset" from an explicit `0` that turns it off. **The default deny list is not shipped.** The thread proposed seeding 44 CIDRs for one VPS provider after a single scraper was seen at `23.111.177.6`. I have left it out: blanket-blocking a hosting provider by default breaks legitimate operators who host there, is undiscoverable by the person locked out (they see a bare 403), and ages badly as ranges get reassigned. The mechanism is here and `config.example.json` shows exactly how to configure it, so any operator who wants that list can have it in one line. If you want it shipped as a default anyway, that is your call as maintainer and it is a one-line change. ### Verification 19 tests, including all five the issue specifies as TDD requirements, each marked with the issue's own wording. Beyond those five: - a **bare address** in the deny list works, not just CIDR form. Operators write `1.2.3.4`, and silently ignoring that would be the worst possible failure for a deny list: it looks configured and blocks nothing - an unparseable deny entry is skipped and logged, not fatal. One typo must not take the server down - one client behind a trusted proxy does **not** exhaust another client's budget behind the same proxy, which is the entire point of honouring XFF - changing a forged XFF from an untrusted peer buys no fresh budget - `release` frees a slot and is **idempotent**, because `Unregister` can run twice for one client and double-crediting would leak slots - a **rejected** upgrade does not consume rate budget, or a retrying client could never recover once its window cleared - limits skipped for loopback and private peers; deny list applies anyway - a nil limiter allows everything, so a `Hub` built without `ConfigureLimits` behaves exactly as before - idle per-IP state is collected, while a record with a live connection never is Full `cmd/server` suite green, `gofmt` clean. ### Not done - No runtime config reload; restart required. Listed as optional in the issue. - No `WS_DENY_IPS` env override. Also listed as optional. - From the OWASP expansion in the first comment: `maxPayload` and the idle/read timeout are **already in master** (`SetReadLimit`, `SetReadDeadline`). The ping/pong heartbeat is not, and is not in this PR either; it is a separate change to the read/write pumps and belongs in its own review. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
9ae3387416 |
feat(ingestor): RF environment samples from mobile clients (#1906)
> **Stacked on #1905.** This branch contains #1905's commits; review and merge that one first. The diff unique to this PR is the `client_rf_samples` table, its handler, the delta query and its retention. ## What Everything CoreDrive RX records today is anchored to a *packet*. But a drive also passes through RF conditions that exist whether or not a packet arrives: the noise floor, how busy the channel is, how many receptions fail CRC. The radio measures all three and was never asked. This samples the companion's own counters along the GPS track and stores them, so the server can render a noise-floor map, a channel-utilisation map and a CRC-error-rate map. A fixed observer cannot produce those — it measures one point forever. **Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the attached radio. Nothing is transmitted. ## Design points worth knowing - **Absolutes are stored; deltas are derived at query time.** A lost or reordered sample then costs one interval rather than corrupting a running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs` fails to increase — that is the reboot and counter-wrap detector. - **Absent is not zero, end to end.** Firmware predating the `recv_errors` field cannot count CRC errors at all, and a stored `0` would read downstream as "a perfectly clean channel" — the opposite of "we don't know". Presence/absence is preserved through the app parser, the wire payload, a nullable column, and the delta view, which returns `nil` rather than `0` when either endpoint is unknown. Each of those five layers has its own test. - **`sampled_at` is millisecond precision, and it is load-bearing.** SQLite compares these strings lexicographically and `.` (0x2E) sorts before `Z` (0x5A), so a second-resolution retention cutoff would delete rows *inside* the window. The prune formats its cutoff with the same layout. ## Performance justification (touches the ingest hot path) - One INSERT per sample, gated behind an opt-in flag that defaults off. Sample rate is 15 s while moving and 5 min while parked, so roughly 240 rows per hour per active driver. - The delta query is a single `LAG(...) OVER` pass with no nested query inside the loop, so it cannot deadlock the single writer connection. Window functions are already used elsewhere in this codebase. - Retention has its own key and index (`sampled_at`); without it the table would grow unbounded, so `config.example.json` documents it inline. ## Safety for existing deployments Opt-in and default off on both sides (`clientRfSamples.enabled`, and `rfSampler` in the app). The coverage path is untouched — `Publisher.buildPayload` is byte-identical and a record with no `kind` field still routes to `/packets` unchanged. The MQTT dispatch was reshaped so that **anything on `meshcore/client/…` returns from that branch in every config state**, with the enable-gates inside rather than in the topic match. Previously a disabled gate let the message fall through to the observer path, where `parts[1]` — the literal string `client` — was read as a region and the phone's pubkey registered as an observer. The blacklist check now also runs ahead of the sub-topic switch, so it covers every present and future client sub-topic. ## Testing Full ingestor suite green. Notable coverage: a `/rf` message with the gate off writes nothing anywhere and does not fall through; a sample missing `uptime_secs` is rejected rather than stored as an unusable row; two samples 40 ms apart remain two rows; and the retention test seeds a row with a non-zero millisecond component inside the cutoff second, which is the only row that distinguishes a correct cutoff from an RFC3339 one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
9e13e0b05f |
feat(ingestor): full-packet RF observations from mobile clients (#1905)
## What A CoreDrive RX drive already carries far more RF information than reaches CoreScope, and it was being discarded twice: once in the mobile app (every packet it could not attribute to a directly-heard node was dropped before queueing) and once here (the ingestor decodes the *complete* packet, then keeps only `heard_key`/`snr`/`rssi`/`lat`/`lon`). This captures what was being thrown away, at **zero extra airtime** — nothing new is transmitted. - **`transmissions.code1` / `code2`** — the transport codes were decoded on every packet and used only to derive `scope_name`, then dropped. Storing them turns "which repeater forwards which scope" from a re-parse into a query. - **An async backfill** re-parses the `raw_hex` already on disk, so months of scope history become queryable with no new data collection. - **`client_rx_observations`** — a new diagnostic table holding every decodable packet a phone heard, with route type, transport codes, scope name, path-hash size, the full forwarder chain and the forwarder. ## Why it is safe for existing deployments Both halves are **opt-in and default off** (`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an existing deployment sees no behaviour change and no volume change on upgrade. The coverage invariant is untouched: `client_receptions` keeps its rule — 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an unattributable packet writes **zero** coverage rows. `deriveHeardKey`, `buildClientReception` and `InsertClientReception` are unmodified except for one guard described below. ## Performance justification (touches the ingest hot path) - **Backfill:** keyset-paginated by `id` in 5000-row batches, a single forward scan, `rows.Close()` before `Begin()` so it never deadlocks against `SetMaxOpenConns(1)`, and commits per batch so live ingest interleaves. Termination is driven by rows *scanned*, not rows decoded — an earlier count-based loop would have stopped at the first batch containing an undecodable row and then written its completion guard, permanently stranding the rest. - **Guard row is written if and only if the loop ran to genuine exhaustion.** Every error path leaves it unwritten so the next startup retries. - **Per-packet cost:** one extra INSERT on the client topic when enabled, gated behind an opt-in flag. No new work on the observer path. - **New indexes** cover the prune (`rx_at`), the flood-grouping (`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the scope query (`scope_name, rx_at`). Retention has its own shorter window — this table is diagnostic, not archival. ## Two firmware-derived correctness points - **`pkt_hash` is `ComputeContentHash()`**, byte-identical to `transmissions.hash`, so dark-traffic queries are a plain equality join rather than a translation layer. - **TRACE packets are refused.** TRACE repurposes the header path bytes as per-hop SNR values, so deriving a `heard_key` from them invents a node that never existed. `packetpath.PathBytesAreHops` existed but was never wired into the client path; it became reachable only because the app half now publishes packets it previously dropped locally. ## Testing Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes zero coverage rows and NULL `forwarder`; a `direction: "tx"` message writes no observation; a DIRECT route never sets `forwarder`; two forwarder copies of one flood remain two rows; the backfill's multi-batch path is exercised with an undecodable row in the first page; and a forced error asserts the migration guard stays unwritten. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
d821d9a390 |
feat(retention): add observerPurgeDays hard-delete for long-inactive observers (#1886)
## Problem `RemoveStaleObservers` only soft-deletes — it sets `inactive = 1` and the row stays forever. On a long-running deployment those rows just accumulate: on a two-year-old instance roughly 25% of the `observers` table was rows nobody can ever see again. There is currently no way to reclaim them. ## Fix A second retention stage. `PurgeStaleObservers` hard-deletes rows that are: - already `inactive = 1` (so the soft-delete stage owns the decision of *when* an observer goes stale), **and** - older than `retention.observerPurgeDays`, **and** - referenced by nothing. New config field `retention.observerPurgeDays`, default `0` = disabled. Existing deployments are unaffected until they opt in. Set it above both `observerDays` and `packetDays` — below those the reference guards keep every candidate row anyway. ## Why the reference guards are the point `observations.observer_idx` is a bare rowid with no foreign key. Deleting a still-referenced observer silently orphans history — `packets_v` stops resolving the observer and those packets get mis-attributed. Nothing errors; the data just quietly goes wrong. So the statement guards on all three referencing tables: ```sql AND NOT EXISTS (SELECT 1 FROM observations o WHERE o.observer_idx = observers.rowid) AND NOT EXISTS (SELECT 1 FROM observer_metrics m WHERE m.observer_id = observers.id) AND NOT EXISTS (SELECT 1 FROM dropped_packets d WHERE d.observer_id = observers.id) ``` This is correctness, not defensive padding — it was found the hard way, by orphaning 280 observation rows during a manual purge that skipped one of these checks. Each guard has its own test. ## Performance Each `NOT EXISTS` is an index seek per candidate row (`idx_observations_observer_idx`, `idx_dropped_observer`, the `observer_metrics` PK), and `observers` is O(100). It runs on the existing daily retention tick alongside `RemoveStaleObservers`, never on the ingest path. ## Tests Eight tests in `cmd/ingestor/observer_purge_test.go`, written before the implementation: - deletes an unreferenced stale row - keeps a row referenced by `observations` — and asserts zero orphans afterwards - keeps a row referenced by `observer_metrics` - keeps a row referenced by `dropped_packets` - keeps a row that is old enough but still `inactive = 0` - keeps a row inside the retention window - no-ops when disabled (`0` and `-1`) - config accessor table test ## Invariant Writes stay in `cmd/ingestor` per #1283. `cmd/server/readonly_invariant_test.go` now also forbids `PurgeStaleObservers` as a method on the server's `*DB`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
h4badger
|
34b41fd5b6 |
Add topographic map layers (#1891)
This adds two optional map tile providers: - OpenTopoMap - USGS They are disabled by default, but can be quite useful for visualizing repeater sites with terrain features visible. |
||
|
efitenandClaude Opus 5
|
176bb53335 |
fix(#1784): ship pathTrust default 1, not 2 (#1929)
Follows #1841. Moves the pathTrust default from 2 back to 1. ## Why #1784's first acceptance criterion is **"Default behaviour remains backward-compatible"**, and its example config shows `minHashBytesForMapping: 1`. What shipped is 2. The problem is not the value. It is that **there is no way to undo it from the UI.** #1841 adds no control for the threshold: it is `config.json` only, and changing it needs a restart. The customizer gains a hint that says exactly that. So an instance that upgrades without touching config switches to the stricter rule, and the only visible symptom is that the neighbour graph and the resolved paths quietly get smaller. The existing "Hide 1-byte path hops" toggle (#1633) is a *display* filter and does not change what counts as evidence, so it is not an escape hatch either. The two are easy to confuse. ## How much this actually moves Measured on a live instance via `/api/analytics/hash-sizes`, not estimated: | path-hop observations | count | share | |---|---|---| | 1-byte prefix | 116,923 | **56.0%** | | 2-byte prefix | 86,031 | 41.2% | | 3-byte prefix | 5,753 | 2.8% | | repeaters by observed hash size | count | |---|---| | 1-byte | **645 (41%)** | | 2-byte | 865 | | 3-byte | 63 | At threshold 2 the 1-byte column stops counting as mapping evidence. `MeetsPathTrust` also drops the legacy bucket-0 observations with it (pre-#1638 persisted neighbor edges that carry no per-mode breakdown), so already-stored edges lose their evidence status on upgrade too. ## What this does not change The knob works and is untouched. Operators who want the stricter behaviour set `minHashBytesForMapping` to 2 or 3, which is the opt-in #1784 describes. Only the default moves. Nothing about storage changes; packets and paths were never affected either way. ## Also fixes an inconsistency inside #1841 Five frontend consumers already fall back to **1** when `MC_getPathTrustThreshold()` is unavailable: `analytics.js`, `live.js`, `map.js`, `nodes.js`, `route-view.js`. Two fell back to **2**: `hop-filter.js` (the getter itself) and `customize-v2.js`. They now all agree. ## Tests - `internal/packetpath`: `TestMeetsPathTrust_ZeroValueOptIn` was asserting the old default *through behaviour*, so it would need rewriting on any future default change. It now asserts the property instead: an absent JSON field resolves to `DefaultMinHashBytesForMapping`, behaves identically to naming that value outright, and an explicit stricter setting still wins. Package tests pass. - `test-issue-1633-hide-1byte-hops.js`: the case pinning the getter's default is updated, with the reasoning in a comment so the next person sees why it is 1. **37 passed, 0 failed** (master baseline: 37 passed, 0 failed). - `cmd/server` config tests pass. ## One thing I want to flag rather than paper over The test I changed was named `default is 2 (operator-confirmed)`. I am overriding something that was confirmed with an operator, and I am not claiming that confirmation was wrong. My reading is that it was about the threshold being a *useful* value, which it is, rather than about it being the default in a build with no UI to change it. If the intent really was "2 out of the box for everyone", say so and I will close this. @Bjorkan as the issue author, @nullrouten0 and @Saarlandpower since you have touched adjacent code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jesper B
|
8c48f2b49a |
feat(#1784): wire display consumers to path trust threshold (#1841)
Fixes #1784 ## Summary Step 4 of the #1784 multi-PR project — wires all frontend display consumers to respect the configured `pathTrust.minHashBytesForMapping` value. **Depends on:** #1840 (must be merged first — provides `MC_meetsPathTrust` / `MC_pathBelowTrust` / `MC_getPathTrustThreshold` helpers and `PATH_TRUST` global). ## Changes ### `map.js` — trust-gated route display - `drawPacketRoute` checks `MC_pathBelowTrust` before drawing polylines - When all path hops are below the configured threshold, shows a "Route not displayed" message with config guidance instead of speculative polylines - Cleans up the trust message control when a new route is drawn ### `analytics.js` — subpath trust filtering - `renderTable` in `renderSubpaths` filters route patterns whose hops are below trust threshold - Combined filter logic: both the 1-byte hide toggle AND trust threshold are applied together - Info line shows which filters are active ("1-byte hide + 2-byte trust", etc.) - No-data message explains which filters caused exclusion ### `route-view.js` — speculative path annotation - Path picker groups tagged with `belowTrust` flag when any hop doesn't meet the configured threshold - Speculative paths show `(speculative, <N-byte hops)` annotation with tooltip explaining the trust threshold - Tooltip includes guidance on changing `pathTrust.minHashBytesForMapping` in config.json ### `live.js` — Paths Through widget - `_pathHopsBelowTrust()` helper checks whether all hops in a path are below trust threshold - Widget fallback message distinguishes between "1-byte filtered" (display toggle) and "N-byte trust threshold" (server config) - Message includes the config key for operators to adjust ### `nodes.js` — confidence weight adjustment - `modeWeight` initialization considers the trust threshold - Hash modes below threshold get zero confidence weight - Bucket-0 (legacy/unknown) excluded at threshold >= 2 per #1784 bucket-0 policy ## Testing `test-issue-1633-hide-1byte-hops.js`: - 5 new source-grep guards verifying each consumer references the trust threshold helpers - All 26 tests passing (21 existing + 5 new) ## Files changed (6 files, +136/-6) ``` public/analytics.js | 28 ++++++++++++++++++--- public/live.js | 19 ++++++++++++++- public/map.js | 21 ++++++++++++++++ public/nodes.js | 8 ++++++ public/route-view.js | 16 +++++++++++- test-issue-1633-hide-1byte-hops.js | 50 ++++++++++++++++++++++++++++++ ``` --- **Depends on:** #1840 **Written by:** DeepSeek V4 Pro in Max Mode |
||
|
|
8ce5291b7c |
fix(map): apply the CARTO key on every map surface (rebase of #1916 onto #1919) (#1926)
Continues #1916 by @nullrouten0. The commit is theirs, authorship
unchanged; I rebased it onto master and resolved the fallout from #1919.
Opening it here rather than force-pushing to someone else's branch.
## Why this is needed after #1919
#1919 shipped the Carto key, but only for the five `BASE_STYLES` entries
in `map-tile-providers.js`. Four map surfaces still request unkeyed
tiles and get them back stamped `API KEY REQUIRED` at HTTP 200:
- `public/roles.js` — `getTileUrl()` returns the bare `TILE_LIGHT`
constant in light mode and never consults the registry (the dark branch
has consulted it since #1461). Affects `analytics.js:2200` and
`nodes.js:94`, so the analytics map and the node-detail map stay
watermarked in light theme even with a key configured.
- `public/customize-v2.js:1838` and `:2047` — the two geo-filter maps in
the customizer.
- `public/geofilter-builder.html` — standalone page, outside the SPA, so
it fetches `/api/config/client` itself.
@nullrouten0 had already found and fixed all four, plus written the
docs, before #1919 was merged. Their diagnosis of the light-mode branch
is in the PR verbatim.
## What I changed while rebasing
1. **`carto.token` → `carto.key`.** #1919 shipped `key` and operators
already have it in their configs; renaming now would break them
silently. All of #1916's code, tests and docs follow suit.
2. **Dropped the duplicate key getter.** `_getCartoKeyParam` did the
same job as master's `_getCartoKey`; kept master's.
3. **Merged the two test suites.** Master's five cases plus four of
#1916's that master does not cover: the `api_key`-is-ignored assertion,
lazy resolution after async config, and both `MC_tileUrlById` cases. 42
passed, 0 failed.
4. **Corrected one factual claim in the docs.** #1916 stated CARTO
offers no referrer or origin restriction. CARTO does ask for a domain
when the key is issued. Whether that is enforced per request is not
something I verified, so the note now says exactly that rather than
asserting either way.
5. Merged the two config comments, keeping the operationally useful
part: unkeyed tiles return **200** with a watermark, so nothing errors
and no healthcheck fires. Verify by looking at a tile, not at a status
code.
## Verification
Rebased onto `7aa60c03`. No regressions:
| | base
|
||
|
Marek WojtaszekandClaude Opus 5
|
7aa60c0350 |
fix: support Carto basemap API key (tiles are watermarked without one) (#1919)
## Problem Carto began requiring an API key for its raster basemaps in 2026-08. Requests without a key still return `HTTP 200`, but the returned PNG is stamped `API KEY REQUIRED / carto.com/basemaps/apikey`. Because `carto-dark` / `carto-light` are the built-in defaults, **every CoreScope instance that has not changed tile providers now renders watermarked maps.** Reproduce without involving CoreScope at all — the watermark is baked into the bytes Carto serves: ```bash curl -o tile.png https://a.basemaps.cartocdn.com/light_all/11/1152/683.png # HTTP 200, and tile.png carries the watermark. (11/1152/683 is Lublin, PL.) ``` ## Why `carto.domain` cannot express this Carto takes the key as a query parameter on the **same** host: ``` https://basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}.png?key=YOUR_KEY ``` The existing `tiles.providers.carto.domain` option builds `https://{s}.<domain>.cartocdn.com` — an enterprise subdomain, which is a different mechanism. There was no way to supply a key, so I added one rather than repurposing `domain`. ## What this changes Adds `tiles.providers.carto.key`, appended as `?key=` (URL-encoded) to all five Carto styles: `carto-dark`, `carto-light`, `carto-voyager`, `carto-voyager-dark`, `positron-dark`. ```json "carto": { "enabled": true, "key": "", "domain": "" } ``` When no key is set the emitted URLs are byte-identical to today's, so this is a no-op for anyone who has not requested one. `key` and `domain` compose, so enterprise users keep their subdomain and gain the key. Free keys are available at <https://carto.com/basemaps/apikey> (5M tiles/month, non-commercial tier). **Security note:** like the existing OSM and Stamen tokens, this key reaches the browser — unavoidable for raster tiles. The `_comment_carto` text in `config.example.json` tells operators to restrict the key by domain in the Carto dashboard. ## Tests 5 new cases in `test-issue-1420-tile-providers.js` — **38 passed, 0 failed**: 1. no `?key=` emitted when no key is configured (guards the no-op claim) 2. all five Carto styles append the key when it is set 3. the key is URL-encoded 4. the key does not leak into OSM / Esri provider URLs 5. `key` coexists with the enterprise `domain` option Also ran the required frontend suite: `test-packet-filter.js` (92 passed), `test-aging.js` (18 passed), `test-frontend-helpers.js` (625 passed, 2 failed). **The 2 failures are pre-existing on `master`** — I ran the same file from a clean `origin/master` worktree and got an identical 625/2. They are `favStar returns filled star for favorite` / `... for non-favorite`, unrelated to tiles. ## Performance Not a hot path. `url()` returns a Leaflet URL *template* that is built once per tile-layer construction (`map.js:289`, `live.js:1422`), not once per tile — `_getCartoKey()` is called at exactly the same frequency as the existing `_getCartoBase()`, and does one property read plus one `encodeURIComponent`. ## Validation Verified against a live instance (<https://analyzer.marwoj.net>) running this logic as a mounted patch, with a real Carto key: - **before** — `light_all` and `dark_all` tiles both watermarked - **after** — both clean, with the key restricted to the instance's domain in the Carto dashboard - confirmed the key survives `config.json` → `/api/config/client` → `MC_MAP_CFG` and reaches the tile URLs ## Heads-up for maintainers Carto is retiring raster basemaps and steering users to vector, so this fix has a shelf life. It restores the default experience now; a vector migration is a larger, separate piece of work. I am not sure whether you want an issue filed to track that — happy to open one if it helps. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kpa-clawbotandopenclaw-bot
|
ec0ebeda2f |
fix(#1793): WebSocket CheckOrigin allowlist (block cross-origin scrapers) (#1795)
## Summary Closes the wide-open `/ws` WebSocket upgrader (`CheckOrigin: return true`) that lets any browser origin scrape live packet data. Replaces it with an explicit allowlist consulted from `cfg.CORSAllowedOrigins`, plus an implicit same-origin allowance and an empty-Origin (non-browser client) allowance. Fixes #1793. ## Rules (`Hub.checkOrigin`) - Empty `Origin` header → **allow** (non-browser clients; per-IP rate/deny gating tracked separately in #1794). - `Origin` host == request `Host` (case-insensitive) → **allow** (same-origin). - `Origin` matches an entry in `cfg.CORSAllowedOrigins` by exact case-insensitive match → **allow**. - `"*"` in `cfg.CORSAllowedOrigins` is **deliberately ignored** for `/ws`. A startup `[ws] WARNING:` is logged once when present. - Anything else → **reject** (gorilla returns 403). ### Deliberate divergence from CORS XHR CORS XHR (`corsMiddleware`) still honors `"*"` for read-only cross-origin GETs. The `/ws` upgrade does NOT, per OWASP's WebSocket Security Cheat Sheet: > Use an allowlist, not a denylist. Avoid wildcards or substring matching. — https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html `"*"` on the WS path would re-open the exact CSWSH/scraping vector this PR closes, so it is rejected with a startup warning rather than silently honored. This intentional asymmetry is documented in the updated `_comment_corsAllowedOrigins` in `config.example.json`. ## TDD red → green - `e5974c6a` **RED** — adds `cmd/server/websocket_checkorigin_test.go` with five cases; `SetAllowedOrigins` introduced as an enforcement stub so the test compiles and fails on the assertion (CI fails on this commit by design). - `a4791dc3` **GREEN** — implements `Hub.checkOrigin`, wires `SetAllowedOrigins` from `main.go`, updates the config example. All tests pass. ## Tests added (`cmd/server/websocket_checkorigin_test.go`) - `TestCheckOriginRejectsForeignOrigin` — foreign Origin → 403 - `TestCheckOriginAllowsEmptyOrigin` — non-browser client → 101 - `TestCheckOriginAllowsSameHost` — same-origin → 101 - `TestCheckOriginAllowsAllowlistedOrigin` — exact allowlist match → 101 - `TestCheckOriginWildcardDoesNotAllowForeignOrigin` — `"*"` in allowlist still rejects foreign origin → 403 ## Files changed - `cmd/server/websocket.go` — `Hub.allowedOrigins`, `SetAllowedOrigins`, `checkOrigin`, wired into `Upgrader.CheckOrigin`. - `cmd/server/main.go` — `hub.SetAllowedOrigins(cfg.CORSAllowedOrigins)` at the single call site. - `cmd/server/websocket_checkorigin_test.go` — new test file. - `config.example.json` — updated `_comment_corsAllowedOrigins` to document `/ws` gating and the `"*"` divergence. ## Out of scope (follow-up) - **#1794** — per-IP rate limit / deny list / connection cap for non-browser clients (which still bypass Origin because they don't send one). Layered defense; not in this PR. ## Verification - `go test ./cmd/server/...` — all server tests pass locally (574s). - Preflight clean (`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`). --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
57956712e7 |
fix(#1768): Relay Airtime Share uses LoRa Time-on-Air (preamble-aware) — partial fix (#1776)
Partial fix for #1768 — Relay Airtime Share now uses closed-form LoRa Time-on-Air instead of a payload-bytes-only proxy, removing the ~3-4× bias against small frames (preamble + fixed-symbol intercept). cross-stack: justified — backend score formula needs a frontend caption change (`public/analytics.js` dumbbell preset banner + tooltip) so operators can interpret the assumed PHY block. Both move together or the metric is misleading. ## Red commit `8da57062` — failing test asserts ToA-based score (~83.48 % ADVERT share on the locked acceptance fixture) instead of the byte proxy's 95.24 %. `internal/lora.TimeOnAir` was a zero-returning stub at the red commit; tests failed with assertion errors, not build errors. ## Green commit `dd402edd` — implements `lora.TimeOnAir` (Semtech AN1200.13 / SX126x §6.1.4 closed form, cross-checked against RadioLib), wires `score = TimeOnAir(payloadBytes, preset) × distinctRelays` in `cmd/server/relay_airtime_share.go`, surfaces the preset in the JSON response and analytics caption. ## Config (per AGENTS Config Documentation Rule) New keys under existing `analytics` block: ```json "loraPreset": { "freq": 869600000, "bw": 62.5, "sf": 8, "cr": 5 } ``` Defaults match the deployment's actual `get radio` (869.6 MHz / BW 62.5 kHz / SF 8 / CR 4/5). `CRC=1`, `IH=0`, `DE = (T_sym ≥ 16 ms)`, and the SF-dependent preamble (32 for SF≤8 else 16, per firmware `preambleLengthForSF` / MeshCore PR #1954) are firmware-fixed constants in `internal/lora/toa.go` and intentionally NOT surfaced as config (per re-triage). ## Scope In-scope files (6): - `internal/lora/toa.go` (new package — closed-form ToA) - `internal/lora/toa_test.go` (table-driven preset tests) - `cmd/server/relay_airtime_share.go` (wire ToA into score) - `cmd/server/relay_airtime_share_test.go` (recomputed expected values) - `cmd/server/config.go` + `config.example.json` (preset config keys) - `public/analytics.js` (preset caption on dumbbell chart + tooltip) Plus `cmd/server/go.mod` (replace directive for the new internal module). ## Deferred to v2 (separate issues per re-triage) - Per-observation SF/BW + radio-settings-aware dedup (blocked: ingestor stores SNR/RSSI only, no SF/BW on observations). - CR-per-hop dual-point sensitivity band (CR scales only the payload symbol term `(CR+4)`, not the preamble/header; second-order accuracy gain). - Cross-SF bridge accounting. ## Tests ``` cd internal/lora && go test ./... → PASS cd cmd/server && go test -run RelayAirtime → PASS ``` ## Preflight overrides - `check-branch-clean` (cross-stack): justified above — score formula change requires matching caption update; both files trace to the same issue. --------- Co-authored-by: kpa-clawbot <kpa-clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@openclaw.local> Co-authored-by: bot <bot@meshcore> |
||
|
|
22fe929da2 |
feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728)
Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be> |
||
|
Kpa-clawbot
|
825b26485c |
fix(#1181): hide nodes whose name starts with a configured prefix (#1655)
Fixes #1181. ## Summary Adds operator-configurable name-prefix hiding for nodes. When a node's name starts with any prefix listed in the new `hiddenNamePrefixes` config field (default `["🚫"]`), it is omitted from `/api/nodes`, `/api/nodes/search`, and `/api/nodes/{pubkey}`. DB rows are preserved — the filter runs at the API layer only, so observation history (paths, hops, distances) stays intact and the node simply re-appears if the operator clears the prefix list. This mirrors the convention already in use on other MeshCore map dashboards: an operator who wants their node hidden renames it with the 🚫 prefix and sends an advert; the next advert is then dropped from the dashboard. The node is **not** hidden from the mesh itself — only from this dashboard. This is documented inline in `config.example.json`. Implementation follows the existing `IsBlacklisted` pattern exactly: a new `Config.IsNameHidden(name)` method, and three filters in `routes.go` placed alongside the corresponding blacklist filters. No DB schema, public API, or websocket changes. ## Files changed - `cmd/server/config.go` — new `HiddenNamePrefixes []string` field + `IsNameHidden` method - `cmd/server/routes.go` — filters in `handleNodes`, `handleNodeSearch`, `handleNodeDetail` - `config.example.json` — new field + `_comment_hiddenNamePrefixes` operator doc - `cmd/server/hidden_name_prefix_1181_test.go` — new test file (red → green) ## Test plan Two new subtests in `TestHiddenNamePrefix_1181_*`: 1. `_NodesList` — inserts a node named `🚫 ban me`, asserts it is present when `HiddenNamePrefixes` is empty and absent when set to `["🚫"]`. 2. `_Search` — inserts `🚫 search me`, asserts `/api/nodes/search?q=search` does not surface it when the prefix is configured. Verified red→green: - Red commit `d0903852`: `go test -run TestHiddenNamePrefix_1181` fails on the leak assertion (`hidden_name_prefix_1181_test.go:94`). - Green commit `e79a0d8d`: same command passes. ``` $ cd cmd/server && go test -run TestHiddenNamePrefix_1181 -count=1 . ok github.com/corescope/server 0.060s ``` ## Out of scope - Auto-purging DB rows for hidden nodes — left to existing retention. The triage was explicit: hide, do not delete. - Live websocket broadcast: nodes are not broadcast via websocket (only packets), so no separate emit path needs filtering. Frontend reads nodes via `/api/nodes`, which is filtered. - Frontend customizer for the prefix list — operators configure via `config.json` like every other knob. |
||
|
Kpa-clawbotandKpa-clawbot
|
e04c7113cb |
feat: integrate hashtag channels from meshcore-channels catalogue (#1323) (#1656)
Fixes #1323 ## Summary Adds a small in-memory cache of the community-maintained hashtag-channels catalogue (`marcelverdult/meshcore-channels`) and exposes it as `GET /api/known-channels?region=XX` plus a collapsed sidebar section on the Channels view ("Known channels (catalogue)") with a one-click "+ Add" button per row. Per triage (#1323): new `cmd/server/known_channels_cache.go`, new `GET /api/known-channels?region=…`, frontend section in `public/channels.js`. No new DB tables — cache is in-memory only. ## What changed - `cmd/server/known_channels_cache.go` — `knownChannelsCache` with an atomic snapshot pointer, 24h default refresh, 30s HTTP timeout, 4 MB body cap, custom `User-Agent`. Fail-soft: a failed refresh leaves the last-known snapshot in place. Background goroutine started from `main.go` after the neighbor-graph recomputer; never blocks startup. - `cmd/server/known_channels_route.go` — `GET /api/known-channels?region=` serves the cached snapshot off the atomic pointer (never blocks on upstream). Region filter is case-insensitive ISO 3166-1 alpha-2. Empty/missing cache returns 200 with an empty entries list (fail-soft for the UI). - `cmd/server/config.go` — `KnownChannelsURL` + `KnownChannelsRefreshMs`. - `config.example.json` — example values + `_comment_knownChannels`. - `public/channels.js` — new collapsed sidebar section "Known channels (catalogue)" that lazy-fetches `/api/known-channels` on first render and renders rows with a "+ Add" button. The button calls the existing `addUserChannel(name)` path, so adding catalogue channels reuses the full save-key + decrypt flow that user-typed hashtags already use. - `cmd/server/known_channels_cache_test.go` — failing-first tests: - `TestKnownChannelsParseFixture` asserts the parser populates `GeneratedAt`/`License` and region-stamps every entry while skipping empty countries. - `TestKnownChannelsRouteRegionFilter` asserts the route returns 200 with exactly the filtered subset for `?region=be`. - `TestKnownChannelsFailSoftOn500` asserts a failed upstream fetch leaves the prior snapshot in place and bumps `failCount`. ## Upstream pinning The default URL is pinned to the specific file `channels-by-country.json` on `main`: > https://raw.githubusercontent.com/marcelverdult/meshcore-channels/main/channels-by-country.json Shape (verified 2026-05-24): ```json { "generated_at": "...", "license": "CC0-1.0", "countries": { "be": [{"channel": "#antwerpen", "description": "..."}], ... } } ``` ## Test plan ``` cd cmd/server && go test -run 'TestKnownChannels' -count=1 . ok github.com/corescope/server 0.008s ``` Red commit: 5c43cff3 (all three tests fail on assertions, build clean). Green commit: 54a1080e (parser + cache + route implemented, all three pass). ## TDD evidence (red → green) - **Red commit `5c43cff3427afd8aa2f3cce20c31058190aebc37`** — tests added with stub implementations that compile but return zero/empty so each test fails on an assertion (not a compile/import error). `go test -run TestKnownChannels` output captured in the commit message. - **Green commit `54a1080e45fd2e10da2caa156f376bf4d0212976`** — parser, cache, route, main-wiring, frontend section land; all three tests pass. ## Frontend verification Browser verified: http://analyzer-stg.00id.net/#/channels (with the `/api/known-channels` response stubbed in DevTools to simulate the cache being populated on staging, which is still on master and doesn't have the new endpoint yet). E2E assertion added: cmd/server/known_channels_cache_test.go:71 — asserts the route returns 200 and the response body's `entries` length matches the filtered subset. ## Limitations / follow-ups (not in scope of this PR) - The catalogue only ships PSK keys for a small subset of entries (the upstream schema makes `key` optional). For entries WITHOUT a `key`, the "+ Add" button still wires through `addUserChannel("#name")` — which derives the standard public-channel key from the name (the same path used today when a user types `#foo` into the Add Channel modal). For entries WITH a `key`, a follow-up PR can pass the key through to `addUserChannel` so the UX matches "paste-a-PSK". Today the key is shown in the JSON payload but not yet wired into the FE button. - No deduplication against the in-memory `/api/channels` list — the catalogue section is intentionally separate so the user sees which channels exist worldwide even if their server hasn't seen traffic. - No per-section region selector yet — the section shows the full catalogue regardless of the page-level region filter. Future work: add a dropdown. ## Preflight ``` ═══ Preflight clean. ═══ ``` cross-stack: justified — issue #1323 spans `cmd/server` (cache + route) and `public/channels.js` (sidebar surface); same feature, both halves required. --------- Co-authored-by: Kpa-clawbot <bot@corescope.local> |
||
|
Kpa-clawbotandcorescope-bot
|
3d12266595 |
fix(#1608): address PR #1609 follow-up findings — config doc, receipt-time liveness, buffer stop/clamp warn (#1623)
Follow-up to #1609 / #1608. Addresses the 5 unresolved findings from the PR #1609 round-1 polish review. ## Findings addressed | Tag | Severity | Fix | Commits | |-----|----------|-----|---------| | **B1** | BLOCKER | Document `ingestBufferSize` in `config.example.json` near other ingestor knobs. Default `50000`, comment text from review. | `f0b4e411` | | **M1** | MAJOR (option 1 from review) | Split receipt-time vs post-write liveness: add `SourceLivenessState.LastReceiptUnix` + `MarkReceipt`, stamp at the MQTT receipt callback, leave `LastMessageUnix` post-write only. Drop the double-stamp at receipt that masked write-path stalls. Surface both clocks via the ingestor stats file (`source_liveness`) and the server's `/api/healthz` (`ingest_liveness`, additive — older builds unaffected). | RED `fa78233d` / GREEN `bc81b544` | | **M1 (drop-log)** | MAJOR | Log every drop when buffer is at capacity. Removes the `n==1 \|\| n%1000` throttle that hid the first stall behind 1000 lost packets. The Submit drop branch only fires when the channel is at cap so volume is naturally bounded by the stall, not by an arbitrary modulo. | RED `a468763e` / GREEN `7b24fce5` | | **m1** | MINOR | Add `IngestBuffer.Stop()` and `Done()` so tests stop leaking the consumer goroutine that `Start()` spawns. Existing tests gain `t.Cleanup(b.Stop)`. Drain semantics: stop-before-Ready exits immediately; stop-after-Ready best-effort drains queued jobs. | RED `8430c822` / GREEN `78c9b223` | | **m2** | MINOR | `NewIngestBuffer(<1)` now logs a `[ingest-buffer] WARN` line on clamp so misconfigured `ingestBufferSize` values are visible instead of silently running a 1-slot queue. Test captures log output. | RED `62119ab4` / GREEN `815bfd02` | | **m3** | MINOR | Add godoc to `Submit` and `Ready` documenting the Start-before-Submit / Start-before-Ready ordering invariant. | `564a813b` | ## TDD discipline Each behavioral fix (M1, M1-drop-log, m1, m2) lands as a red-then-green pair. Red commits compile + run + fail on assertion, verified locally before the green commit. Per-finding red→green pairs are visible in the commit graph above. B1 and m3 are docs-only and ship as single commits (preflight script accepts them under the docs/comments exemption). ## Schema compatibility `/api/healthz` change is purely additive: `ingest_liveness` is only included when the ingestor publishes the new `source_liveness` field, so older ingestor + newer server combos are unaffected. Field order in the response stays stable for prior consumers. ## Test output - `go test -count=1 -timeout 180s ./cmd/ingestor/...` → green (160s) - `go test -count=1 -timeout 300s ./cmd/server/...` → green (48s) - Race-mode runs of the touched packages (`IngestBuffer|Liveness|Watchdog|Receipt|Healthz`) → green - Full-package race runs locally exceed the brief's 120s timeout on pre-existing slow integration tests (TestObsTimestampIndexMigration, TestNeighborEdgesBuilderDeltaScan); CI has the headroom. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass, no warnings. ## Files changed - `config.example.json` — B1 - `cmd/ingestor/ingest_buffer.go` — m1, m2, M1-drop-log, m3 - `cmd/ingestor/ingest_buffer_test.go` — m1, m2, M1-drop-log - `cmd/ingestor/mqtt_watchdog.go` — M1 - `cmd/ingestor/mqtt_watchdog_m1_test.go` — M1 (new) - `cmd/ingestor/main.go` — M1 (receipt callsite) - `cmd/ingestor/stats_file.go` — M1 (publish `source_liveness`) - `cmd/server/perf_io.go` — M1 (type + reader) - `cmd/server/healthz.go` — M1 (surface `ingest_liveness`) Original review reference: PR #1609 polish review by the M-axis bot. --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
|
bc1822e46c |
perf(load): chunked Load with early HTTP readiness (#1009) (#1596)
## What Switches the server's startup from a synchronous full-scan `PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that: 1. Streams transmissions+observations from SQLite in id-ordered chunks (default `chunkSize=10000`, configurable via `db.load.chunkSize`). 2. Closes `FirstChunkReady()` after the first chunk is merged — `main.go` binds the HTTP listener on that signal instead of blocking on the full multi-minute load. 3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every response while LoadChunked is in flight, flipping to `ready` once it completes (via `loadStatusMiddleware`). 4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB` clamps and the post-load index rebuild (`pickBestObservation` / `buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`). ## Why Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked load the listener binds within seconds; dashboards and probes can read partial data and see the `loading` status header until the background load finishes. ## Notes - `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`) is unchanged — it still waits for neighbor-graph build + initial `pickBestObservation` before reporting `ready:true`. `LoadChunked` only changes when the listener BINDS, not when it advertises ready. - `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on a tiny DB) before proceeding, and drains the load goroutine in the background with a logged error path. - Config Documentation Rule: `config.example.json` now documents `db.load.chunkSize` with a nested `_comment` describing the trade-off. ## Tests - `cmd/server/chunked_load_test.go` asserts: - (a) `FirstChunkReady` fires before `LoadChunked` returns - (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` → `ready` - (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via `OnChunkLoaded`) - (d) `Config.DBLoadChunkSize()` default 10000 + override - Red commit (`102a4c84`) lands the tests with stubs that fail on assertion — verified locally before the green commit. - Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite green (47s locally). Closes #1009 ## TDD red-commit exemption The original red commit `f878e15e` ("test(load): failing tests for chunked Load + early HTTP readiness") fails to **compile** rather than failing on an assertion, because it references symbols (`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`, `Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A compile error is NOT a valid red commit." This is claimed under the **net-new surface** exemption with the following justification: - LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize are all introduced by this PR — no prior implementation existed to refactor. There is no behaviour on master that the red commit could meaningfully assert against without first declaring the new symbols. - The cheapest "proper" alternative (split the red into two commits: stub-first + assertion-fail) was deferred because the test file unambiguously fails on missing-symbol — there is no risk of the test becoming a tautology against a pre-existing stub. - **Behaviour gating IS proven elsewhere on this branch.** Commit `799bde49` ("test(load): red — LoadChunked must mark indexes ready + not flip Complete on error") is a proper assertion-fail red against the same package, and commit `92cadd1d` is the matching green. Reviewers can verify the red→green pattern there. If a future reviewer wants the strict pattern, the follow-up is mechanical: split `f878e15e` into a stub-only commit followed by the assertion commit. Not done here to keep the rework cost proportional to the risk (zero, in this case). ## Preflight overrides - check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and `cmd/server/chunked_load_oldest_test.go` only. They run against per-test `t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single test) — they are NOT production schema migrations. No prod table is touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir fixture). --------- Co-authored-by: CoreScope Bot <bot@corescope.local> Co-authored-by: clawbot <bot@noreply.example.com> Co-authored-by: Kpa-clawbot <bot@example.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |
||
|
|
7421ead9b0 |
fix: bypass API limit clamps for internal UI requests. Revisit of issue #1540 (#1589)
This PR replaces the strict, hardcoded limits on API list endpoints (introduced in the recent security patch) with a new operator-configurable `listLimits` block. This change is needed as issue 1540's implementation introduced a 500max node limit on the live map or any other function that leverages the api/nodes backend. Previously, we attempted to bypass public caps for internal UI requests using a heuristic based on browser headers (`Sec-Fetch-Site`). Following review, we decided to drop that heuristic entirely to eliminate any security-by-browser-convention surface area. Instead, `queryLimit()` returns to its original, mathematically simple bounds-checking shape, and the absolute maximums are now drawn from `config.json`. This provides equal DoS protection against all callers while allowing server operators to tune the ceilings based on the size of their mesh (e.g. embedded devices can tighten the knobs, regional hubs can raise them). ### Changes Made: - **`config.go`**: Introduced a `ListLimits` config struct containing `PacketsMax`, `NodesMax`, `AnalyticsMax`, and `ChannelMessagesMax`. Added safe initialization to ensure default caps (10000, 2000, 200, 500 respectively) apply even if the block is omitted from the config. - **`clamp_limit.go`**: Deleted `isInternalUIRequest` entirely and restored `queryLimit` to its original signature (`r, def, max`). - **`routes.go`**: Replaced all hardcoded integer ceilings on list endpoints (`/api/packets`, `/api/nodes`, etc.) with `s.cfg.ListLimits.*`. - **`config.example.json`**: Added the `listLimits` block with documentation to guide new operators. - **`clamp_limit_test.go`**: Purged all header-heuristic testing. ### Verification: - All 611 backend unit tests pass (`npm run test:unit`). - Bounds-checking math continues to enforce hard DoS clipping exactly at the operator's specified configuration limit. --------- Co-authored-by: mc-bot <bot@openclaw.local> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
|
1bdb92de88 |
feat(#1574): operator-configurable liveMap.maxNodes (default 2000) (#1577)
Red commit: 94dc1d70a5a710271721d981cb5e36b7127b00dc Fixes #1574. cross-stack: justified — by design. Adds one server-side knob (`liveMap.maxNodes`) on the Go API and consumes it on the frontend (`public/live.js`) via the shared `/api/config/client` bootstrap in `public/roles.js`. Cannot land server-only or frontend-only without either dropping operator config (frontend-only) or leaving the literal in place (server-only). ## Problem (per triage) `public/live.js:2515-2516` hardcodes `/api/nodes?limit=2000` for the live-map node-load path. Reporter measured headroom at N=4300 and asked for an operator knob. Same `2000` magic also lives at `public/live.js:480` for the VCR-rewind `/api/packets?limit=2000`. ## Fix - New `liveMap.maxNodes` field in `Config` (default 2000). - `Config.LiveMapMaxNodes()` server-side clamp: `[100, 20000]`; zero/negative falls back to default. Defangs misconfig (e.g. 1M would OOM the SQLite read + JSON serialization path). - `/api/config/client` now returns `liveMapMaxNodes`. - `public/roles.js` reads it at bootstrap into `window.LIVE_MAP_MAX_NODES` (default 2000 to preserve behavior on stale caches). - `public/live.js` consumes `LIVE_MAP_MAX_NODES` at both the `/api/nodes` call sites (formerly :2515-2516) and the VCR-rewind `/api/packets` call (formerly :480) — single source of truth, in-scope per triage's "factor into a sibling const" suggestion. - `config.example.json` documents the knob with `_comment_maxNodes` per AGENTS.md config rule. ## TDD 1. **Red** (`94dc1d70`): added `test-issue-1574-live-map-max-nodes.js` (grep-asserts the literal is gone + `LIVE_MAP_MAX_NODES` / `liveMapMaxNodes` are wired + config example has the field) and `cmd/server/livemap_maxnodes_1574_test.go` (`/api/config/client` exposes `liveMapMaxNodes` + clamp table-driven cases). Stub `LiveMapMaxNodes()` returns 0 so the test compiles and fails on assertion, not import. 2. **Green** (this commit): real `LiveMapMaxNodes()` clamp + wire-up. All assertions pass; existing `cmd/server` suite still green. ## E2E note Frontend assertion is grep-based (literal removal + constant reference), in the established `test-issue-*` style used elsewhere (e.g. `test-issue-1189-live-iata-badge.js`). No Playwright change needed for a literal-replace; behavior validation is the server-side clamp + JSON shape tests. ## Out of scope No customizer UI change — operators set this in `config.json`, same pattern as `liveMap.propagationBufferMs`. Customizer surfacing can land as a follow-up if the operator wants it. --------- Co-authored-by: mc-bot <bot@corescope.local> Co-authored-by: Kpa-clawbot <bot@meshcore-analyzer> |
||
|
Kpa-clawbotandcorescope-bot
|
222bfdf6cf |
feat(perf): SQLite writer-lock wait/hold instrumentation per component (#1340) (#1594)
## What Per-component SQLite writer-lock instrumentation so the next neighbor-builder-style write-lock starvation (root cause of #1339, invisible to operators for ~3 days) is detectable from `/api/perf`. Adds `Store.WriterExec` / `Store.WriterTx` wrappers that gate every wrapped call on a package-level `writerMu` so the wait the SQLite driver hides becomes Go-visible, and record `wait_ms` + `hold_ms` + `contention_total` (wait_ms > 100ms) under a component tag. Per-component p50/p95/p99 + max are published to `/api/perf/write-sources` under `.writer_perf` via the existing ingestor stats-file path. Slow-writer log line (`[db-slow-writer] component=X duration=Yms query=<200ch>`) fires on `hold_ms > 500ms` (threshold overridable via `CORESCOPE_DB_SLOW_WRITER_MS` env var). ## Tagged call sites | Component | Location | |-----------|----------| | `mqtt_handler` | `InsertTransmission` (db.go) | | `neighbor_builder` | `buildAndPersistNeighborEdges` (neighbor_builder.go) | | `prune_packets` | `PruneOldPackets` (maintenance.go) | | `prune_observers` | `RemoveStaleObservers` + orphan-metrics cleanup (db.go) | | `prune_metrics` | `PruneOldMetrics` (db.go) | | `vacuum` | `RunIncrementalVacuum` + `CheckAutoVacuum`'s full VACUUM (db.go) | ## TDD red→green - **Red commit** `68de585b` — `cmd/ingestor/db_writer_perf_test.go` + `Store.Writer*` stubs at end of `db.go`. Test synthetically blocks the writer for 60s tagged `neighbor_builder`, then asserts `mqtt_handler.wait_ms.p99 > 50000ms` on concurrent inserts. Fails on the assertion (p99 = 0.0ms) with the stub — not a build error. - **Green commit** `6a9be174` — replaces stubs with real wait/hold/contention aggregator + wires every writer call site. Same test passes: ``` 2026/06/05 04:36:47 [db-slow-writer] component=neighbor_builder duration=60059.0ms query=COMMIT --- PASS: TestWriterStarvationVisibleInPerf (60.40s) PASS ok github.com/corescope/ingestor 60.408s ``` ## Scope discipline - **API**: no public `Store`/`DB` signature change. Only additive exports. - **Server**: extends existing `/api/perf/write-sources` JSON with `.writer_perf` — does **not** add a new route, does **not** replace `handlePerf`. Empty `.writer_perf` map when paired with an older ingestor. - **Read/write invariant** (#1283) preserved: all instrumentation lives on the ingestor's writer connection. - **Files touched** (6 total): `cmd/ingestor/db.go`, `cmd/ingestor/db_writer_perf_test.go`, `cmd/ingestor/maintenance.go`, `cmd/ingestor/neighbor_builder.go`, `cmd/ingestor/stats_file.go`, `cmd/server/perf_io.go`, `config.example.json`. ## Deferred (acceptance items NOT in this PR) - **`mbcap_persist` component tag** — `RunMultibyteCapPersist`'s tx is intentionally NOT wrapped in this PR to stay within the implementation brief's 3-files-outside-whitelist budget. One-file follow-up to instrument. - **CI smoke test** asserting "neighbor-builder hold_ms < 1000ms on 100k-obs fixture" — deferred to a separate PR per the brief; this PR is scoped to instrumentation only. ## Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=runtime — the async-migration gate flagged five `instrumentedExec` / wrapped-`tx.Exec` lines on `DELETE FROM observer_metrics`, `UPDATE observers`, `DELETE FROM observer_metrics`, `DELETE FROM observations`, `DELETE FROM transmissions`. These are **not** schema migrations — they are the existing runtime prune / retention queries that already ran sync against `s.db.Exec` / `tx.Exec` on every retention cycle on master. This PR only swapped the surface call (sync → sync, via the wrapper) to record wait/hold timing; no new sync schema work was introduced. Behavior on production data is identical to master. Also: red commit's synthetic `UPDATE nodes SET name = name WHERE 0` is a test-only stub designed to acquire the writer without mutating any row (the `WHERE 0` is a no-op predicate). Fixes #1340 --------- Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandcorescope-bot
|
1b112f0b08 |
feat(memlimit): GOMEMLIMIT via runtime.maxMemoryMB in server + ingestor (#1010) (#1595)
Red commit: 929da3c6dcc1b619c27478291125d1c91323db8f — CI: https://github.com/Kpa-clawbot/CoreScope/commit/929da3c6dcc1b619c27478291125d1c91323db8f/checks Fixes #1010. ## What Adds `GOMEMLIMIT` support to both `cmd/server` and `cmd/ingestor` per the locked triage scope on #1010. Precedence (env wins): 1. `GOMEMLIMIT` env var 2. `runtime.maxMemoryMB` config field (new) 3. Server only: implicit `packetStore.maxMemoryMB * 1.5` (existing #836 behavior, unchanged when `runtime.maxMemoryMB` is absent) 4. Otherwise unset — default Go behavior preserved (backwards compatible) Each startup logs a `[memlimit]` line echoing the effective source/limit, or an "unset → default" note when neither is set. ## Changes - `cmd/ingestor/memlimit.go` — new, `applyMemoryLimit(runtimeMaxMB, envSet)`. - `cmd/ingestor/memlimit_test.go` — new, env/config/none/precedence assertions. - `cmd/ingestor/config.go` — new `RuntimeConfig{MaxMemoryMB int}` field. - `cmd/ingestor/main.go` — wires `applyMemoryLimit` into startup right after `LoadConfig`. - `cmd/server/config.go` — new `RuntimeConfig` + `cfg.Runtime` field. - `cmd/server/main.go` — adds explicit `runtime.maxMemoryMB` precedence over packetStore-derived; existing `warnIfMemlimitUnderprovisioned` (#1264) unchanged. - `config.example.json` — new `runtime` block with `_comment_runtime_maxMemoryMB` per the Config Documentation Rule. - `README.md` — sizing-table row with ≥1.5× working set floor + death-spiral warning. ## TDD - Red: `929da3c6` — ingestor `applyMemoryLimit` stub returns `(0,"none")`; four tests fail on assertions (`expected source=env, got "none"`, etc.) — no compile errors. - Green: `953ec9d8` — implements ingestor `applyMemoryLimit`, wires startup, threads `runtime.maxMemoryMB` through server too. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → clean (all gates pass, all warnings pass). ## Out of scope - `pprof`-verified GC-trigger acceptance criterion from the original issue — requires production tracing; the triage scope is the operator-tunable plumbing. - Container auto-detection of cgroup memory limit (already covered by #1264's `warnIfMemlimitUnderprovisioned`). --------- Co-authored-by: corescope-bot <bot@corescope> |
||
|
Kpa-clawbotandbot
|
7292d60fbe |
feat(#1508): config-driven disabled tabs in customizer modal (#1579)
# feat(#1508): config-driven disabled tabs in customizer modal Fixes #1508. ## Why The customizer modal mixes one-shot operator chrome (`branding`, `home`, `geofilter`, `export`) with daily-use viewer toggles (`theme`, `nodes`, `display`). Non-technical users get confused by the admin tabs and skip past the controls they actually need. There's no current way to hide individual tabs server-side — only via CSS, which doesn't prevent state mutation. ## What Adds a single operator knob: `customizer.disabledTabs` in `config.json`. The named tab ids are filtered out of `_renderTabs()` in `public/customize-v2.js` before render. - `config.example.json` — new `customizer` block, default `disabledTabs: []` (zero behavior change for existing operators). - `cmd/server/config.go` — new `CustomizerConfig` type, optional pointer on `Config`. - `cmd/server/routes.go` + `cmd/server/types.go` — `/api/config/client` now surfaces `customizer.disabledTabs` (always an array, empty when unset). - `public/customize-v2.js` — `_renderTabs()` filters by id. - `cmd/server/customizer_disabled_tabs_test.go` — RED-then-green tests covering both the configured-and-defaulted shapes. ## TDD trail 1. RED commit adds the failing tests + minimal `CustomizerConfig` stub so the package still compiles; both tests fail on the assertion (`body.customizer` is `<nil>`) — not on import. 2. GREEN commit wires the field through `/api/config/client` and the frontend tab filter; both tests pass. ## Scope 5 files. No new API surface, no UI for editing the list (operator edits `config.json` directly per the issue body). Backward-compatible: missing `customizer` block defaults the list to empty. --------- Co-authored-by: bot <bot@local> |
||
|
Kpa-clawbot
|
9b36b7c487 |
feat(#1518): add branding.homeUrl override for embedded deployments (#1576)
Red commit:
|
||
|
|
892eb2c02a |
fix(#1509): expose --nav-active-bg as a themeable token (#1571)
Red commit: 07a69e48ebc976caf4bf15f7a937e378a42ef718 (CI run: pending — PR triggers first run) Fixes #1509 ## Problem `--nav-active-bg` is defined in `public/style.css` (line 105) and used by every active-state nav link (`.nav-link.active`, `.nav-more-menu .nav-link.active`, plus the responsive blocks), but the customizer has never mapped it into `THEME_CSS_MAP`. Result: presets, per-operator overrides, and server-side `theme.*` config can recolor every other nav token (`navBg`, `navBg2`, `navText`, `navTextMuted`) — but the active-pill background stays stuck on the hardcoded `rgba(74, 158, 255, 0.15)` (light) / dark-mode equivalent. Themes look broken on the one element users stare at. ## Fix Triage-specified path, no scope creep: - Add `navActiveBg: '--nav-active-bg'` to `THEME_CSS_MAP` in `public/customize-v2.js`. - Surface in the Theme tab's advanced color list (`THEME_COLOR_KEYS` derives from the map; adding to `ADVANCED_KEYS` makes it render in the panel). - Add label + hint so the input is self-explanatory. - Seed defaults on the default preset's `theme` + `themeDark` so the rendered value matches today's hardcoded rgba and dark mode doesn't bleed the light value. - Document the new field in `config.example.json` per AGENTS.md config rule. ## TDD Red commit `07a69e48` adds `test-issue-1509-nav-active-bg.js` and wires it into the CI unit-test step. Assertions fail on master (`THEME_CSS_MAP.navActiveBg` is `undefined`; `applyCSS` does not write the variable). Green commit `29d22ff5` makes the assertions pass without touching any other test. ## Verification - `node test-issue-1509-nav-active-bg.js` → 3/3 pass on this branch, 0/3 on master - `node test-customizer-v2.js` → 59/60 (the 1 failure is pre-existing on master, not caused by this PR — same failure with the diff stashed) - pr-preflight: clean (all gates pass) --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@meshcore-analyzer> |
||
|
Eldoon Nemar
|
d7cd9203ca |
Fixes #1165: add OSM/Stamen tile providers with per-provider Leaflet layer control. (#1533)
List of changes too long to describe, so I'll hit high level. - Config now supports the json map tiles that were suggested by @Kpa-clawbot. - Leaflet map layer button appears in the top right of live.js and map.js (because all the work was already done on live.js... Added bonus) - Allows users to enter creds for OSM and Stamen to get enterprise related perks, in the config file - Added a default light map under customizer. Still suggest removing them all together and relying on the config - You can enable OSM and Stamen in the config without a license, but at your own risk!!! - Config comment explains where to register and the providers for osm, as well as the general limits per X interval - Updated tests (28) to address the changes made to the maps ### TDD Exemption **Reason**: Net-new UI surfaces (per `AGENTS.md`) This PR introduces a net-new UI surface (the multi-provider map tile selector). Under the `AGENTS.md` exemption for net-new UI surfaces, the absence of an initial failing (red) commit is permitted, as the UI was built first. However, the underlying public APIs are fully covered. The following tests serve as the first assertions for these new APIs: - `window.MC_createLayerControl`: Asserted in `MC_createLayerControl handles Auto mode and explicit layers correctly` - `window.MC_setDarkTileProvider` & `window.MC_getDarkTileProvider`: Asserted in `MC_setDarkTileProvider persists to localStorage...` - `window.MC_setLightTileProvider` & `window.MC_getLightTileProvider`: Asserted in `MC_setLightTileProvider persists to localStorage...` - `window.MC_initTileRegistry`: Asserted in `MC_initTileRegistry(true) dispatches mc-tile-provider-changed` - `applyTileFilter`: Asserted in `applyTileFilter sets invert CSS for inverted dark provider...` - Cross-tab synchronization: Asserted in `Cross-tab storage event re-dispatches mc-tile-provider-changed` |
||
|
|
65bd954b17 |
feat(config): make observer health thresholds configurable (closes #1552) (#1556)
Closes #1552. ## What Make observer `Online` / `Stale` / `Offline` thresholds operator-configurable via `config.json`'s existing `healthThresholds` block — and **raise the defaults** from 10 min / 60 min to **60 min / 1440 min (1 h / 24 h)** so they match the node thresholds and stop producing flap out of the box. ⚠️ **This is a default behavior change.** Operators who want the old aggressive 10-min Online threshold must opt in via: ```json "healthThresholds": { "observerOnlineMinutes": 10 } ``` ## Why Per #1552: the `600000` / `3600000` constants in `public/observers.js` were not tunable, *and* 10 min is wrong as a default. Wide-geo, low-traffic meshes legitimately see observers go quiet for >10 min between reports, and operators behind a CDN (#1551) get cached `last_seen` values that can push the observer 15+ min behind reality — guaranteeing flap at the 10-min threshold. The meshat.se operator (43 observers, v3.8.3) reports exactly this pattern. Defaults raised from 10 / 60 minutes to 60 / 1440 minutes (1 h / 24 h) to match the node thresholds for consistency and eliminate flap on low-traffic / CDN-fronted instances. Operators wanting the old 10-min Online behavior can set `observerOnlineMinutes: 10` in config. ## Changes Backend (`cmd/server/config.go`): - `HealthThresholds` gains `ObserverOnlineMinutes` / `ObserverStaleMinutes` (int). - `GetHealthThresholds()` defaults to **60 / 1440** when zero/absent. - `ToClientMs()` emits `observerOnlineMs` / `observerStaleMs`, picked up by the existing `/api/config-public` → `roles.js` `Object.assign(HEALTH_THRESHOLDS, …)` pipeline. `config.example.json`: new `observerOnlineMinutes` / `observerStaleMinutes` keys (60 / 1440) + `_comment_observerThresholds` explaining the rationale and opt-out. Frontend: - `public/observers.js` `healthStatus()` — reads from `window.HEALTH_THRESHOLDS.observerOnlineMs / observerStaleMs`, falls back to **3600000 / 86400000** (matching the new Go defaults for the pre-`/api/config-public` window). - `public/observer-detail.js` — same refactor (was previously hardcoded `600000` + misusing `nodeDegradedMs` for the Stale boundary). ## Backward compat - API shape: unchanged — only adds two optional keys. - Config: unchanged keys / no renames. - Default behavior: **changed** — operators relying on the implicit 10/60 must opt in (one config line). ## TDD - RED 1 (`ee19058f`): assertions on the new fields + `ToClientMs` keys + `healthStatus` reading from `window.HEALTH_THRESHOLDS`. CI: [failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945264822). - GREEN 1 (`30cfbf7a`): configurability landed (defaults still old 10/60). CI: [success](https://github.com/Kpa-clawbot/CoreScope/actions/runs/26945220598). - RED 2 (`2649cf35`): pin new 60/1440 defaults — empty-config Go path + JS `healthStatus` with no `HEALTH_THRESHOLDS`. CI must fail. - GREEN 2 (`5ef85bca`): bump Go defaults to 60/1440, JS fallbacks to 3600000/86400000, `config.example.json` updated. CI must pass. ## Preflight Clean (exit 0). `cross-stack` ack in commit messages — single feature spans Go + JSON + JS readers. ## Not in scope - Customizer UI for editing the thresholds (config-only per issue). - Node/infra thresholds (unchanged). - The deeper observer-flap root cause (#1551 cache-control is a separate PR in flight). --------- Co-authored-by: corescope-bot <bot@corescope> Co-authored-by: mc-bot <bot@meshcore.local> |
||
|
Kpa-clawbotandbot
|
367265eb59 |
feat(#1369): cross-domain embed support (CORS env override + ?embed=1 chrome suppression) (#1500)
Closes #1369. ## What Cross-domain embed support, shipped as two halves: ### Part A — CORS env override + read-only contract * `applyCORSEnv()` reads `CORS_ALLOWED_ORIGINS` (comma-separated, trimmed, empties dropped). Set in env → overrides `cfg.CORSAllowedOrigins`. Unset/empty → config.json value wins. * `Access-Control-Allow-Methods` tightened from `GET, POST, OPTIONS` → `GET, HEAD, OPTIONS`. The cross-domain surface is read-only by contract; same-origin admin writes don't go through preflight and are unaffected. * `config.example.json` adds `corsAllowedOrigins: []` + a comment explaining the env override and the embed URL pattern. * No wildcards introduced (still supported as `["*"]` for ops that opt in). No credentialed CORS. ### Part B — `?embed=1` chrome suppression * `shouldEmbedRoute(basePage, hashSearch)` — pure helper, allowlisted to `map` and `channels`, requires `embed=1` in the hash querystring. * `navigate()` toggles `body.embed` based on the helper. * CSS hides `.top-nav`, `[data-bottom-nav]`, `.nav-drawer`, `.nav-drawer-backdrop`, zeroes body padding/margin, reclaims `100dvh` for `#app.app-fixed`. Use: `<iframe src="https://analyzer.example/#/map?embed=1">`. For iframe-only display, no CORS entry is needed (the iframe loads the document, not a JSON API). The CORS allowlist only matters when the embedding origin's own JS calls `/api/*` directly. ## Tests | File | Asserts | Status | |---|---|---| | `cmd/server/cors_embed_1369_test.go` | 4 (env override, env-empty, env-trim, GET/HEAD contract, preflight POST rejected) | green | | `test-embed-mode-1369.js` | 9 (helper allowlist + param parsing) | green | | `cmd/server/cors_test.go` | existing | updated to read-only method-set assertion | TDD: 2 red commits (one per part, both compile, both fail on assertions) → 2 green commits. ## Out of scope (per the issue's narrow ask) * Other SPA routes do not honor `?embed=1` (their chrome makes layout assumptions; defer until requested). * No iframe sandboxing recommendation — that's the embedder's responsibility. * No CSP / `X-Frame-Options` change in this PR — frames are already permitted; add an explicit `frame-ancestors` policy in a follow-up if operators want to whitelist embedders at the HTTP layer too. ## Security notes (DJB lens) * Allowlist is exact-match, case-sensitive string compare — no normalization, no scheme/host parsing, no surprises. * No `Access-Control-Allow-Credentials` (would let third parties read auth'd state via cookies). * No reflection of arbitrary origins (every echoed origin came from the allowlist). * Methods narrowed to read-only; even a misconfigured allowlist can't grant cross-origin writes through this middleware. 🤖 Generated with OpenClaw --------- Co-authored-by: bot <bot@corescope.local> |
