mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 15:57:33 +00:00
417c8dd00a243f29822f34e5c9aaadbd2e522213
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efitenandClaude Opus 5.5
|
e149568c6c |
feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on
|
||
|
efitenandClaude Opus 5.5
|
a08f2f4e7c |
fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
de364a8bb6 |
feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a watched node goes offline, comes back, or reports a low battery. Admins can add instance events: a new foreign node, and an observer going offline or back. This covers the per-user mail part of #775. Stacked on #2139 (part D, which builds on #2138). Review the commits after that branch. ## The situation A sysop learns that a repeater went silent, or that its battery is running down, only by opening CoreScope. #775 asks for notifications. Accounts (part A) now give a verified address to mail and a mailer with delivery status. ## What this PR adds **Events.** They use the instance thresholds, so a mail does not disagree with the node page: - `node.offline`: silent for the role's `healthThresholds` window. Last heard comes from the packet store, and repeaters and rooms count relayed traffic (#1598). - `node.battery`: advert telemetry below `batteryThresholds.lowMv`, recovered at `lowMv + 100`. - Admins only: `foreign.new` (once per node) and `observer.offline`. **Store.** Schema v5 with `notification_prefs`, `notification_watches` and `notification_state`. **Server** (only with `userManagement.notifications.enabled`) - A notifier next to the janitor, every `intervalMinutes` (5). The evaluator is a pure function of watches, prefs, stored states and node snapshots. - No burst after a restart or a new watch: a subject's first evaluation stores its state without mailing. The loop waits for the startup load. While ingest is stale (newest packet older than 30 minutes) the offline checks pause, and after recovery they wait one silent window. - One mail per user per check. Limits: 20 per user and 100 per instance per rolling 24 hours, 50 watches per user. A change over a limit is recorded and never mailed later. - Every mail has a one-click unsubscribe (`List-Unsubscribe` and `List-Unsubscribe-Post`). The GET only redirects to a confirm page, so link scanners cannot unsubscribe anyone. Node names are cleaned of control and bidi characters before they go into a mail. - Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE /api/account/notifications/watches/{pubkey}`, `POST /api/account/notifications/watch-my-nodes` (copies the synced "my nodes" list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message` gains `Headers`. **Frontend.** A "Notify me" toggle on the node pages, a Notifications section on the account page (`#/account?section=notifications`), an unsubscribe page, and the notification mail count on the admin overview. Spec: [`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md). ## Performance Each check reads watches, prefs and states from `users.db` (one query each), the watched nodes from the analyzer DB in chunks of 500, and last-heard times from the packet store. | Measurement | Result | |---|---| | One check on our staging instance, 130,000 packets in memory, 1 watcher | 6.9 ms, of which 0.93 ms holding the store read lock | | Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes | 1.75 ms | No work is added to ingest, broadcast or any request path. ## Verification - `internal/users` and `internal/mailer` with `-race`; the `cmd/server` suite plus `-race` on the notifier tests; 74 new Go tests. The same Windows-only failure as noted in #2138 applies. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 26 of 26 steps locally. With notifications off the new steps fail, so they do test the flag. - On our staging and production instance since 7 October 2026. ## Not in this PR - Other channels (Discord, Telegram, webhooks). Detection is separate from delivery, so one can be added. - The topology, RF and anomaly alerts of #775. - If the whole server starts after a feed outage that already ended, the grace window is unknown and a watcher can get one wrong offline mail. The user guide says so. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |