mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 14:38:28 +00:00
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
613 lines
27 KiB
Go
613 lines
27 KiB
Go
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/meshcore-analyzer/users"
|
|
)
|
|
|
|
var evNow = time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
|
|
|
|
const (
|
|
evUser = int64(1)
|
|
evAdmin = int64(2)
|
|
evAdmin2 = int64(3)
|
|
)
|
|
|
|
var (
|
|
evPkA = strings.Repeat("a", 64)
|
|
evPkB = strings.Repeat("b", 64)
|
|
evPkF = strings.Repeat("f", 64)
|
|
evPkG = strings.Repeat("e", 64)
|
|
)
|
|
|
|
func evAgo(d time.Duration) string { return evNow.Add(-d).Format(time.RFC3339) }
|
|
func evMv(v int) *int { return &v }
|
|
|
|
// evInput: one user watching evPkA with the default node events, two
|
|
// admins without prefs, no stored states.
|
|
func evInput() notifyInput {
|
|
return notifyInput{
|
|
Now: evNow, Health: (&Config{}).GetHealthThresholds(), LowMv: 3300,
|
|
Accounts: map[int64]users.User{
|
|
evUser: {ID: evUser, Role: users.RoleUser, Status: users.StatusActive},
|
|
evAdmin: {ID: evAdmin, Role: users.RoleAdmin, Status: users.StatusActive},
|
|
evAdmin2: {ID: evAdmin2, Role: users.RoleAdmin, Status: users.StatusActive},
|
|
},
|
|
Prefs: []users.NotifyPrefs{{UserID: evUser, Enabled: true, Events: []string{users.NotifyNodeOffline, users.NotifyNodeBattery}}},
|
|
Watches: []users.NotifyWatch{{UserID: evUser, Pubkey: evPkA}},
|
|
Nodes: map[string]notifyNode{},
|
|
Heard: map[string]time.Time{},
|
|
}
|
|
}
|
|
|
|
func evStateRow(uid int64, event, subject, state string) users.NotifyState {
|
|
return users.NotifyState{NotifyKey: users.NotifyKey{UserID: uid, Event: event, Subject: subject}, State: state, ChangedAt: evNow.Add(-time.Hour)}
|
|
}
|
|
|
|
func evState(res notifyResult, uid int64, event, subject string) (string, bool) {
|
|
for _, s := range res.States {
|
|
if s.UserID == uid && s.Event == event && s.Subject == subject {
|
|
return s.State, true
|
|
}
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
func TestEvaluateFirstEvaluationStoresWithoutChanges(t *testing.T) {
|
|
in := evInput()
|
|
in.Watches = append(in.Watches, users.NotifyWatch{UserID: evUser, Pubkey: evPkB})
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(48 * time.Hour)}
|
|
in.Nodes[evPkB] = notifyNode{Pubkey: evPkB, Name: "Bravo", Role: "companion", LastSeen: evAgo(time.Hour), BatteryMv: evMv(3100)}
|
|
res := evaluateNotifications(in)
|
|
if len(res.Changes) != 0 {
|
|
t.Fatalf("changes on the first evaluation: %+v", res.Changes)
|
|
}
|
|
for _, c := range []struct{ event, subject, want string }{
|
|
{users.NotifyNodeOffline, evPkA, users.NotifyBad},
|
|
{users.NotifyNodeOffline, evPkB, users.NotifyGood},
|
|
{users.NotifyNodeBattery, evPkB, users.NotifyBad},
|
|
} {
|
|
if got, ok := evState(res, evUser, c.event, c.subject); !ok || got != c.want {
|
|
t.Errorf("%s %s = %q (stored %v); want %q", c.event, c.subject[:4], got, ok, c.want)
|
|
}
|
|
}
|
|
if _, ok := evState(res, evUser, users.NotifyNodeBattery, evPkA); ok {
|
|
t.Error("battery state stored for a node without telemetry")
|
|
}
|
|
for _, s := range res.States {
|
|
if !s.ChangedAt.Equal(evNow) {
|
|
t.Errorf("ChangedAt = %v; want the input's Now", s.ChangedAt)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestEvaluateOfflineAndBack(t *testing.T) {
|
|
in := evInput()
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(24 * time.Hour)}
|
|
in.States = []users.NotifyState{evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood)}
|
|
res := evaluateNotifications(in)
|
|
ch := res.Changes[evUser]
|
|
if len(ch) != 1 || ch[0].Event != users.NotifyNodeOffline || ch[0].Subject != evPkA || ch[0].To != users.NotifyBad ||
|
|
ch[0].Name != "Alpha" || !ch[0].At.Equal(evNow) {
|
|
t.Fatalf("changes = %+v; want one offline change for Alpha", ch)
|
|
}
|
|
// One minute inside the 24 h companion window: still online, nothing to write.
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(24*time.Hour - time.Minute)}
|
|
if res := evaluateNotifications(in); len(res.Changes) != 0 || len(res.States) != 0 {
|
|
t.Fatalf("unchanged online node produced %+v", res)
|
|
}
|
|
// Stored bad, advert 30 h old, but the packet store heard it an hour ago.
|
|
in.States = []users.NotifyState{evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyBad)}
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(30 * time.Hour)}
|
|
in.Heard[evPkA] = evNow.Add(-time.Hour)
|
|
if ch := evaluateNotifications(in).Changes[evUser]; len(ch) != 1 || ch[0].To != users.NotifyGood {
|
|
t.Fatalf("changes = %+v; want one back-online change", ch)
|
|
}
|
|
}
|
|
|
|
// Liveness is last_seen or the store's certain last heard (#2260): a
|
|
// repeater whose advert is days old but that certainly relayed an hour ago is
|
|
// online; one with nothing certain inside the silent window is offline. There
|
|
// is no separate relay input any more: last_relayed also counts 1-byte prefix
|
|
// matches of other repeaters and kept a silent repeater online for days.
|
|
func TestEvaluateCertainHeard(t *testing.T) {
|
|
cases := []struct {
|
|
name, role, lastSeen string
|
|
heard time.Duration
|
|
want string
|
|
}{
|
|
{"repeater certainly relaying", "repeater", evAgo(80 * time.Hour), time.Hour, users.NotifyGood},
|
|
{"room certainly relaying, role case", "Room", evAgo(80 * time.Hour), time.Hour, users.NotifyGood},
|
|
{"repeater silent exactly 72 h", "repeater", evAgo(72 * time.Hour), 0, users.NotifyBad},
|
|
{"repeater 71 h", "repeater", evAgo(71 * time.Hour), 0, users.NotifyGood},
|
|
{"repeater last certain packet 73 h ago", "repeater", evAgo(80 * time.Hour), 73 * time.Hour, users.NotifyBad},
|
|
{"no timestamp at all", "repeater", "", 0, users.NotifyBad},
|
|
}
|
|
for _, c := range cases {
|
|
in := evInput()
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Role: c.role, LastSeen: c.lastSeen}
|
|
if c.heard > 0 {
|
|
in.Heard[evPkA] = evNow.Add(-c.heard)
|
|
}
|
|
if got, _ := evState(evaluateNotifications(in), evUser, users.NotifyNodeOffline, evPkA); got != c.want {
|
|
t.Errorf("%s: %q; want %q", c.name, got, c.want)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestEvaluateBatteryHysteresis(t *testing.T) {
|
|
cases := []struct {
|
|
prev string
|
|
mv int
|
|
want string
|
|
change bool
|
|
}{
|
|
{"", 3299, users.NotifyBad, false},
|
|
{"", 3300, users.NotifyGood, false},
|
|
{"", 3350, users.NotifyGood, false},
|
|
{users.NotifyGood, 3299, users.NotifyBad, true},
|
|
{users.NotifyGood, 3300, users.NotifyGood, false},
|
|
{users.NotifyBad, 3399, users.NotifyBad, false},
|
|
{users.NotifyBad, 3400, users.NotifyGood, true},
|
|
}
|
|
for _, c := range cases {
|
|
in := evInput()
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(time.Hour), BatteryMv: evMv(c.mv)}
|
|
in.States = []users.NotifyState{evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood)}
|
|
if c.prev != "" {
|
|
in.States = append(in.States, evStateRow(evUser, users.NotifyNodeBattery, evPkA, c.prev))
|
|
}
|
|
res := evaluateNotifications(in)
|
|
got, stored := evState(res, evUser, users.NotifyNodeBattery, evPkA)
|
|
if !stored {
|
|
got = c.prev
|
|
}
|
|
ch := res.Changes[evUser]
|
|
if got != c.want || (len(ch) == 1) != c.change {
|
|
t.Errorf("prev %q, %d mV: state %q, changes %+v; want %q, change %v", c.prev, c.mv, got, ch, c.want, c.change)
|
|
continue
|
|
}
|
|
if c.change && (ch[0].Event != users.NotifyNodeBattery || ch[0].BatteryMv == nil || *ch[0].BatteryMv != c.mv) {
|
|
t.Errorf("prev %q, %d mV: change = %+v", c.prev, c.mv, ch[0])
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestEvaluateAbsentNodeIsOfflineOnce(t *testing.T) {
|
|
in := evInput() // evPkA is watched but not in the analyzer database
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyNodeBattery, evPkA, users.NotifyBad),
|
|
}
|
|
ch := evaluateNotifications(in).Changes[evUser]
|
|
if len(ch) != 1 || ch[0].Event != users.NotifyNodeOffline || ch[0].To != users.NotifyBad || ch[0].Name != evPkA[:8] {
|
|
t.Fatalf("changes = %+v; want one offline change named by the key prefix", ch)
|
|
}
|
|
in.States[0] = evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyBad)
|
|
if res := evaluateNotifications(in); len(res.Changes) != 0 || len(res.States) != 0 {
|
|
t.Fatalf("absent node evaluated again: %+v", res)
|
|
}
|
|
}
|
|
|
|
func TestEvaluateForeignOncePerAdmin(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs[0].Events = append(in.Prefs[0].Events, users.NotifyForeignNew) // a non-admin with an admin event
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin, Enabled: true, Events: []string{users.NotifyForeignNew}})
|
|
in.Nodes[evPkF] = notifyNode{Pubkey: evPkF, Name: "Foxtrot", Foreign: true, LastSeen: evAgo(time.Hour)}
|
|
res := evaluateNotifications(in)
|
|
if len(res.Changes[evAdmin]) != 0 {
|
|
t.Fatalf("first evaluation mailed: %+v", res.Changes[evAdmin])
|
|
}
|
|
if s, ok := evState(res, evAdmin, users.NotifyForeignNew, foreignBaselineSubject); !ok || s != users.NotifyGood {
|
|
t.Fatalf("baseline row = %q, %v", s, ok)
|
|
}
|
|
if s, ok := evState(res, evAdmin, users.NotifyForeignNew, evPkF); !ok || s != users.NotifyTold {
|
|
t.Fatalf("existing foreign node = %q, %v; want told", s, ok)
|
|
}
|
|
if _, ok := evState(res, evUser, users.NotifyForeignNew, evPkF); ok {
|
|
t.Fatal("non-admin evaluated for foreign.new")
|
|
}
|
|
|
|
// Next tick: a second foreign node appears; a second admin opts in now.
|
|
in.States = res.States
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin2, Enabled: true, Events: []string{users.NotifyForeignNew}})
|
|
in.Nodes[evPkG] = notifyNode{Pubkey: evPkG, Name: "Golf", Foreign: true}
|
|
res = evaluateNotifications(in)
|
|
ch := res.Changes[evAdmin]
|
|
if len(ch) != 1 || ch[0].Subject != evPkG || ch[0].To != users.NotifyTold || ch[0].Name != "Golf" {
|
|
t.Fatalf("admin changes = %+v; want Golf once", ch)
|
|
}
|
|
if len(res.Changes[evAdmin2]) != 0 {
|
|
t.Fatalf("second admin's first evaluation mailed: %+v", res.Changes[evAdmin2])
|
|
}
|
|
|
|
in.States = append(in.States, res.States...)
|
|
if res := evaluateNotifications(in); len(res.Changes) != 0 {
|
|
t.Fatalf("foreign nodes mailed twice: %+v", res.Changes)
|
|
}
|
|
}
|
|
|
|
func TestEvaluateObserverOfflineAndBack(t *testing.T) {
|
|
cases := []struct {
|
|
prev string
|
|
age time.Duration
|
|
want string
|
|
change bool
|
|
}{
|
|
{"", 2 * time.Hour, users.NotifyGood, false},
|
|
{"", 25 * time.Hour, users.NotifyBad, false},
|
|
{users.NotifyGood, 24 * time.Hour, users.NotifyGood, false},
|
|
{users.NotifyGood, 24*time.Hour + time.Minute, users.NotifyBad, true},
|
|
{users.NotifyBad, 2 * time.Hour, users.NotifyBad, false},
|
|
{users.NotifyBad, 60 * time.Minute, users.NotifyGood, true},
|
|
}
|
|
for _, c := range cases {
|
|
in := evInput()
|
|
in.Prefs = []users.NotifyPrefs{{UserID: evAdmin, Enabled: true, Events: []string{users.NotifyObserverOffline}}}
|
|
in.Watches = nil
|
|
in.Observers = []notifyObserver{{ID: "OBS1", Name: "Roof", LastSeen: evAgo(c.age)}}
|
|
if c.prev != "" {
|
|
in.States = []users.NotifyState{evStateRow(evAdmin, users.NotifyObserverOffline, "OBS1", c.prev)}
|
|
}
|
|
res := evaluateNotifications(in)
|
|
got, stored := evState(res, evAdmin, users.NotifyObserverOffline, "OBS1")
|
|
if !stored {
|
|
got = c.prev
|
|
}
|
|
ch := res.Changes[evAdmin]
|
|
if got != c.want || (len(ch) == 1) != c.change || (c.change && ch[0].Name != "Roof") {
|
|
t.Errorf("prev %q, age %v: state %q, changes %+v; want %q, change %v", c.prev, c.age, got, ch, c.want, c.change)
|
|
}
|
|
}
|
|
in := evInput()
|
|
in.Prefs = []users.NotifyPrefs{{UserID: evAdmin, Enabled: true, Events: []string{users.NotifyObserverOffline}}}
|
|
in.Observers = []notifyObserver{{ID: "OBS2"}}
|
|
if s, _ := evState(evaluateNotifications(in), evAdmin, users.NotifyObserverOffline, "OBS2"); s != users.NotifyBad {
|
|
t.Fatalf("observer without last_seen = %q; want bad", s)
|
|
}
|
|
}
|
|
|
|
func TestEvaluateOnlyChosenEventsAndKnownAccounts(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs[0].Events = []string{users.NotifyNodeBattery, users.NotifyObserverOffline} // observer.offline: the user is no admin
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Role: "companion", LastSeen: evAgo(48 * time.Hour), BatteryMv: evMv(4000)}
|
|
in.Observers = []notifyObserver{{ID: "OBS1", LastSeen: evAgo(time.Hour)}}
|
|
res := evaluateNotifications(in)
|
|
if len(res.States) != 1 || res.States[0].Event != users.NotifyNodeBattery {
|
|
t.Fatalf("states = %+v; want only node.battery", res.States)
|
|
}
|
|
in.Accounts = map[int64]users.User{} // the user was deleted after the read
|
|
if res := evaluateNotifications(in); len(res.States) != 0 {
|
|
t.Fatalf("deleted user evaluated: %+v", res.States)
|
|
}
|
|
}
|
|
|
|
func TestEvaluateChangesAreOrderedPerUser(t *testing.T) {
|
|
in := evInput()
|
|
in.Watches = []users.NotifyWatch{{UserID: evUser, Pubkey: evPkB}, {UserID: evUser, Pubkey: evPkA}}
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(48 * time.Hour), BatteryMv: evMv(3000)}
|
|
in.Nodes[evPkB] = notifyNode{Pubkey: evPkB, Name: "Bravo", Role: "companion", LastSeen: evAgo(48 * time.Hour)}
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkB, users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyNodeBattery, evPkA, users.NotifyGood),
|
|
}
|
|
var got []string
|
|
for _, c := range evaluateNotifications(in).Changes[evUser] {
|
|
got = append(got, c.Event+" "+c.Name)
|
|
}
|
|
if want := "node.offline Alpha|node.offline Bravo|node.battery Alpha"; strings.Join(got, "|") != want {
|
|
t.Fatalf("order = %q; want %q", strings.Join(got, "|"), want)
|
|
}
|
|
}
|
|
|
|
// 100 users x 50 watches over 2,000 nodes: the spec's Performance figure
|
|
// (5,000 state rows per event type). Run with -bench and quote the result
|
|
// in the PR.
|
|
func BenchmarkEvaluateNotifications(b *testing.B) {
|
|
in := notifyInput{Now: evNow, Health: (&Config{}).GetHealthThresholds(), LowMv: 3300,
|
|
Accounts: map[int64]users.User{}, Nodes: map[string]notifyNode{}, Heard: map[string]time.Time{}}
|
|
for i := 0; i < 2000; i++ {
|
|
pk := fmt.Sprintf("%064x", i)
|
|
in.Nodes[pk] = notifyNode{Pubkey: pk, Role: "repeater", LastSeen: evAgo(time.Duration(i) * time.Minute), BatteryMv: evMv(3200 + i%300)}
|
|
}
|
|
for u := int64(1); u <= 100; u++ {
|
|
in.Accounts[u] = users.User{ID: u, Role: users.RoleUser, Status: users.StatusActive}
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: u, Enabled: true, Events: users.NodeNotifyEvents})
|
|
for w := 0; w < 50; w++ {
|
|
in.Watches = append(in.Watches, users.NotifyWatch{UserID: u, Pubkey: fmt.Sprintf("%064x", (int(u)*37+w*11)%2000)})
|
|
}
|
|
}
|
|
in.States = evaluateNotifications(in).States
|
|
b.ResetTimer()
|
|
for i := 0; i < b.N; i++ {
|
|
evaluateNotifications(in)
|
|
}
|
|
}
|
|
|
|
// P7: an admin who loses the admin role loses their admin-event states, like
|
|
// opting out of those events, so a re-promotion starts a fresh baseline.
|
|
func TestEvaluateDemotedAdminDropsAdminEventStates(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin, Enabled: true,
|
|
Events: []string{users.NotifyNodeOffline, users.NotifyForeignNew, users.NotifyObserverOffline}})
|
|
in.Watches = append(in.Watches, users.NotifyWatch{UserID: evAdmin, Pubkey: evPkA})
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(time.Hour)}
|
|
in.Nodes[evPkF] = notifyNode{Pubkey: evPkF, Name: "Foxtrot", Foreign: true}
|
|
in.Observers = []notifyObserver{{ID: "OBS1", Name: "Roof", LastSeen: evAgo(time.Hour)}}
|
|
in.States = evaluateNotifications(in).States
|
|
if s, ok := evState(notifyResult{States: in.States}, evAdmin, users.NotifyForeignNew, foreignBaselineSubject); !ok || s != users.NotifyGood {
|
|
t.Fatalf("baseline row = %q, %v", s, ok)
|
|
}
|
|
|
|
// Demoted: the admin-event rows are dropped, the node rows stay, nothing mailed.
|
|
admin := in.Accounts[evAdmin]
|
|
admin.Role = users.RoleUser
|
|
in.Accounts[evAdmin] = admin
|
|
in.Nodes[evPkG] = notifyNode{Pubkey: evPkG, Name: "Golf", Foreign: true}
|
|
in.Observers[0].LastSeen = evAgo(48 * time.Hour)
|
|
res := evaluateNotifications(in)
|
|
if len(res.Changes) != 0 || len(res.States) != 0 {
|
|
t.Fatalf("demoted admin evaluated: %+v", res)
|
|
}
|
|
want := map[users.NotifyKey]bool{
|
|
{UserID: evAdmin, Event: users.NotifyForeignNew, Subject: foreignBaselineSubject}: true,
|
|
{UserID: evAdmin, Event: users.NotifyForeignNew, Subject: evPkF}: true,
|
|
{UserID: evAdmin, Event: users.NotifyObserverOffline, Subject: "OBS1"}: true,
|
|
}
|
|
if len(res.Drop) != len(want) {
|
|
t.Fatalf("drop = %+v; want %d admin-event keys", res.Drop, len(want))
|
|
}
|
|
for _, k := range res.Drop {
|
|
if !want[k] {
|
|
t.Errorf("dropped %+v", k)
|
|
}
|
|
}
|
|
dropped := map[users.NotifyKey]bool{}
|
|
for _, k := range res.Drop {
|
|
dropped[k] = true
|
|
}
|
|
var kept []users.NotifyState
|
|
for _, s := range in.States {
|
|
if !dropped[s.NotifyKey] {
|
|
kept = append(kept, s)
|
|
}
|
|
}
|
|
in.States = kept
|
|
if res := evaluateNotifications(in); len(res.Drop) != 0 {
|
|
t.Fatalf("dropped again: %+v", res.Drop)
|
|
}
|
|
|
|
// Re-promoted: Golf appeared and the observer went offline meanwhile; the
|
|
// first evaluation stores a new baseline and mails nothing.
|
|
admin.Role = users.RoleAdmin
|
|
in.Accounts[evAdmin] = admin
|
|
res = evaluateNotifications(in)
|
|
if len(res.Changes) != 0 || len(res.Drop) != 0 {
|
|
t.Fatalf("re-promoted admin: changes %+v, drop %+v", res.Changes, res.Drop)
|
|
}
|
|
for _, c := range []struct{ event, subject, want string }{
|
|
{users.NotifyForeignNew, foreignBaselineSubject, users.NotifyGood},
|
|
{users.NotifyForeignNew, evPkG, users.NotifyTold},
|
|
{users.NotifyObserverOffline, "OBS1", users.NotifyBad},
|
|
} {
|
|
if got, ok := evState(res, evAdmin, c.event, c.subject); !ok || got != c.want {
|
|
t.Errorf("%s %s = %q (stored %v); want %q", c.event, c.subject, got, ok, c.want)
|
|
}
|
|
}
|
|
}
|
|
|
|
// A user who watches nodes but has no preferences row (never opened the
|
|
// settings, or created before the notification schema) gets the defaults:
|
|
// enabled, node events on, admin events off.
|
|
func TestEvaluateWatcherWithoutPrefsGetsDefaults(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs = nil
|
|
in.Watches = []users.NotifyWatch{{UserID: evAdmin, Pubkey: evPkA}}
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(48 * time.Hour), BatteryMv: evMv(3000)}
|
|
in.Nodes[evPkF] = notifyNode{Pubkey: evPkF, Name: "Foxtrot", Foreign: true}
|
|
in.Observers = []notifyObserver{{ID: "OBS1", LastSeen: evAgo(time.Hour)}}
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evAdmin, users.NotifyNodeOffline, evPkA, users.NotifyGood),
|
|
evStateRow(evAdmin, users.NotifyNodeBattery, evPkA, users.NotifyGood),
|
|
}
|
|
res := evaluateNotifications(in)
|
|
var got []string
|
|
for _, c := range res.Changes[evAdmin] {
|
|
got = append(got, c.Event+" "+c.To)
|
|
}
|
|
if want := "node.offline bad|node.battery bad"; strings.Join(got, "|") != want {
|
|
t.Fatalf("changes = %q; want %q", strings.Join(got, "|"), want)
|
|
}
|
|
for _, s := range res.States {
|
|
if users.IsAdminNotifyEvent(s.Event) {
|
|
t.Errorf("admin event evaluated without opting in: %+v", s)
|
|
}
|
|
}
|
|
in.Accounts = map[int64]users.User{}
|
|
if res := evaluateNotifications(in); len(res.States) != 0 {
|
|
t.Fatalf("watcher without an account evaluated: %+v", res.States)
|
|
}
|
|
}
|
|
|
|
// A disabled user keeps being evaluated: states follow the node, nothing is
|
|
// mailed, and re-enabling does not mail what changed while off.
|
|
func TestEvaluateDisabledUserUpdatesStatesSilently(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs[0].Enabled = false
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin, Enabled: false, Events: []string{users.NotifyForeignNew}})
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(time.Hour)}
|
|
in.Nodes[evPkF] = notifyNode{Pubkey: evPkF, Name: "Foxtrot", Foreign: true}
|
|
in.States = evaluateNotifications(in).States
|
|
|
|
// While off: Alpha goes offline and Golf appears.
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(48 * time.Hour)}
|
|
in.Nodes[evPkG] = notifyNode{Pubkey: evPkG, Name: "Golf", Foreign: true}
|
|
res := evaluateNotifications(in)
|
|
if len(res.Changes) != 0 {
|
|
t.Fatalf("disabled users mailed: %+v", res.Changes)
|
|
}
|
|
if s, ok := evState(res, evUser, users.NotifyNodeOffline, evPkA); !ok || s != users.NotifyBad {
|
|
t.Fatalf("disabled user's offline state = %q, %v; want bad stored", s, ok)
|
|
}
|
|
if s, ok := evState(res, evAdmin, users.NotifyForeignNew, evPkG); !ok || s != users.NotifyTold {
|
|
t.Fatalf("disabled admin's foreign state = %q, %v; want told stored", s, ok)
|
|
}
|
|
in.States = append(in.States, res.States...)
|
|
|
|
// Re-enabled: the next evaluation has nothing to send.
|
|
in.Prefs[0].Enabled = true
|
|
in.Prefs[1].Enabled = true
|
|
if res := evaluateNotifications(in); len(res.Changes) != 0 || len(res.States) != 0 {
|
|
t.Fatalf("re-enabling mailed old changes: %+v", res)
|
|
}
|
|
}
|
|
|
|
// While ingest is stale nobody can tell a silent node from a silent
|
|
// MQTT feed: offline comparisons pause with their states unchanged;
|
|
// battery and foreign nodes go on.
|
|
func TestEvaluateIngestStalePausesOfflineChecks(t *testing.T) {
|
|
in := evInput()
|
|
in.IngestStale = true
|
|
in.Watches = append(in.Watches, users.NotifyWatch{UserID: evUser, Pubkey: evPkB})
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin, Enabled: true, Events: []string{users.NotifyForeignNew, users.NotifyObserverOffline}})
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(48 * time.Hour), BatteryMv: evMv(3100)}
|
|
in.Nodes[evPkF] = notifyNode{Pubkey: evPkF, Name: "Foxtrot", Foreign: true}
|
|
in.Observers = []notifyObserver{{ID: "OBS1", Name: "Roof", LastSeen: evAgo(48 * time.Hour)}}
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyNodeBattery, evPkA, users.NotifyGood),
|
|
evStateRow(evAdmin, users.NotifyObserverOffline, "OBS1", users.NotifyGood),
|
|
evStateRow(evAdmin, users.NotifyForeignNew, foreignBaselineSubject, users.NotifyGood),
|
|
}
|
|
res := evaluateNotifications(in)
|
|
for _, k := range []struct {
|
|
uid int64
|
|
event, subject string
|
|
}{{evUser, users.NotifyNodeOffline, evPkA}, {evUser, users.NotifyNodeOffline, evPkB}, {evAdmin, users.NotifyObserverOffline, "OBS1"}} {
|
|
if s, ok := evState(res, k.uid, k.event, k.subject); ok {
|
|
t.Errorf("%s %s written as %q while ingest is stale", k.event, k.subject, s)
|
|
}
|
|
}
|
|
if ch := res.Changes[evUser]; len(ch) != 1 || ch[0].Event != users.NotifyNodeBattery || ch[0].To != users.NotifyBad {
|
|
t.Fatalf("user changes = %+v; want only the battery", ch)
|
|
}
|
|
if ch := res.Changes[evAdmin]; len(ch) != 1 || ch[0].Event != users.NotifyForeignNew || ch[0].Subject != evPkF {
|
|
t.Fatalf("admin changes = %+v; want only the foreign node", ch)
|
|
}
|
|
|
|
in.IngestStale = false
|
|
res = evaluateNotifications(in)
|
|
if s, _ := evState(res, evUser, users.NotifyNodeOffline, evPkA); s != users.NotifyBad {
|
|
t.Fatalf("offline check after ingest resumed = %q; want bad", s)
|
|
}
|
|
if s, _ := evState(res, evAdmin, users.NotifyObserverOffline, "OBS1"); s != users.NotifyBad {
|
|
t.Fatalf("observer check after ingest resumed = %q; want bad", s)
|
|
}
|
|
}
|
|
|
|
// After a feed outage, evidence older than the recovery counts as heard at
|
|
// the recovery: no node or observer goes offline until a full silent window
|
|
// has passed since then. A state that is already bad is not lifted by it.
|
|
func TestEvaluateGraceAfterIngestResumes(t *testing.T) {
|
|
in := evInput()
|
|
in.Prefs = append(in.Prefs, users.NotifyPrefs{UserID: evAdmin, Enabled: true, Events: []string{users.NotifyObserverOffline}})
|
|
in.Watches = append(in.Watches, users.NotifyWatch{UserID: evUser, Pubkey: evPkB})
|
|
in.Nodes[evPkA] = notifyNode{Pubkey: evPkA, Name: "Alpha", Role: "companion", LastSeen: evAgo(25 * time.Hour)}
|
|
in.Nodes[evPkB] = notifyNode{Pubkey: evPkB, Name: "Bravo", Role: "companion", LastSeen: evAgo(48 * time.Hour)}
|
|
in.Observers = []notifyObserver{{ID: "OBS1", Name: "Roof", LastSeen: evAgo(25 * time.Hour)}, {ID: "OBS2", Name: "Mast", LastSeen: evAgo(48 * time.Hour)}}
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkA, users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyNodeOffline, evPkB, users.NotifyBad),
|
|
evStateRow(evAdmin, users.NotifyObserverOffline, "OBS1", users.NotifyGood),
|
|
evStateRow(evAdmin, users.NotifyObserverOffline, "OBS2", users.NotifyBad),
|
|
}
|
|
in.IngestResumedAt = evNow.Add(-10 * time.Minute)
|
|
if res := evaluateNotifications(in); len(res.Changes) != 0 || len(res.States) != 0 {
|
|
t.Fatalf("10 minutes after recovery: changes %+v, states %+v; want none", res.Changes, res.States)
|
|
}
|
|
|
|
// One silent window (24 h companions, observerStaleMinutes 1440) plus a
|
|
// minute after recovery, still no new evidence: offline.
|
|
in.Now = in.IngestResumedAt.Add(24*time.Hour + time.Minute)
|
|
res := evaluateNotifications(in)
|
|
if ch := res.Changes[evUser]; len(ch) != 1 || ch[0].Subject != evPkA || ch[0].To != users.NotifyBad {
|
|
t.Fatalf("node changes after the window = %+v; want Alpha offline", ch)
|
|
}
|
|
if ch := res.Changes[evAdmin]; len(ch) != 1 || ch[0].Subject != "OBS1" || ch[0].To != users.NotifyBad {
|
|
t.Fatalf("observer changes after the window = %+v; want Roof offline", ch)
|
|
}
|
|
|
|
// A node heard after the recovery behaves as without the grace.
|
|
in.Now = evNow
|
|
in.Nodes[evPkB] = notifyNode{Pubkey: evPkB, Name: "Bravo", Role: "companion", LastSeen: evAgo(5 * time.Minute)}
|
|
in.Observers[1].LastSeen = evAgo(5 * time.Minute)
|
|
res = evaluateNotifications(in)
|
|
if ch := res.Changes[evUser]; len(ch) != 1 || ch[0].Subject != evPkB || ch[0].To != users.NotifyGood {
|
|
t.Fatalf("node heard after recovery = %+v; want Bravo back online", ch)
|
|
}
|
|
if ch := res.Changes[evAdmin]; len(ch) != 1 || ch[0].Subject != "OBS2" || ch[0].To != users.NotifyGood {
|
|
t.Fatalf("observer heard after recovery = %+v; want Mast back online", ch)
|
|
}
|
|
}
|
|
|
|
func ownInput(events ...string) notifyInput {
|
|
in := evInput()
|
|
in.Watches = nil
|
|
in.Prefs = []users.NotifyPrefs{{UserID: evUser, Enabled: true, Events: events}}
|
|
in.Observers = []notifyObserver{
|
|
{ID: "obs1", Name: "Roof", LastSeen: evAgo(25 * time.Hour)},
|
|
{ID: "OBS2", Name: "Other", LastSeen: evAgo(25 * time.Hour)},
|
|
}
|
|
in.ObserverOwners = map[string]int64{"OBS1": evUser, "OBS2": evAdmin2}
|
|
in.States = []users.NotifyState{
|
|
evStateRow(evUser, users.NotifyObserverOfflineOwn, "obs1", users.NotifyGood),
|
|
evStateRow(evUser, users.NotifyObserverOfflineOwn, "OBS2", users.NotifyGood),
|
|
}
|
|
return in
|
|
}
|
|
|
|
func TestEvalOwnObserverOffline(t *testing.T) {
|
|
res := evaluateNotifications(ownInput(users.NotifyObserverOfflineOwn))
|
|
ch := res.Changes[evUser]
|
|
if len(ch) != 1 || ch[0].Subject != "obs1" || ch[0].Event != users.NotifyObserverOfflineOwn || ch[0].To != users.NotifyBad {
|
|
t.Fatalf("changes = %+v; want one bad change for obs1", ch)
|
|
}
|
|
if _, ok := evState(res, evUser, users.NotifyObserverOfflineOwn, "OBS2"); ok {
|
|
t.Fatal("someone else's observer was evaluated")
|
|
}
|
|
}
|
|
|
|
func TestEvalAdminWithBothGetsOneMail(t *testing.T) {
|
|
in := ownInput(users.NotifyObserverOffline, users.NotifyObserverOfflineOwn)
|
|
in.Prefs[0].UserID = evAdmin
|
|
in.ObserverOwners = map[string]int64{"OBS1": evAdmin}
|
|
in.States = []users.NotifyState{evStateRow(evAdmin, users.NotifyObserverOffline, "obs1", users.NotifyGood)}
|
|
res := evaluateNotifications(in)
|
|
n := 0
|
|
for _, c := range res.Changes[evAdmin] {
|
|
if c.Subject == "obs1" {
|
|
n++
|
|
if c.Event != users.NotifyObserverOffline {
|
|
t.Errorf("event = %q; want observer.offline", c.Event)
|
|
}
|
|
}
|
|
}
|
|
if n != 1 {
|
|
t.Fatalf("%d changes for obs1, want 1: %+v", n, res.Changes[evAdmin])
|
|
}
|
|
}
|
|
|
|
func TestEvalOwnObserverIngestStaleNoChange(t *testing.T) {
|
|
in := ownInput(users.NotifyObserverOfflineOwn)
|
|
in.IngestStale = true
|
|
res := evaluateNotifications(in)
|
|
if len(res.Changes[evUser]) != 0 {
|
|
t.Fatalf("changes during stale ingest: %+v", res.Changes[evUser])
|
|
}
|
|
}
|