A small follow-up to parts A to E of #2128: users can download all data the instance holds about them, and the server backs up `users.db`. Stacked on #2140. Review the commits after that branch. ## The situation - An instance with accounts holds personal data, but a user has no way to get a copy (GDPR articles 15 and 20). Self-delete already exists. - `users.db` is the only file the server writes, and nothing backs it up: `GET /api/backup` snapshots the analyzer database only. ## What this PR adds **Data export.** `GET /api/account/export` and a "Download my data" button on the account page give one JSON file with: - the profile, including activation and a pending email change; - sessions (times and user agent); - synced settings, own proposals, notification prefs and watches; - audit rows where the user is actor or target, with other accounts as an id only; - the mail log. Password hash, session and link tokens and the unsubscribe token are left out: they are credentials. Each export is audited as `user.export`. **users.db backup** - Daily `VACUUM INTO` snapshots through `internal/users`, on by default when user management is on. They go to `backups/` next to `users.db`, the 7 newest are kept, the directory is 0700 and the files 0600. Rotation only touches `users-<timestamp>.db` files and never the snapshot it just wrote. Config: `userManagement.backup {enabled, dir, keep}`. - `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to keep off the server; audited as `user.backup`. - `docs/user-guide/accounts.md` gains a restore procedure. It covers what a restore brings back (accounts deleted after the snapshot, old passwords, revoked sessions) and how to clean that up afterwards. Spec: [`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md). ## Verification - `internal/users` with `-race` and the `cmd/server` suite pass locally, 30 new Go tests. The file-mode tests skip on Windows. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 27 of 27 steps locally. - On our production instance since 8 October 2026; the first snapshot there was 143 kB. ## Not in this PR - Importing an export into another instance. - Off-site upload of the snapshots. The admin download covers that by hand. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
18 KiB
Accounts (optional user management)
CoreScope runs without accounts by default: every page is public and settings live in each visitor's browser. Operators can turn on accounts. The dashboard stays public; logging in only adds things.
- Visitors can register with an email address, a display name and a password, and activate the account through a link mailed to that address.
- Admins get an admin area (account menu, Admin) with an overview, the user list (activate, disable, delete, promote) and the audit log, and can use the operator actions (geofilter save, prune, backup, perf reset) without the API key.
With the feature off (the default) the account routes are not registered: requests to
/api/auth/*, /api/account/* and /api/admin/* fall through to the normal page,
and no login or account control appears in the interface.
For operators
1. Get a Brevo account
- Create a free account at brevo.com. The free plan's daily volume is far more than account mails need.
- Verify the sender address or domain you will send from (Brevo: Senders, Domains & Dedicated IPs). Mail from an unverified sender is rejected or lands in spam.
- Create an API key (SMTP & API, API Keys).
2. Configure
Add to config.json (see config.example.json for every key):
"userManagement": {
"enabled": true,
"adminEmails": ["you@example.org"],
"publicBaseUrl": "https://corescope.example.org",
"mail": { "fromEmail": "noreply@example.org", "fromName": "My CoreScope" }
}
Provide the API key as mail.brevoApiKey or, better, the environment variable
CORESCOPE_BREVO_API_KEY. Restart the server. It refuses to start if the mail setup is
incomplete, and the log says what is missing.
| Key | Meaning |
|---|---|
adminEmails |
Addresses that become admin on activation. They cannot be demoted, disabled or deleted from the UI. Remove an address here first. |
publicBaseUrl |
The address visitors use. Every mail link is built from it, never from the request. It must match the browser origin, because state-changing requests from another origin are refused. |
dbPath |
Where accounts are stored. Default: users.db next to the analyzer database. |
sessionDays |
Login lifetime, extended while in use. Default 30, maximum 365. |
trustedProxies |
CIDRs of your reverse proxy, so the per-IP login limits see real client IPs. Without it, behind a proxy every client shares the proxy's IP for the per-IP limits (they are switched off when that IP is loopback or private). Per-address and per-account limits apply either way. |
mail.webhookSecret |
Enables delivery status (below). At least 16 characters. |
3. The first admin
Register at #/account/register with an address from adminEmails, click the activation
link and enter the password you chose. You are now admin and can promote others in
Admin, Users.
4. Delivery status (optional)
To see per user whether mails were delivered, bounced, blocked or marked as spam:
- Set
mail.webhookSecret(16+ random characters), orCORESCOPE_BREVO_WEBHOOK_SECRET. The webhook route exists only when a secret is set. - In Brevo, create a transactional webhook to
https://<publicBaseUrl>/api/mail/brevo/webhookwith these events: delivered, opened, clicked, soft bounce, hard bounce, invalid email, deferred, spam, blocked, error. Use bearer authentication with the same secret.
If Brevo cannot reach your instance, use Refresh status in a user's details instead: CoreScope then asks Brevo directly. "Opened" is indicative only. Some mail apps load tracking pixels automatically, and others block them.
When mail fails
- In Admin, Users, Resend mail sends a fresh activation link to a pending account.
- Activate activates a pending account by hand. The address is then unverified; the row shows "manual" and the audit log records who did it.
The admin area
Admin in the account menu (on phones: the account page) opens three tabs (four with channel proposals on):
- Overview: a Needs attention list when something applies (pending accounts older
than 24 hours, addresses whose mail bounces, accounts with 5 or more failed logins in
24 hours, MQTT sources that are disconnected or silent for 10 minutes), user figures
(accounts, registrations, active users, logins, mail of the last 7 days) and the system
status (version, uptime, MQTT sources, observers). It refreshes every minute while the
tab is visible. Each item links to the matching user list or audit entries.
With node notifications on, the Users card also shows the notification mails of the
last 24 hours against
maxMailsPerDay, and how many nodes are watched by how many users (counts only; watch lists are private). - Users: the user list. Bouncing mail only shows addresses whose mail bounces.
- Audit: every recorded action, newest first, filtered by action, period or user. Successful and failed logins are recorded without IP address and deleted after 90 days; a failed login for an address that has no account is not recorded. Other actions are kept.
- Proposals (only with channel proposals on): proposed hashtag channels by status, with Approve, Reject and Revoke and an optional note that the proposer sees. Approving asks first, because the channel becomes readable for every visitor.
Old #/admin/users links still work and open the Users tab.
Channel proposals (optional)
Hashtag channels are public by construction: the key is derived from the name, so
anyone who knows the name can read the channel. CoreScope decrypts only the hashtag
channels in hashChannels. With channel proposals on, logged-in users propose a
hashtag channel from Channels, Add channel, Propose for everyone, and admins decide
on the Proposals tab.
"userManagement": {
"channelProposals": { "enabled": true, "maxPending": 100, "maxApproved": 128, "perUserPerDay": 5 }
}
- An approved channel is decrypted by the ingestor from its next refresh (once a minute, no restart) and listed for every visitor on the Channels page, also before it has traffic and regardless of the region filter. Messages received before the approval stay encrypted.
- Revoking stops decryption of new messages from the next refresh; stored messages
stay. A channel that is also in
hashChannelsorchannelKeysstays decrypted. - A name in
hashChannels, or achannelKeysname written with its leading#, cannot be proposed: the answer is "this channel is already decrypted on this instance". The comparison is case-sensitive. - A rejected name cannot be proposed again until 90 days after the decision; a revoked one can be proposed again at once, and the new proposer then replaces the old one (the audit log keeps both). Rejected and revoked proposals are deleted 90 days after the decision.
maxApprovedbounds the decryption work: every approved key is tried on every group message.perUserPerDayandmaxPendinglimit proposals (HTTP 429).- The ingestor reads
users.dbread-only, fromuserManagement.dbPathor next to the analyzer database. SetdbPathexplicitly when the server and the ingestor are not given the same analyzer database path (for exampleDB_PATHset for one of them). Both log the absolute path they use at startup (server:[users] user management enabled: db=..., ingestor:[proposals] reading approved channels from ...). - Names: at most 31 bytes including the
#(MeshCore stores 32 with the terminator), no invisible or control characters (blank fillers such as U+3164 and spaces other than the plain space count as invisible), case-sensitive, not Public. Emoji work, except subdivision flags (England, Scotland, Wales), which use invisible tag characters. Names that need the zero-width non-joiner (U+200C), as some Persian and Urdu spellings do, are refused as well.
Node notifications (optional)
With node notifications on, logged-in users choose nodes to watch and get a mail when a watched node changes state. Admins can also watch the instance.
"userManagement": {
"notifications": { "enabled": true, "intervalMinutes": 5, "perUserPerDay": 20, "maxMailsPerDay": 100, "maxWatchesPerUser": 50 }
}
- Events: a watched node goes offline or comes back, using the node page's thresholds:
offline once it has not been heard for the
healthThresholdssilent hours of its role (heard is the latest of its last advert, the packet store's newest packet involving it, and for repeaters and rooms its last relay hop). A watched node also reports a battery belowbatteryThresholds.lowMv, and recovers atlowMv + 100mV. Admins can add a new foreign node (once per node) and an observer going offline (observerStaleMinutes) or back (observerOnlineMinutes). - A check runs every
intervalMinutes(the first one an interval after startup, and only after the startup load). All changes for one user in one check go into one mail. The first check of a newly watched node stores its state without a mail; states are kept inusers.db, so a restart does not mail again. - While ingest is stale (the newest packet in the packet store is older than 30 minutes,
for example when the MQTT broker or the ingestor is down), the offline checks for
nodes and observers pause: their states stay as they were and nothing is mailed for
them. Battery and foreign-node checks go on. The server log says when the pause
starts and ends. After the feed comes back, offline reports wait one silent window
(the role's silent hours for nodes,
observerStaleMinutesfor observers): until then older evidence counts as heard at the moment the feed came back, so the outage itself mails nobody. A node that died during the outage is reported up to one window late. The server keeps this grace in memory: a restart after the recovery and inside that window loses it, and so does a server that starts after an outage that already ended before its first check (for example a reboot of the whole host after a long outage). - Limits:
perUserPerDaymails per user andmaxMailsPerDayin total, both over a rolling 24 hours. A change over a limit, for an account that is not active, for a bouncing address or while the user has notifications off is recorded and never mailed later.maxWatchesPerUserbounds each watch list. maxMailsPerDaycounts notification mail only; activation, password-reset and address-change mail are not counted and are not limited by it. They share the mail provider's daily quota, though (Brevo's free tier allows 300 a day for the whole account), so keepmaxMailsPerDaywell below that quota. The default 100 leaves 200 a day for account mail and for any other sender on the same provider account.- Every mail carries a one-click unsubscribe link and
List-Unsubscribeheaders; the link turns notification mails off for that account and nothing else. The account page turns them back on. - Watch lists are private: admins see counts on the overview, not lists.
Backups
users.db holds password hashes and addresses. The server keeps its own snapshots of
it: at startup when the newest snapshot is older than 24 hours (or there is none), then
about every 24 hours (the check runs hourly). A snapshot is a complete copy named users-<YYYYMMDD-HHMMSS>.db (UTC),
readable by the server's user only, in backups/ next to users.db. After each new
snapshot the oldest ones beyond keep are deleted, never the one just written.
Temporary files of an interrupted snapshot (users-<YYYYMMDD-HHMMSS>.db.tmp) are
deleted once they are older than 24 hours; other files in that directory are never
touched. A failed snapshot is logged ([users] backup failed: ...) and the next run
tries again.
The server creates the default directory, or a backup.dir that does not exist yet,
with mode 0700. An existing backup.dir is not tightened: make it readable by the
server's user only yourself (chmod 700 <dir>).
A deleted account, including one the user deleted themselves, stays in the snapshots
for up to keep days (one snapshot a day), and in every downloaded copy until you
delete that copy.
"userManagement": {
"backup": { "enabled": true, "dir": "", "keep": 7 }
}
| Key | Meaning |
|---|---|
backup.enabled |
Default true, also when the block is absent. false turns the snapshots off. |
backup.dir |
Where snapshots go. Empty: backups/ next to users.db. A relative path is relative to the server's working directory. |
backup.keep |
How many snapshots are kept. Default 7, also for 0 or less. |
Snapshots on the same disk are lost with that disk. To keep a copy elsewhere, log in as
an admin and open /api/admin/users-backup in the same browser: it downloads a fresh
snapshot (corescope-users-<YYYYMMDD-HHMMSS>.db) and records it in the audit log
(user.backup). Store the download encrypted: it holds every password hash and address.
The analyzer database has its own backup route, GET /api/backup.
What a restore brings back. users.db returns to the moment of the snapshot:
- accounts deleted after it, including accounts users deleted themselves;
- old passwords of users who changed them since;
- sessions that were logged out or revoked since (valid again until they expire, for anyone who still holds the cookie);
- activation, reset and email change links used since (valid again until they expire).
The audit log is replaced too, so note the deletions before you overwrite it.
Restore (not automated). The commands use the sqlite3 command-line tool; replace
2026-10-08 12:00:00 with the snapshot's time from its file name (UTC).
-
Stop the server.
-
List the accounts deleted since the snapshot, from the current
users.db:sqlite3 users.db "SELECT target_user_id, action, datetime(at, 'unixepoch') FROM audit_log WHERE action IN ('user.delete', 'user.delete.self') AND at >= CAST(strftime('%s', '2026-10-08 12:00:00') AS INTEGER);" -
Copy the snapshot over
users.db, and deleteusers.db-walandusers.db-shmif they exist. -
Still with the server stopped, disable the accounts from step 2 (replace
12, 34with their ids) and clear all sessions and links, so everyone logs in again:sqlite3 users.db "UPDATE users SET status = 'disabled' WHERE id IN (12, 34); DELETE FROM sessions; DELETE FROM tokens;" -
Start the server.
-
Delete the accounts from step 2 again in the admin area. Deleting there, not in
sqlite3, also replaces the addresses in the mail history with a hash and records the deletion in the audit log.
A snapshot made by a newer CoreScope version is refused at startup ("database schema
version N is newer than this binary supports"): run that version or newer. Deleting
users.db removes all accounts and nothing else.
For users
- Register: Log in, Create an account, then click the link in the mail within 48 hours and enter your password to finish. Registering a pending (not yet activated) address again replaces the password and sends a new link. Registering an address that is already activated changes nothing; its owner gets a notice mail instead.
- Forgot password: Log in, Forgot password? The link works once, for one hour, and logs out all your devices.
- My account: change your display name, password or address, see your logged-in devices, or delete your account. A new address is confirmed from a link sent to it, and your old address gets a notice. Changing your password logs out your other devices.
- Download my data: My account, My data, Download my data saves one JSON file with
your profile (including when and by whom the account was activated), logged-in devices,
synced settings, proposals, notification settings and watched nodes, and the history of
your account and of the mail sent to you (including the address each mail went to).
An address change you have not confirmed yet is included as
pendingEmail. Password hashes and login, link and unsubscribe tokens are not in it, and neither is the server's internal bookkeeping of which notifications it already sent. Other accounts in your history appear as a number only. Each download is recorded in the audit log (user.export). - Settings sync: while you are logged in, your settings follow you: your nodes, favorites, theme and customizer settings, saved packet filters, and the filter, sort and view choices of each page. Log in on another browser or phone and they are restored; later changes reach your other devices within about a minute, or when you return to the tab. A node or favorite added on one device is never dropped by another, and one you removed stays removed. Saved packet filters are stored in your account as you typed them.
- Propose a channel (when the operator turned channel proposals on): Channels, Add channel, type the hashtag name, Propose for everyone. My account, My proposals shows the status and the admin's note. Approved channels are readable for every visitor of the instance.
- Node notifications (when the operator turned them on): Notify me on a node's page watches it; My account, Notifications lists your watched nodes, turns mails on or off, chooses the events, and Watch my nodes copies your synced My nodes. You get at most one mail per check, and every mail has a link that turns the mails off.
- Never synced: channel keys and decrypted messages, the API key, panel and column sizes, collapsed panels and map positions. They stay in the browser where you set them.
- Logging out asks whether to keep your settings on this device (the default) or remove them; your account keeps its copy either way, and channel keys are never removed. An automatic logout (expired session) keeps everything on the device. On a shared computer choose Remove my settings from this device: settings kept on the device are added to the account of the next person who logs in there.
- My account, Settings sync shows when your settings were last synced, has Sync now, and Delete synced settings from my account, which removes the account's copy only. The settings on your devices stay, and your next change starts a new copy.
Admins can see your display name and email address in the Users list, and when you logged in or someone failed to log in to your account (kept 90 days, no IP address).