meshcore-analyzer/docs/release-notes/v3.16.0.md

v3.16.0

47 commits since v3.15.0: 23 fix, 9 feat, 6 docs, 2 test, 2 perf, 2 ci, 1 ui, 1 deps, 1 chore. Nine feats, so minor not patch.

This release finishes the liveness work of v3.15.0. There, the server stopped counting a relay hop as "heard" when its short prefix is shared with other repeaters. The ingestor still did that for the stored last-seen time, and now it no longer does, so some repeaters keep an older Last Heard after the upgrade. The new features are optional: personal MQTT logins for observers, each observer's latest neighbour report, and the radio setting a packet was heard on. The rest is mostly pages that could show an older answer or a wrong number.

Read this before upgrading

Repeaters that are only heard in paths keep an older Last Heard

A node's Last Heard also follows the last-seen time the ingestor stores for it. The ingestor refreshed that time from relay hops by a weaker rule than the server uses since v3.15.0, so a repeater could still look heard on traffic that was really another one. Three changes close that:

• A relay hop only refreshes a node when its prefix belongs to that node alone (#2334, closes #2329). Before, when a prefix matched several nodes, the ingestor picked the one that neighbours the previous hop.
• Only hops a packet has already passed count, which means flood packets (#2308, closes #2307). In a direct packet the hops in the path are still to come, and in a trace they are the route that was asked for. Tracing or messaging a switched-off repeater kept it looking heard.
• On the server, relay evidence has to come from the path of the same observation, not from another observation of the same packet (#2298, closes #2266).

What you will see after the upgrade:

• On a mesh with 1-byte paths, relay hops practically stop refreshing Last Heard. In a sample from a production database (the last 200,000 observations, about 24 hours, 1,818 nodes), none of the 506,545 one-byte hop instances had a prefix of its own. A repeater on such a mesh stays Active on its own packets, such as its adverts.
• In the same sample, 2.0% of resolved hops no longer count (12,894 of 652,074). 7 of the 1,053 nodes that relay hops refreshed lost all of those refreshes in that window; 1 of those 7 sent no advert in it either.
• The green "actively relaying" mark on a repeater's Last Relayed row now needs certain relay evidence within healthThresholds.relayActiveHours (#2336, closes #2331). Forwarding another node's advert counts, sending its own advert does not. The yellow "alive (idle)" mark is gone: no mark does not mean the repeater is idle or off. The Last Relayed time itself still includes shared prefixes.

Nothing to configure. As in v3.15.0: if a repeater you know is on the air shows Uncertain or Stale, its prefix is shared and its last advert is older than its silent window.

With user accounts on: back up users.db first

users.db is migrated from schema version 6 to 8 at the first start (#2306, #2348), also when you do not turn on observer logins. A v3.15 binary refuses it after that ("database schema version 8 is newer than this binary supports"). To roll back, you need the copy from before the upgrade: the daily backups in backups/ next to users.db, or your own. Without user accounts nothing changes.

Building from source needs Go 1.26, and the image moves to Alpine 3.24

#2297 updates the Go dependencies after a scan of the published image found 19 high and 20 medium advisories in them, and moves the runtime image from Alpine 3.20, which is out of support, to Alpine 3.24.

• The go.mod files of the server, the ingestor and internal/users now ask for Go 1.26.0 (was 1.22). The Docker build and CI already use 1.27. If you build the binaries yourself with an older Go, it downloads a newer toolchain, or refuses to build with GOTOOLCHAIN=local.
• In Docker, the bundled mosquitto, Caddy and supervisor now come from Alpine 3.24. The local build in the PR had mosquitto 2.1.2, Caddy v2.11.4 and supervisor 4.3.0. The user and group ids of the caddy and mosquitto accounts did not change, so mounted volumes keep their owner.

Observer MQTT logins

All three need user accounts (userManagement.enabled), an EMQX broker and userManagement.observerLogins.enabled. Off by default. Setup, with a check and a rollback for every step, is in docs/observer-mqtt-logins.md.

• A personal MQTT login per user (#2306, closes #2319) - observers usually share one broker login, and anyone who has it can publish as any observer. With this on, a user requests a personal login on the account page, CoreScope generates it and mails it, and EMQX asks CoreScope whether that login may connect and publish. A personal login can only publish for its own observers. The first publish for an observer nobody owns yet binds it to that user, up to 20 per account. Admins get a mail when a known observer is claimed, and can reassign it or revoke the login. Logins CoreScope does not manage keep working through the broker's own users, so the shared login goes on working during the move. The owner also gets "My observers go offline" mails once notifications are on. Any account can claim an observer nobody owns yet, so the doc suggests assigning known observers to their owners before you announce the feature.
• One credentials mail for every uploader (#2306, #2347, #2349) - ready-to-paste blocks for meshcoretomqtt, the observer firmware CLI (on MQTT slot 4, which is empty on a fresh install), Home Assistant, OpenHop, and the plain values. The firmware and OpenHop blocks also turn on the neighbour report described below. Older firmware does not know that command, and the mail says to skip the line then.
• Lock the shared login out of moved observers (#2348) - list the broker's shared login names in observerLogins.sharedUsernames. The shared login can then no longer publish as an observer once its owner has published with a personal login. The admin page shows which observers are confirmed this way. An observer that still runs a second uploader on the shared login loses that uploader; the denial in the log names the observer.

Observers

• Latest neighbour report (#2300, closes #2270, #2271, #2330) - for observers that send a /neighbors report, the observers list has a sortable Neighbors column with the time of the last report and the number of neighbours. The observer page lists every neighbour by name, with SNR, when it was heard and its scopes. A report the broker replays after a restart does not make a dead observer look fresh, and a purged observer's report is removed with it.
• The radio setting a packet was heard on (#2292, part of #1869) - CoreScope kept only an observer's current frequency, bandwidth, SF and CR, so after a retune every older packet looked as if it was heard on the new setting. It now keeps each observer's radio history and works out the setting per observation, also per radio on nodes with more than one. The packet detail shows it, the packet filter takes radio, freq, bw, sf and cr (for example sf == 8), and the observer page shows the history once the setting has changed. The history starts at the upgrade, from each observer's current setting. Packets heard between a retune and the observer's next status message, minutes later, count for the old setting.
• The observer page shows its details first (#2339) - name, status and radio appear as soon as they arrive, without waiting for the analytics. An analytics error no longer hides them.
• The observer page stays on the observer you picked (#2323, closes #2316) - a slow answer for an earlier observer or period no longer replaces the one on screen. A new visit now asks for the 7 days it shows; before, it could ask for the period of the previous visit.
• Compare no longer counts transmissions as receptions (#2346) - an observer's own transmitted packets were counted as received, which made the overlap between two observers look higher.

Nodes

• Node previews show the same fields as the full page (#2315, closes #2309) - Last Relayed, Traffic share and Hash Prefix in the side and tablet previews. The tablet preview now shows the node's name.
• "View example packet" (#2296, closes #2291) - the observed-path links now say that they open one example packet.
• Battery badges (#2282, closes #2232) - node analytics showed SVG source text and oversized icons instead of the battery status icon.
• LoRa settings respect hidden nodes (#2276, closes #2268) - the optional radio settings feature (rfParams) no longer returns blacklisted or hidden nodes, also from its cache.

Pages and navigation

• Browser tab titles name the page (#2287, closes #2259) - for example Node abababababab | Reach | CoreScope, with your own site name kept. Useful with many tabs open.
• Maps and charts load from your own server (#2299) - Leaflet, leaflet.heat and Chart.js come from vendor/ instead of unpkg.com. Visitors' browsers no longer fetch these scripts from a third party, and the map and charts work where unpkg is blocked. Map tiles are not affected by this change. The API docs page (/api/docs) still loads Swagger UI from unpkg.
• Navigation - the links no longer overlap More when the account controls appear late (#2286, closes #2216), the More menu keeps keyboard focus when it rearranges, and a late packets answer no longer pulls you back from the page you went to (#2333, closes #2332, #2335).
• Packets - All time survives a reload and a shared link (#2279, closes #2231). The HB column no longer shows 4, a value that cannot exist (#2351): it read the hash size at the wrong place when observers heard the same packet with a different route type. In the last 300,000 observations of a production database, about 4,300 had a different route type than their packet's first observation.
• Analytics - Nodes counts roles within the selected region or area (#2275, closes #2213), and column resize handles stay on their header (#2274, closes #2217).

Coverage and reach

• My coverage and the observer filter on the noise layer (#2326) - on the Mobile RX coverage page, the noise layer showed every companion's samples whatever you selected.
• English tooltips on the coverage leaderboard (#2280) - the score column now explains itself: every cell of about 150 m adds 1, divided by the number of observers that heard something there.
• Reach waits for a free slot instead of answering "busy" at once (#2249) - at most 2 uncached reach reports still run at the same time. The next one now waits up to 3 seconds for a slot; at most 8 wait, and any further one gets the busy answer (429) at once. On a database with 10.5 million observations, uncached reports took from about 9 to over 60 seconds, so there the wait mostly ends in the same 429, 3 seconds later. It helps with two tabs, or switching between 7, 14 and 30 days.
• An interrupted reach report is not cached (#2277, closes #2258) - it could leave an empty or partial result in the cache.

Server and ingestor

• Live packets flow during the startup clean-up (#2328) - the ingestor held incoming MQTT messages until the whole packet-retention sweep at startup had finished. Ingest now opens before that sweep and writes between its batches. No timing was measured. This is a partial fix for #2325, which stays open.
• Distance analytics - after a chunked startup the distance index is built on first use instead of at once (#2278, closes #2227), and a background fill can no longer crash the server or mark an incomplete index as ready (#2284, closes #2281).

Faster

• Bridge and coverage scores of the neighbour graph, on a test graph of 2,000 nodes: 8.2 to 0.82 seconds and 4.9 to 0.69 seconds, with allocations per run from about 2.1 GB and 1.5 GB to about 2.5 MB and 2.2 MB (#2293, closes #2226).
• The health check (/api/healthz) no longer walks every packet in memory once startup is done: 1,229 to 82 microseconds per probe with 100,000 packets (#2305, closes #2301).

Docker and operations

• Containers are healthy only when startup is done (#2285, closes #2229) - the healthchecks of the five compose files and the deploy check of manage.sh use /api/healthz. manage.sh update now records the version and build time of what it checked out, and handles env files with Windows line endings. Only applies if you take the new compose files or manage.sh.
• The backup guide takes a consistent copy (#2345, closes #703) - docs/deployment.md now backs up through /api/backup and checks the download, instead of copying a running meshcore.db, which can miss data that is still in the write-ahead log. It also describes the restore.
• The unused Dockerfile.go and Dockerfile.node are removed (#2314, closes #2310).

Tests, CI and docs

• The API reference now describes the packet-list parameters (#2294), six diagnostics endpoints (#2303), and node resolution and battery history (#2338). The list of undocumented routes went from 24 to 16.
• Old release notes and planning documents moved to docs/release-notes/ and docs/archive/ (#2313, closes #2311; #2321, closes #2318).
• CI runs the JavaScript unit tests before the Go checks (#2322, closes #2317), and runs the browser tests on Ubuntu 24.04, because the browser install failed on 26.04 runners (#2350).
• Two unstable tests are fixed (#2273, closes #2272; #2289, closes #2288).

Issues closed

33 issues, from the 47 pull requests in this range. Collected from each pull request's closing references rather than from commit text.

• #703 feat: SQLite database replication, live backup copies
• #2213 Analytics Nodes shows global role totals when a region is selected
• #2216 bug(nav): primary link floor and late account controls can overlap More
• #2217 bug(analytics): column resize handles attach to page instead of static headers
• #2226 perf: graph score warmups delay startup and allocate gigabytes
• #2227 perf: chunked startup still eagerly builds the lazy distance index
• #2229 bug(operators): correct update metadata, readiness checks and CRLF startup
• #2231 bug(packets): All time does not survive reload or a shared link
• #2232 bug(node analytics): battery badge renders SVG source and mis-sizes icons
• #2258 reach: a cancelled build can cache empty scopes or a partial degree snapshot
• #2259 feat: Set HTML