`ota check` spun a worker task and blocked the caller in a delay loop
until it finished. It now queues the check and returns immediately;
repeating the command reads the cached result with its ID and age. The
flash path keeps its blocking barrier and its done flag becomes atomic.
Observer setters mutated live prefs and rolled back on save failure,
so a concurrent reader could see a value that never reached flash.
Edits now land on a candidate copy that the serializer writes; durable
RAM is replaced only after a verified save.
AsyncTCP handlers read NodePrefs and MQTTPrefs while the loop task
could be editing them, including the password compared at login. The
server now publishes a snapshot between commands and serves every
response from it under the existing mutex, and refuses to start when
that mutex is unavailable.
Per-field writes from the mesh task could be read mid-update by the
SNMP worker. Producers now publish a complete record through a mailbox
and the worker copies it into OID storage once per loop.
The alert reporter sampled slot monitoring, outage start and preset
name in three separate live reads, so a slot could change underneath
them. It now takes one outage snapshot per slot. The role stats JSON
uses the shared slot emitter, and both roles call bridge->loop() so
the loop task can publish its owned inputs to the worker.
Slot objects, counters and NTP state were mutated by the worker and
read live from the loop task, CLI and SDK callbacks through volatile
flags and the diagnostic singleton. The worker now owns them outright:
- SDK callbacks push event records; the worker applies the transitions.
- Configuration reaches the worker as a coalesced mailbox revision
carrying the prefs and radio metadata that caused it.
- Diagnostics read a published snapshot instead of live slots, and
radio stats are sampled on the loop task that owns them.
- Forced NTP sync and the NTP diagnostic become queued jobs, so no CLI
caller blocks the mesh loop; `get mqtt.runtime` reports sample age,
configuration revisions and lost SDK events.
esp_mqtt_client_init() succeeded without a registered event sink when
registration failed, and the WebSocket wrap returned a transport with
the unpadded SDK buffer when the padded allocation failed. Both paths
now fail cleanly instead, the IDF 4 WebSocket layout is pinned to the
validated patch level, and the connection flags esp-mqtt tasks write
become atomic.
Host-testable building blocks for the single-owner runtime model: a
value mailbox, a coalescing configuration mailbox, an async job handle,
an SDK event channel with its slot transition, a completion-driven NTP
schedule, a durable-preference commit and slot stats JSON emission.
Codex review of the previous commit found two real problems.
The routes go live in startLanMode()/startSetupMode(), one tick before
probeBoardCommands() first runs, so a status request can land in between.
That made _board_cmds a genuine cross-task race despite the comment
claiming otherwise, and — worse — let /api/status answer "no board
commands" before the board had been asked. The page reads status once,
so a browser that loaded in that window hid working FEM controls for the
whole session.
The probe result is now published and read under _mux, and /api/status
OMITS board_cmds until the probe has run, which is not the same as
reporting an empty list. The page treats the absent field as "not known
yet" and re-polls with backoff; the audit waits for it the same way,
rather than auditing a board it only half knows.
Probing from the start path instead would close the window, but
Callbacks::execCommand is documented as loop-task-only and `start
webconfig` is itself mid-handleCommand, so that trades a race for
re-entrancy.
The audit never executed fan.lo or fan.hi, which have no getter and so no
round-trip: they were only ever matched as strings. They now run for real,
including the four rejection paths, and the mock grew --probe-delay so the
retry path is exercised rather than assumed.
The FEM commands moved out of CommonCLI into Board::handleCommand(), and
the T-Beam 1W fan control was added there too, but the portal never caught
up: `radio.fem.txgain` and the fan commands were missing from the terminal
table entirely, while `radio.fem.rxgain` was offered on every board even
though a board with no hook now answers "??:" rather than "unsupported".
Whether a node answers these is a property of the board, not of the build,
so the page cannot know from the firmware version. Ask the board instead:
probeBoardCommands() runs each candidate getter once on the loop task at
startup and keeps the ones that answer, which needs no per-variant list
because Board::handleCommand() already reports whether it handled a
command. /api/status names the survivors and the page hides everything
else, so adding a command to a variant means one entry in WC_BOARD_CMDS
rather than an edit per board.
The two FEM keys also become Radio-panel toggles, gated the same way;
their `set` reaches the board hook through the existing config batch, so
only the allowlist and /api/config needed to grow.
webconfig_cli_audit.py now scans variants/*/*Board.cpp for the boards that
actually build the portal, checks set-only keys in the reverse direction,
and verifies every gate is a command some board answers; the mock gained
--board-cmds so both shapes of board are testable. `stop ota` joins
NOT_OFFERED: `start ota` cannot run from the portal, so it has nothing to
stop.
Two gaps found while tracing a ThinkNode M7 report of a missing OTA
manifest. The M7 itself was healthy in both channels; these are what the
investigation turned up around it.
Pass --prune to gen-slim-manifests.py in both workflows. The generator only
ever added, so an env that stopped being built kept a manifest pointing at a
release asset the KEEP_BUILDS=2 prune later deleted — a 404 for `ota update`
on any node still running it. The flasher-side commit adds the flag along
with an empty-input guard and --prune-limit, which fails the run rather than
pruning when a build looks like it silently lost envs.
Add "Verify production channel is baked in", which production lacked while
beta has had it from the start. The manifest base is a compile-time -D, so a
build that lost it or picked up the other channel's is invisible until a node
runs `ota check` — and the failure mode is production hardware OTA-ing itself
onto beta. Checked against real published binaries: passes the production M7
build, rejects the beta one.
OTA_MANIFEST_BASE_URL is now stated explicitly in the production env: block,
equal to build.sh's default, so the verify step asserts against the value the
build was actually handed instead of a second hardcoded copy that could drift.
Neither change triggers a build: .github/** is in both workflows' paths-ignore.
Adds a built-in preset for the Polish MeshCore community broker at
mqtt.marwoj.net: MQTT over TLS on port 8883, username/password auth
with credentials embedded in firmware, and the standard MeshCore topic
layout. The Let's Encrypt chain anchors at the existing ISRG_ROOT_X1
constant, so no new CA certificate is needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The preset table is fleet state, not a build detail. A slot's preset is stored
in /mqtt.json by name, and firmware that does not know a name does not merely
ignore it: MQTTPrefsSerializer repairs it to "none" and CommonCLI writes the
repaired file back to flash. A node that rolls back to a channel missing a
preset therefore loses that slot permanently, and re-upgrading does not bring
it back.
The parity gate compared preset names only, deliberately allowing URL, CA and
credential drift. It now requires src/helpers/MQTTPresets.h to be byte-identical
between the channels, which also catches that drift — the two channels are meant
to dial the same brokers.
That is only workable if the file holds no channel-specific code, so
mqttPresetEnforcesTokenExp() moves to the new MQTTPresetPolicy.h. It was the
sole difference between the two channels' copies, and with it moved they match
exactly today. Policy keyed off the table belongs there from now on; the table
itself stays pure data.
The older name-only comparison stays available without --exact for ad-hoc use,
and the checker's self-test now covers both modes, including that --exact
rejects a config-only change the name check waves through.
The hostname was built once in begin() from the node name, so `set name` left
the node advertising its old name to DHCP until the next reflash. It is now
rebuilt on rename through a new CommonCLICallbacks::onNodeNameChanged() hook,
and begin() shares the same helper.
What this cannot do is rename a live lease. IDF 4.4 states that a hostname
changed after the interface is up "would only be reflected once the interface
restarts/reconnects", and Arduino 2.x's WiFi.setHostname() does not touch a
running netif at all — it writes a static string that the netif reads at
bring-up. Forcing the issue would mean bouncing the link, which costs every
MQTT slot a reconnect; that is not a reasonable price for a rename, so the new
name lands at the next reconnect or boot instead.
Only automatic (Ethernet-preferred) links carry a hostname today, exactly as
before. Giving plain Wi-Fi observers one would change the DHCP identity of the
existing fleet and is a separate decision.
Follow-up to the DNS restore, from a second review round.
lwIP's DNS functions belong to the TCP/IP thread, and both halves were called
from elsewhere: the Ethernet snapshot from the Arduino event task, the Wi-Fi
snapshot and every restore from the MQTT task. Both now go through
esp_netif_tcpip_exec(), which is also what serializes the snapshot itself, so
it no longer needs atomics. Neither caller is the TCP/IP thread, so the call
cannot deadlock on itself.
The snapshot keeps the whole ip_addr_t instead of an IPv4 word. These builds
compile lwIP with IPv6 enabled, so reinterpreting a v6 server as v4 would have
installed four meaningless bytes as a resolver. RDNSS is disabled here, so a v6
server should not arise today; storing the tagged value means it stays correct
if that changes.
Restore now writes every slot, empty ones included, so returning to a
one-server network cannot leave the other medium's second server behind as a
fallback that only fails slowly. A lease that carried no resolver at all is not
recorded, so a medium that has never had DNS leaves the current resolver alone
instead of wiping it.
Two defects found by an adversarial review of the merge.
A link transition no longer stops a slot whose connect attempt is in flight.
Stopping one means esp_mqtt_client_stop(), which waits on the SDK's API mutex
and its task's stopped event with no bound; the client task notices only when
it returns from whatever transport call it is in. During teardown the bridge's
StopUnproven timeout contains that, but during a transition nothing does, so a
routine WiFi flap could freeze the sole MQTT worker for far longer than the 2 s
this path advertises, with the bridge's own stop handshake queued behind it.
The attempt is left to resolve instead: unlike a reconfigure, a transition
changes neither endpoint nor credentials, so an attempt that completes is
credited to the broker it actually reached and drops with the old route.
Failing back to Ethernet left the node using the WiFi network's DNS server.
lwIP keeps one global server list and IDF 4.4 has no per-interface retention,
so the medium that leased last owns DNS for every socket. Each link now
remembers the resolver its own DHCP lease installed and puts it back when it is
selected again; a medium that never held a lease leaves the current resolver
alone. Where the two networks are on different subnets this was a silent
outage: the link read as connected while every broker and NTP hostname failed
to resolve, until Ethernet's own DHCP renewal happened to fix it.
#54 lands first, so the network abstraction is reworked on top of the
per-slot client ownership model instead of alongside it.
Two resolutions are semantic, not textual:
- Link-transition teardown goes through the ownership API. It called
softDisconnect() on every started client, which bypassed client_state and
ignored the typed result. A slot that is still Starting now gets a real
stop: softDisconnect() returns immediately on a client that is not yet
connected, so its attempt would otherwise complete against the old route and
deliver a CONNECTED event indistinguishable from the new one's (F04). On a
medium switch the old route can still be briefly usable, so that is not
hypothetical. A connected client keeps the cheap bounded path, and a
quarantined one is left alone.
- NetworkLink::applyPowerPrefs() adopts WifiPowerSavePolicy. Its local
`2 ? MAX : NONE` mapping would have reintroduced F11 and read the new stored
value 3 (explicit `min`) as none. It also applies the setting when the link
starts already associated, which is the case the bridge used to cover.
The NTP probe from #54 keeps its validation and its one-attempt-per-server
bound, but resolves and gates on the selected link rather than on WiFi, so it
works on an Ethernet-preferred node.
- The room server checked canFlashAfterStop() only when the bridge was
running at OTA time. After a timed-out restart the bridge reads as
stopped while its unacknowledged task may still own TLS/client state,
so an OTA could erase and write flash under it. It now refuses while
the stop is unproven, after first reaping any late ack. The repeater
already gated unconditionally.
- MyMesh::loop() now reaps a late stop acknowledgement whenever it lands,
releasing the withheld queue and buffers, and restarts only when a
resume is pending and the bridge is enabled. Before, a bridge disabled
during StopUnproven kept those resources until re-enabled or rebooted.
pollLateStopAck() is public for this.
- The wrapper's destructor no longer stops an already-stopped client:
destroySlotClients() had just stopped it, so every shutdown logged five
spurious "esp_mqtt_client_stop failed: ESP_FAIL" errors.
Hardware (Heltec V4, 1 s test stop deadline): restart -> StopUnproven ->
`set bridge.enabled off`; the late ack was reaped ("releasing withheld
resources"), status read "not running", and `set bridge.enabled on`
started cleanly without a second release.
setBridgeState(false) returned early when the bridge was already down,
so `set bridge.enabled off` after a timed-out restart left the resume
pending and MQTT restarted against the operator's choice once the task
acknowledged. Disabling now clears the pending resume before that early
return, and the resume also requires bridge_enabled. The StopUnproven
log and `get mqtt.status` no longer say only "reboot to recover", since
a late ack now recovers on its own.
Hardware (Heltec V4, 5 live slots, 1 s test stop deadline): a restart
went StopUnproven, the task acknowledged 27 s later and the bridge
resumed with all slots reconnecting; with `set bridge.enabled off` in
between it stayed stopped until `set bridge.enabled on`.
The renewal bounce called reconnect()/connect() directly, so a successful
bounce never advanced applied_token_expires_at. The renewal decision
reads that value, so a renewed slot stayed due and, on a broker that
enforces exp, minted and reconnected every minute for the rest of the
token's life. Both the renewal bounce and the corrected-clock bounce now
go through reconnectSlotClient(), which starts a stopped client, refuses a
quarantined one, moves the slot to Starting and records the token in use.
The corrected-clock no-bounce case settles the renewal as the renewal
path already does.
- A stop that timed out into StopUnproven refused begin(), and only a
later begin() honoured the task's late acknowledgement. Nothing called
it, so restartBridge() (any `set mqtt...` restart) or an aborted OTA left
the observer offline for the rest of the boot. MyMesh now records that a
start was refused and restarts the bridge once stopAcknowledgedLate()
reports the ack. The OTA-abort alert no longer claims "bridge resumed"
when the restart was refused.
- refreshNTP() now runs the validated probe with one attempt per server.
With two attempts and a 1 s pause it blocked the MQTT task ~18 s every
hour on networks that drop UDP/123, where the old async SNTP cost
nothing. Loop comment and docs updated to match.
Stored 0 was the shipped default from 2026-01-02 to 2026-03-28, and every
association path has run it with power save off since. Reading it as
MIN_MODEM, as F11 did, would have put every node set up in that window
into modem sleep on its next association. 0 now reads as `none`, and
`set wifi.powersave min` stores a new value 3. /mqtt.json accepts 0..3;
older firmware repairs 3 to `none` on load. Binary snapshots are no longer
written, so 3 never reaches a legacy layout.
startLanMode() took the caller's IP, read before the route lock. A
Wi-Fi/Ethernet switch in between left WebConfig locked to the new link
while advertising the old link's address. It now reads the selected
link's address after lockSwitching(), as startOTAUpdate() already does.
The upstream `password` command saves /prefs.json without reporting the
result, and WebConfig overwrites its reply with "OK". If that write failed
while the later /mqtt.json write succeeded, wifi.setup_complete was set and
the factory password came back after reboot with first-run setup
suppressed. onInitialSetupComplete() now re-saves /prefs.json and records
completion only when it succeeds; otherwise the batch fails, setup stays
open, and no reboot is queued.
The old route is already down or switched away when a transition fires,
so waiting the default 5 s per slot for a DISCONNECTED event only delays
recovery (up to 25 s across five slots). A client whose event is late is
aborted by keepalive and retried by the normal backoff loop, the same as
the token-renewal path's softDisconnect() + reconnect().
- Link down/switch edges now softDisconnect() every started slot instead
of disconnect(). The full stop could wait forever for a DISCONNECTED
event on the Core 0 MQTT task, and it destroyed and recreated every
slot's esp-mqtt task on each Wi-Fi drop, where the bridge previously
stopped none (measured: 62 s deauth, five slots, zero stops).
softDisconnect() is bounded and keeps the task; reconnectSlotClient()
then calls reconnect() on the new route. Rename the transition action
to disconnect_started_slots to match.
- Ethernet no-IP recovery and init retries rebuild the CH390 netif, so
they now run inside the route-switch mutation gate. An OTA/WebConfig
lock taken after maintain() samples the lock count can no longer have
the interface torn down underneath it.
Fold the CH390 overlay and NETWORK_PREFER_ETHERNET into
ThinkNode_M7_{repeater,room_server}_observer_mqtt and drop the separate
*_observer_mqtt_ethernet twins. The env names (and so the OTA manifests)
are unchanged, so existing M7 observers OTA into the Ethernet-preferred
image; without a cable they select stored Wi-Fi after a brief boot probe.
- Guard the boot-time link bootstrap to observer ESP32 builds. It ran
unconditionally in MyMesh::begin(), breaking every non-observer
repeater/room server build (ESP32 and nRF52).
- Rename NetworkInterface -> NetworkLink (class, accessor, files).
Arduino-ESP32 3.x ships its own NetworkInterface class and header,
which broke the ESP32-C6 builds. Drop WiFi.setAutoConnect(), a no-op
on 2.x and removed in 3.x.
- Refresh stored Wi-Fi credentials every bridge tick so the STA
reconnect loop picks up `set wifi.ssid` / `set wifi.pwd` without a
reboot, as the bridge did before the link moved out of it. Skip
reconnects while the SSID is empty.
- Restore the "WiFi connected: <ip>" / "WiFi disconnected: reason N"
debug lines the bridge used to print.
- Alert on Ethernet only once it has held a lease this boot or when no
Wi-Fi is configured; Wi-Fi-only installs of an Ethernet-preferred
image keep Wi-Fi alerts instead of reporting "Ethernet down".
- Record wifi.setup_complete only for Ethernet LAN onboarding, so
Wi-Fi builds keep the SSID-based first-boot portal rule.
- Use seq_cst for the route-switch lock/mutation flag handshake.
- Docs: SNMP RSSI sentinel is -127; describe link-return vs medium-switch
reconnect behavior accurately; note runtime credential pickup.
- Test: unknown keys inside a known /mqtt.json group are ignored, which
keeps wifi.setup_complete downgrade-safe.
The upstream/dev merge kept our side of variants/thinknode_m7/platformio.ini,
losing -D ENABLE_WIFI_INTERFACE and +<helpers/wifi/*.cpp>. Companion WiFi is
now gated on that flag, so the env built silently without WiFi.
Today if a user builds firmware with an incorrect WiFi SSID or password,
they get a generic `WiFi disconnected.` error message.
Extend logging to descirbe the reason that WiFi was disconnected.
I gathered the updated logs for the 3 scenarios below.
```
WiFi: Attempting manual WiFi reconnect...
WiFi: WiFi disconnected (reason=ASSOC_LEAVE). Flagging for reconnect...
WiFi: WiFi disconnected (reason=4WAY_HANDSHAKE_TIMEOUT). Flagging for reconnect...
WiFi: WiFi disconnected (reason=4WAY_HANDSHAKE_TIMEOUT). Flagging for reconnect...
WiFi: WiFi disconnected (reason=4WAY_HANDSHAKE_TIMEOUT). Flagging for reconnect...
```
```
WiFi: WiFi disconnected (reason=NO_AP_FOUND). Flagging for reconnect...
```
```
WiFi: WiFi connected successfully!
WiFi: Got connection
```
agessaman