`client->connected()` answers "is the network up", and it was being used for
"is there anything to stop". A client resolving DNS, negotiating TLS or waiting
after a failed CONNECT reports not-connected, so `teardownSlot()` skipped it:
selecting `none` cleared the bridge flags while the SDK task kept running, and
when the in-flight handshake completed the callback marked the disabled slot
connected and scheduled its status.
Each slot now carries the client's SDK lifecycle state (Absent, Configured,
Starting, Connected, Disconnected, Stopped, Quarantined) and a generation
counter for the logs. The bridge task owns that state, which makes it the
authority the callbacks consult:
- a CONNECTED event for a slot that is disabled, never started, or already
stopped is logged and dropped instead of marking the slot connected;
- teardown stops any *live* client, so a mid-handshake client can no longer
outlive its configuration.
Teardown also gains a reason, because "stop the task" and "close the transport"
are different needs:
- `Disable` (preset `none`, shutdown) stops the client — the stop is the point;
- `Reconfigure` closes the transport with `softDisconnect()` and keeps the
esp-mqtt task. Stopping it there would return its 6 KiB stack into the hole
the two 16 KiB mbedTLS record buffers just vacated, which is the fork's
documented internal-heap fragmentation driver — and per the soak campaign's
own conclusion, feeding `softDisconnect()` into the reconfigure path is the
fix for it, not serialisation. The campaign closed 2026-08-20, so the
reconfigure churn is no longer anyone's measurement lever.
A stop that does not complete now quarantines that client: its SDK task was
never joined, so it is never reused, never destroyed, and the token buffer its
config still points at is never freed. `get mqttN.diag` reports `quarantined`.
Two knock-ons this required:
- `setupSlot()` starts or reconnects according to the SDK state.
`esp_mqtt_client_start()` fails on an already-started client, and now that
its result is honoured, a reconfigure that kept the task would otherwise
leave the slot permanently unactivated.
- the active-slot cap counts resource holders as well as configured ones. It
keyed off `initial_connect_done`, which teardown clears, so a started or
quarantined client stopped counting against the cap and a board could
oversubscribe past the concurrent-TLS limit the cap exists to enforce.
`connect()`, `reconnect()`, `disconnect()`, `softDisconnect()` and `forceStop()`
all returned void, so every caller in the bridge treated "asked" as "done":
- `setupSlot()` marked a slot activated after `connect()` whatever happened. A
failed `esp_mqtt_client_start()` therefore consumed one of the scarce
active-slot positions, handed the slot to a reconnect ladder that is gated on
activation, and was never retried by the deferred-setup path.
- `reconnect()` explicitly proceeded after `esp_mqtt_set_config()` failed,
reconnecting on the previous configuration — the renewed token in the buffer,
the old one on the wire.
- the renewal path recorded the new expiry before the bounce succeeded, so a
failed bounce left the live session on the old credential with the next
renewal not due for a whole token lifetime.
- `softDisconnect()` logged its timeout and told its caller nothing.
- `disconnect()` waited for the DISCONNECTED event with no bound, on the very
task whose stop acknowledgement the shutdown waits for.
Now every one of them returns `esp_err_t`, a failed configuration transaction
aborts rather than starting or reconnecting on a half-updated config, and
`disconnect()`'s wait is bounded (it still stops the client, and reports
ESP_ERR_TIMEOUT when the event never arrived).
Bridge consequences:
- a failed start leaves the slot unactivated, so the existing deferred-setup
retry revisits it and it holds no active-slot position;
- a failed renewal bounce re-arms the renewal instead of recording it, so the
next maintenance pass retries;
- a reconnect that fails *locally* rolls back the backoff advance made for it.
The ladder and the breaker bound broker and network faults; an uninitialised
client or an uncommitted config transaction is neither, and inflating the
ladder for it was how a local fault could trip a breaker meant for a broker.
Non-PSRAM boards allocated 512-byte MQTT buffers for non-JWT slots, so the
desired buffer size was a per-slot variable while the allocated size was fixed
at `esp_mqtt_client_init()`: IDF 4.4 does not resize those buffers in
`esp_mqtt_set_config()`, and the wrapper's own reassembly buffer is allocated
once for the client's lifetime. A slot reconfigured from non-JWT to JWT
therefore kept 512-byte buffers, and a valid JWT CONNECT (frame plus a 768-byte
token) could not fit — with no error naming the cause. Stopping and starting the
same handle could not change the capacity either.
Give every client 896 bytes on every board. The cost is 384 bytes per client on
a non-PSRAM board, which caps at 2 active slots, against a TLS handshake that
needs 16 KB of contiguous internal DRAM. In exchange the capacity transition
stops existing, so the reconfigure path never needs to recreate a client to grow
its buffers.
The decision half of the client-configuration fix, as a pure header so every
transition is a host test rather than a hardware run.
`MqttEffectiveConfig` is the complete owned description of what a slot's client
should be configured with — URI (its own copy, because `slot.broker_uri` is
rewritten in place), transport, trust policy, auth mode, credential pointers,
buffer size, keepalive. Two things are deliberate:
- the transport is derived from the URI scheme, and the trust policy is
normalised against it: an unencrypted transport verifies nothing, whatever
certificate material was requested, so a plaintext endpoint can never look
like it retained a verified policy;
- absent credentials are represented as nullptr and written as "", never
omitted. IDF's `esp_mqtt_set_if_config()` treats NULL as "leave unchanged",
so a cleared wrapper pointer cannot erase an SDK-held credential, while an
empty string overwrites it and leaves the CONNECT's username flag clear
(verified on an ESP32-S3 against a local broker on IDF 4.4).
`mqttConfigRecreateDecision()` says whether the existing client can be
reconfigured in place. Reuse covers every credential and auth-mode change and
any endpoint move within one scheme — the cases that matter for reconnect and
token renewal, where a client create/destroy cycle is the fork's documented
internal-heap fragmentation driver. Recreation is reserved for the three fields
that cannot be overwritten safely: a transport (scheme) change, a trust-policy
or CA-certificate change, and growth beyond the allocated buffer capacity,
which IDF 4.4 fixes at client init.
Client certificates are excluded on purpose: `setClientCertificate()` has no
caller in the firmware, so modelling mutual TLS here would be untested
configuration surface. Noted in the header.
Nothing consumes this yet; the bridge is migrated onto it after the result
propagation and generation-guard steps, which it depends on.
The timeout fallback did the two things that cannot be done safely to a task
that may still be inside mbedTLS or holding esp-mqtt's API mutex: it deleted
that task from the loop task, then force-stopped and deleted the clients it
owned, then freed the queue and buffers it can reach. `esp_mqtt_client_stop()`
waits on the API mutex and the task's stopped event with no bound, so killing
its caller strands the next one; a second stop sees the run flag already clear
and returns failure without joining. Nothing in that path established
quiescence, and `canFlashAfterStop()` could only withhold OTA afterwards.
Changes, all of them about who owns what:
- `MQTTLifecycle` gains a `StopUnproven` state (appended, so existing values
keep their numbering). `StopTimedOut` now leads there and releases NOTHING;
it still fires `ota_release` so the OTA barrier aborts. `acceptsNewWork`,
`mayRestart` and `isStopInProgress` are false there, `mayTouchOwnedState` is
true, and `end()`'s wait loop still terminates.
- `begin()` refuses to start while a stop is unproven. A new start was never
proof that the previous clients stopped, and it used to clear the dirty latch
that withheld OTA. Because a start is now impossible from `StopUnproven`,
clearing that latch on an accepted start is sound.
- A late acknowledgement is honoured. The task publishes it only after tearing
its clients down, so it proves the same thing whenever it arrives; the
deadline is an availability policy, not a statement about what the ack means.
`pollLateStopAck()` releases the withheld resources and makes the bridge
restartable — a slow stop (a blackholed WSS broker can hold the SDK well past
the budget) no longer costs a reboot.
- The handshake flags become `std::atomic<bool>` with release/acquire, and the
ack moves into the task trampoline immediately before `vTaskDelete(nullptr)`,
gated on a `_teardown_complete` flag the loop sets. Published from inside the
loop it proved teardown returned, not that the task had stopped executing —
and `volatile` ordered none of it. This is the soak campaign's blocker #20,
the one item on its list that can corrupt memory.
- The force-stop paths are gone with their only caller: `destroySlotClients()`
and `teardownSlot()` no longer take a `force` flag.
- `get mqtt.status` reports "previous stop unproven, reboot to recover" instead
of a bare "not running", latched so it survives end() clearing the singleton.
Two lifecycle tests asserted the old contract and are re-encoded, not deleted:
a timed-out stop is no longer `Stopped`-and-released, and "repeated failed
stops leave the bridge restartable" becomes "repeated *slow* stops recover on
their late ack" plus a new case pinning that a never-acknowledged stop stays
unusable for the boot. That availability trade is the point of the change.
The applied mode had no observable trace on a running node, which is how the
CLI and the reconnect path could disagree about what `min` means for as long as
they did. Log it at the one place that applies it, reading the value back with
esp_wifi_get_ps() rather than printing what was requested.
Used to confirm F11 on hardware: with `min` stored, a Heltec V4 now reports
"WiFi power save: min (mode=1)" on association, where the old mapping applied
WIFI_PS_NONE (mode=0).
NTPClient::forceUpdate() treated any non-empty datagram arriving on its fixed
local port 1337 as a time response: it ignored the read length, the version,
the mode, the stratum, the leap indicator and any request/response correlation,
then handed the bytes at offset 40 to the bridge. The bridge only checked that
the derived epoch was at least 2026-01-01 before calling settimeofday() and
writing the RTC. A host probe against the installed library source accepted a
one-byte non-NTP datagram and produced epoch 2085978496. Anyone able to land a
UDP datagram during a query window could set a bogus clock, which then feeds JWT
issuance, certificate validity and packet timestamps.
Replace that path with probeNtpServer(): one exchange per server on a fresh
ephemeral socket, closed again on every exit. A reply is accepted only if it is
a full 48 bytes, from the address and port queried, NTPv3/v4 mode 4, from a
synchronised server (no leap alarm, stratum 1-15), echoing the random transmit
timestamp of the request, with an epoch inside a plausible range. Rejected
datagrams leave the clock alone and do not end the wait, so an early bogus
packet cannot pre-empt the real answer. Era-1 (post-2036) timestamps convert
forward instead of wrapping into 1900.
The diagnostic now runs the same validated probe, so `get mqtt.ntp.diag`
answers the question it is asked — would this server be trusted? — instead of
reporting a datagram nothing checked. It also no longer leaves its UDP socket
open after the probe, and each server's per-probe result carries the reason it
failed ("DNS failed", "unsolicited reply", "server unsynced", ...). The
diagnostic still blocks its caller; making it asynchronous is a separate change.
The acceptance rules live in NtpValidation.h (pure, host-tested), including the
review's one-byte-datagram reproduction.
Only the periodic status path consulted the setting. The connect callback armed
a status publish unconditionally and publishStatusToSlot() never checked it, so
every boot and every reconnect published a status message — including the
metadata an operator turned the setting off to suppress. The documented contract
is "Enable/disable status messages".
Read the toggle live from prefs at publish time, matching the periodic path, and
also skip a slot the operator has disabled: teardown only stops a client that
reports connected, so a slot switched off mid-connect can still complete its
handshake and arm this publish.
Four of the hand-written error labels named the wrong failure against the
installed SDK: Wi-Fi reason 201 is NO_AP_FOUND (reported as "security
mismatch"), 202 is AUTH_FAIL (reported as "auth mode rejected"), 39 is TIMEOUT
(reported as "SSID not found") and 34 is MISSING_ACKS (reported as an
"AP state mismatch"). esp-tls 0x8008 is TCP_CLOSED_FIN, not a timeout, and
0x8010 is CERT_PARTLY_OK, not a generic mbedTLS error. 0x800B, labelled
"cert verify failed", is not an esp-tls error at all. So the one line an
operator reads to find out why a node is offline could name the wrong cause.
Move the tables into MQTTErrorLabels.h (pure, host-tested) with every value
static_asserted against the SDK enum/#define actually being compiled, so a
framework bump fails the build instead of relabelling errors in the field.
Codes the SDK does not define — including 61/88/168, which had labels no
header supports — now return no label and the caller prints the raw number.
Also keep the CONNACK return code from a broker refusal and show it in
`get mqttN.diag` ("refused: bad user/password (4)"). A refusal is not a
transport failure, so the TLS/socket fields are empty and a slot with wrong
credentials previously showed no useful detail at all.
`set wifi.powersave min` stored 0 and applied WIFI_PS_MIN_MODEM, but the
bridge's post-reconnect mapping read the same 0 as WIFI_PS_NONE. A node
explicitly configured for minimum modem sleep therefore ran with power save off
after its first reconnect, while `get wifi.powersave` still reported min. The
stored default is already 1 (`none`), so nothing here changes the product
default — it stops an operator's explicit choice from being reinterpreted.
Move the value/name/mode table into WifiPowerSavePolicy (pure, host-tested) and
use it from the CLI setter and getter, the web config snapshot and the bridge.
The bridge now also applies the mode when it finds the STA already associated at
start — that path has no connect transition to carry the setting, so a bridge
restart used to leave whatever mode was set before.
The watchdog took max(last_recv, last_irq, last_tx) of three raw millis()
timestamps. Around the 32-bit wrap a transmit stamped in the upper half of the
range is numerically larger than every fresh post-wrap receive, so it kept
winning for up to ~24 days: the watchdog measured silence from that stale
transmit, concluded the radio had gone deaf, and reset a radio that was
receiving normally — once per watchdog interval, until the old value stopped
winning.
Extract the decision into RadioWatchdog (pure, host-testable). It compares each
timestamp only against its own previous value, so any change is fresh activity,
and stamps it against a monotonic 64-bit clock extended from millis(). Evidence
older than one 32-bit cycle now stays old instead of aliasing back to "recent",
which the previous unsigned-subtraction age could not do either.
Covered in test/test_radio_watchdog: pre-wrap TX with continuous post-wrap RX,
genuine silence across the wrap, silence past a full millis() cycle, no activity
since boot, disabled watchdog, TX-only traffic, and a stalled mesh loop.
The motherboard has two RF daughterboard slots on separate GPIOs, mirrored by
its "LNA P" / "LNA S" jumpers, but the variant only ever pinned slot 1.
The shared configuration moves into [Station_G3_ESP32_common], with the SPI and
LNA lines split into per-slot sections. [Station_G3_ESP32] (slot 1) and
[Station_G3_ESP32_r2] (slot 2) each compose common + their own pins, so neither
set of -D flags has to shadow the other on the compile line - appending an
override would work only by last-wins, and esp32_base's -w hides the resulting
redefinition warning. Every pre-existing env keeps its name and resolves to
byte-identical options; verified by diffing pio project config --json-output.
PA PL1 is one board-level jumper shared by both slots, so P_PA1_EN stays in the
common section along with the TX power calibration.
Adds r2 twins of the repeater, room server and both observer envs, each an exact
mirror of its slot-1 sibling apart from the base it extends.
Slot 2 pins come from on-hardware testing in agessaman/MeshCore#49 and are not
documented publicly; the vendor wiki serves no content to fetchers and Meshtastic
implements slot 1 only.
Known limitation, unchanged from that PR: GPIO 42 and 43 fall outside the
ESP32-S3 RTC GPIO range (0-21), so the rtc_gpio_hold_en() calls in
ESP32Board::enterDeepSleep() and StationG3Board::powerOff() return
ESP_ERR_INVALID_ARG on slot 2 and the NSS and LNA pins are not latched through
deep sleep. Fixing it means gpio_hold_en() plus gpio_deep_sleep_hold_en() in
shared code.
Adds both settings to docs/cli_commands.md in the existing format, and records
the R8 dashboard, display controls and the two hardware fixes in the changelog.
The display.flip entry calls out that it is persisted config which survives a
firmware update, since a node still carrying flip=1 from testing looks exactly
like a firmware whose orientation was never fixed.
display.flip lives in /mqtt.json, so it survives a reflash and is invisible
while someone is chasing a wrong orientation - a node still carrying flip=1
from testing looks exactly like a firmware that was never fixed. Boot now
reports "Display: flip off" or "flip on (rotated 180)".
Rotation 2 was consistently upside down on the Expansion Kit V2 panel, so 0
becomes the compiled default for both portrait observer targets.
display.flip is unchanged and still defaults to off: the compiled constant
should be the correct orientation, with the setting reserved for a board
mounted the other way up. A node already carrying `display.flip 1` from
testing needs `set display.flip 0` after this.
Landscape targets are untouched - they take the driver's own DISPLAY_ROTATION
default of 3.
Power-off now needs a 3 second hold; any shorter press toggles the display.
MomentaryButton reports a CLICK for any release short of its threshold, so
that single value defines both.
The button also felt unreliable - "a brief press doesn't wake it, more often
than not". MomentaryButton's multi-click detection withholds a CLICK for
MULTI_CLICK_WINDOW_MS (280 ms) after release, and folds a second press
arriving inside that window into a DOUBLE_CLICK. Since the handler only acts
on CLICK, an impatient second press produced nothing at all: press, see
nothing, press again, still nothing. Multi-click is now off for these targets,
so CLICK fires on release.
Both settings are build flags defaulted in variants/heltec_v4_r8/target.cpp
and overridden only on the two TFT observer bases, because the companion
builds share this user_btn and do use double/triple click.
DISPLAY_TOUCH_DEBUG additionally logs which input caused a toggle
("Display: button -> on"), so any remaining flake can be attributed to the
button or to a spurious touch read rather than guessed at.
Addresses a Codex review of the preceding three commits.
Touch could disable itself for the whole session. begin() latched _present
from a single address probe, but this controller NACKs whenever it has nothing
to report, so an idle probe at boot was indistinguishable from absent
hardware. The probe is now diagnostics only; checkTap() polls regardless, and
a NACK costs one quiet bus cycle.
Touch no longer mutates shared bus state. Wrapping the read in
setTimeOut()/restore was an unsynchronised global write, and the MQTT task
drives the same Wire through AutoDiscoverRTCClock in its NTP fallback, so it
could inherit the short timeout. The address probe alone removes the
ESP_ERR_TIMEOUT stalls that motivated it, and it also ran before the timeout
was installed, so the transaction most exposed to a wedged bus was unprotected
anyway.
A single failed read could fake a release. The debounce window was 40 ms
against a 50 ms poll, so a state change was confirmed by the very next sample
and one NACK mid-touch produced a release followed by a second tap. It is now
80 ms - two consecutive consistent reads.
RadioActivityWindow froze after a gap longer than ~24.8 days. tick() treated
any delta past the signed halfway mark as an out-of-order timestamp, so a node
left unserviced that long kept a month-old packet in the 20-minute window and
reported it as recently received. A backwards step is now only believed when
it is small, which is what an out-of-order reading between two call sites
actually looks like.
Not changed: a downgrade that then saves prefs drops the display group, since
/mqtt.json is rewritten from the older serializer's known schema. That is
inherent to every appended field in this format, and bumping the JSON version
would be worse - older firmware would reject the file rather than ignore one
group. Documented instead.
`set display.flip 0|1` (also off/on) turns the panel 180 degrees from its
compiled DISPLAY_ROTATION, persisted in MQTTPrefs alongside display.timeout
and applied live without a reboot.
Adding 2 to the compiled rotation rather than setting an absolute value keeps
portrait portrait and landscape landscape, so the DisplayViewport geometry
never changes with it and the setting cannot produce a nonsensical mix.
DisplayDriver gains a defaulted no-op setFlipped(), so no other display
driver is affected.
The compiled rotation was verified identical across `pio run` and `build.sh`
on two machines (movi a11, 2 at the setRotation call site), yet the panel read
upside down for one tester and upright for another - which is what a board
mounted either way up looks like. No single compiled constant satisfies both,
so orientation becomes a setting rather than another rebuild.
Runtime-only, like display_timeout_secs: LegacyV1MQTTPrefs and the frozen
binary payload sizes are untouched, and the JSON group is an append that older
firmware skips.
Four defects found by hardware testing of the Expansion Kit V2.
Touch never registered. The panel's controller does not use the point-count
encoding the reference CHSC6X drivers document: byte 0 reads 0x00 idle and
0x1F while a finger is down, so testing for a count of 1 never fired. A
partly-failed read leaves 0xFF, which must not count as a press either, so
the test is now != 0x00 && != 0xFF.
Touch polling could stall the UI loop for ~1 s at a time. The controller
NACKs its address whenever it has nothing to report, and calling requestFrom()
unconditionally logged a bus error on every 50 ms poll and, once the bus
wedged, burned a full ESP_ERR_TIMEOUT inside loop(). Probe the address first,
which reports the same NACK quietly, and bound the read with setTimeOut().
The display could not be woken once it blanked; only RST brought it back.
turnOn() re-ran the whole display.init(), which re-enters SPI setup, spends
~500 ms in Adafruit's reset delays and pulses GPIO 21 - the line shared with
TP_RST, so it reset the touch controller on every wake. Since turnOff() no
longer parks that line low, the panel stays configured while dark and waking
is just the backlight. Toggling also clears the refresh deadline so the
current frame is drawn immediately instead of the stale one.
Power-off rebooted instead of staying off. powerOff() went through
enterDeepSleep(), which always arms an ext1 wake on P_LORA_DIO_1; a deep-sleep
wake is a full reboot, so a node in live traffic restarted within seconds of
showing "Turning OFF". It now disables every wake source, so the node stays
down until RST or a power cycle.
Note that the display off/on cycle had never been exercised on this board
before: observer builds pinned AUTO_OFF_MILLIS=0, so the panel never blanked
until display.timeout made it a runtime setting.
Replace the sparse Heltec V4 R8 observer home screen with a padded dark
analytics dashboard, add manual display control, and make blanking a runtime
setting.
Dashboard (DISPLAY_ACTIVITY_DASHBOARD, the four R8 TFT observer envs):
- RadioActivityWindow: 20 one-minute buckets of valid RX packets, no heap.
The caller's 32-bit millis() is extended to a monotonic 64-bit clock, so
nothing downstream has a rollover case; an always-on node passes 2^32 ms
after ~49.7 days, which would otherwise re-enter warm-up and divide 20
minutes of traffic by seconds. Rates use 19 whole minutes plus the elapsed
part of the current one rather than a fixed 1200 s.
- ObserverDashboard: header, radio strip, headline totals, a 20-bar
packets-per-minute graph and RF/status footers, with separate portrait and
landscape layouts. A text row is a fixed 16 px, which is 3.2 logical units
in portrait but 4.27 in landscape, so one shared grid would overlap.
Text is trimmed by character budget, not measured width: getTextWidth()
reports an over-long string at the portrait driver's fallback scale, so
DisplayDriver::drawTextEllipsized() under-trims and the row renders at half
height.
- Six per-row signatures computed from what is actually drawn, so only the
rows whose pixels changed repaint. No startFrame(), no whole-screen clear.
Link state moved out of the full-frame signature, so a DHCP renewal or WiFi
flap repaints one footer row instead of the panel.
- Dark theme by retuning the UIColor statics at runtime, which needs no
display-driver edit and carries boot, setup, reboot and power-off with it.
Touch and button (DISPLAY_TOUCH_TOGGLE):
- CHSC6X at I2C 0x2E, polled; TP_INT is unusable (optional R13, and GPIO 43
is U0TXD). The point-count byte is tested against a valid count, never
against non-zero: an idle read returns 0xFF, which reads as a finger held
down forever and latches the tap detector after one event.
- turnOff() no longer parks PIN_TFT_RST low on this board. GPIO 21 is a
shared LCD_RST/TP_RST net, so doing that held the touch controller in
reset for as long as the display was off. Verified against Heltec's
expansion-board and mainboard schematics and the V4-R8 datasheet pinout,
which also correct the pin comment in HeltecV4R8Board.cpp.
- The USER button click now toggles the display too; it previously did
nothing whenever the display was already on.
display.timeout:
- `set display.timeout <secs>` / `get display.timeout`, 0 = stay on, 60 s
default, 3600 max. Read live, so a change applies without a reboot and
restarts the countdown rather than firing on the old deadline.
- Stored in MQTTPrefs (/mqtt.json), keeping NodePrefs aligned with upstream.
Runtime-only: LegacyV1MQTTPrefs and the four frozen binary payload sizes
are unchanged. No JSON format-version bump - the loader skips keys no
def() claims, so older firmware reads newer files and this firmware reads
older ones with the default applied. Both directions are covered by tests.
- Joins the observer atomic-setter contract, so a failed save rolls the live
value back instead of only claiming to.
New periodic work uses a wrap-safe deadline check; `millis() >= deadline`
fires every loop for a whole interval before each rollover.
Adds test_radio_activity_window, test_observer_dashboard (driving the real
renderer against a recording DisplayDriver in both orientation profiles) and
test_touch_tap_detector. 440 native cases pass.
agessaman