`set wifi.powersave min` stored 0 and applied WIFI_PS_MIN_MODEM, but the
bridge's post-reconnect mapping read the same 0 as WIFI_PS_NONE. A node
explicitly configured for minimum modem sleep therefore ran with power save off
after its first reconnect, while `get wifi.powersave` still reported min. The
stored default is already 1 (`none`), so nothing here changes the product
default — it stops an operator's explicit choice from being reinterpreted.
Move the value/name/mode table into WifiPowerSavePolicy (pure, host-tested) and
use it from the CLI setter and getter, the web config snapshot and the bridge.
The bridge now also applies the mode when it finds the STA already associated at
start — that path has no connect transition to carry the setting, so a bridge
restart used to leave whatever mode was set before.
The stop timeout bounded only the cooperative attempt. Its fallback tears down
clients from the calling task, and both helpers skipped the stop entirely when
the client reported not-connected — then deleted it anyway. That is the exact
shape of the dangerous case: a client whose DISCONNECTED callback has already
cleared its connected flag while it sits inside esp_mqtt_client_stop() reports
not-connected, so the stop was skipped precisely when it mattered and the
object was freed from under a live ESP-MQTT task.
Both helpers take a force flag, and only the dirty branch passes it. force is
deliberately not gated on connected(), and routes through the existing
forceStop() rather than disconnect(), whose wait for the DISCONNECTED event is
unbounded. Every other caller — the non-ESP32 release path, the Core-0
cooperative teardown, reconfigure, and the slot-cap path — keeps its current
behaviour byte for byte.
This does not make the path bounded. esp_mqtt_client_stop() waits on the
client's STOPPED_BIT with portMAX_DELAY on the pinned framework, so a client
wedged inside mbedTLS can still block the caller. The change trades a
free-under-a-live-task for a wait, which is the safer of the two failure modes.
Bounding it properly, and the surrounding destroy-while-still-stopping hazard,
need the ownership handoff reworked; that is tracked separately.
The preset-application loop in begin() only had a positive branch, so when
preferences named no preset the slot kept whatever the previous begin() left in
RAM. Nothing else resets it: teardownSlot() deliberately preserves enabled and
preset so a reconfigure can reuse the mbedTLS context, and the constructor
clears them exactly once. A bridge restart could therefore resurrect a broker
the operator had disabled and reconnect to it with the old credentials.
Config fields only. The client belongs to destroySlotClients() and the token
buffer to releaseSlotAuthToken(); clearing either here would strand a pointer
esp-mqtt still holds in its config.
Note the positive branch re-enabling a slot is correct and unchanged: a slot
capped off at startup is disabled in RAM only, with preferences still naming a
real preset, and the cap decision has to be re-made on each start.
Reachable via WebConfig, where a pending full restart discards the per-slot
restart mask, so a batch that disables a slot can end in a restart that never
applied the disable. A plain CLI set cannot reach it: restartBridgeSlot()
applies the change to the live bridge immediately, keeping RAM and preferences
in agreement. Verified on hardware as a non-regression check for that reason —
disable, restart, slot stays down; restore, restart, all slots return.
ESP-IDF stores the mbedTLS stack error as a positive magnitude (it captures
-ret), so negating it before printing produced "mbedtls:-0xFFFF8100" instead
of "mbedtls:-0x7F00" for the record-buffer allocation failure. %04X is a
minimum width, so nothing masked it.
Normalisation moves into MQTTReplyFormat.h as mbedtlsErrorMagnitude() rather
than staying inline in the bridge: inline is why this survived, since the
existing test passes the already-correct magnitude straight into replyAppendf
and never exercised the caller. It accepts either sign so a later SDK storing
the real negative code still renders, and widens to int64_t before negating
because negating INT32_MIN is undefined behaviour.
MQTTReplyFormat.h also gains the stdint.h it was always missing: it compiled
only because MQTTBridge.cpp pulls stdint in via other headers, and the host
test includes the header standalone.
Found on hardware while validating the SNTP fix. `set mqtt.ntp bogus.invalid`
reported SUCCESS with a correct epoch, in 4 s, with no retry and without ever
reaching the SNTP fallback:
[E] hostByName(): DNS Failed for bogus.invalid
[E] beginPacket(): could not get host from dns: 11
MQTT: Time synced: 1786764354 (via bogus.invalid)
Three pieces compose it. WiFiUDP::beginPacket(const char*, port) returns 0 on a
DNS failure and leaves remote_ip/remote_port at their previous values.
NTPClient::sendNTPPacket() discards that return and calls endPacket() regardless.
endPacket() sends to whatever remote_ip still holds. So the request went to the
pool address resolved at boot, that server answered with a genuine timestamp, and
the loop recorded ntp_server_used as the name that had never been contacted.
This sits one layer above the fallback that b1ceaf01 made honest — control never
reaches it — so `set mqtt.ntp <typo>`, whose whole purpose is to fail fast, still
reported OK and the fleet kept a server name it had never spoken to.
The DNS pre-check was already here and only logged a warning. Make it decide:
skip a name that does not resolve rather than attempt a send that cannot go where
it claims. IP literals are unaffected — hostByName() returns them via
fromString() without a lookup — and the lookup already ran, so no latency is
added. Moved setPoolServerName() below it so the client is never pointed at a
server being skipped.
Residual, narrower window: our lookup succeeds and NTPClient's own
gethostbyname() then fails, which needs the entry to leave the lwIP cache between
two calls microseconds apart. Closing it properly needs the resolved IP handed to
NTPClient, and this version exposes no setPoolServerIP(); the constructor is the
only way in.
Not host-testable — NTPClient and WiFiUDP both. Verified by inspection of both
library sources plus the captured hardware trace above.
The usable-clock fallback asked libc only, which does not answer for the case it
was written to cover. On a cold boot ESP32RTCClock::begin() stamps libc with a
2024 placeholder on power-on; AutoDiscoverRTCClock::begin() probes the chip but
never copies its time across, and getCurrentTime() reads the chip directly. So a
Station G3 or T-Beam Supreme that knows exactly what time it is, on a network
with UDP/123 blocked, still failed the plausibility test, left _ntp_synced false,
and brought up no slots — precisely the deployment the fallback exists for.
Ask the RTC when libc is below the floor. libc still wins when it is usable: a
clock SNTP set recently outranks a chip that may have drifted. Accepting the RTC
value then flows through the same block, so settimeofday() repairs libc and the
epoch is written back to the chip.
The choice is chooseFallbackClock() in MQTTConnectionPolicy, host-tested across
the four states including the power-on placeholder and the exact floor. Also
corrects the previous commit's claim that configTime() is called only when a
server replied — the fallback necessarily points it at each server before knowing
that; it is the post-acceptance call that is now conditional.
The reset was on the wrong side of configTime(). configTime() configures the
server, calls sntp_init(), and returns — the new request is live before it comes
back — so a fast reply could set SNTP_SYNC_STATUS_COMPLETED inside that call, and
the reset immediately after would erase it. The following ten seconds of polling
would then see nothing and reject a server that had in fact answered. On the
`set mqtt.ntp` path that surfaces as a good server failing validation.
Stop any running session first, discard its status, then start the new one, so
the only completion observable is the one being waited for.
Requiring a real SNTP completion took away something the plausible-clock test was
doing by accident. _ntp_synced gates slot setup outright (:1386, :2894), so a
device that cannot reach NTP now brings up no slots at all — and a network that
blocks UDP/123 while allowing 443 is an ordinary firewall configuration, not a
corner case. An RTC-backed observer there used to stay synced and keep minting
JWTs against a perfectly good clock.
Accept the existing clock explicitly when every server has failed, logged as what
it is rather than as a claim about a server that never replied. Excluded from the
`set mqtt.ntp` validation path, where the question is whether that server works
and the clock cannot answer it. configTime() is now called only when a server did
answer, since otherwise there is nothing new to point SNTP at.
The corrected-clock path reconnected a disconnected slot whether or not
createSlotAuthToken() had produced anything, which re-presented the credentials
the correction had just invalidated. Minting fails for recoverable reasons —
allocation pressure is treated as recoverable elsewhere in this file — so the
path is reachable, and the reconnect it spends is one that cannot succeed.
Move the decision into MQTTConnectionPolicy as classifyStaleToken(), where the
four outcomes are named and host-tested rather than spelled out in nested
conditions: Defer on a failed mint, Reconnect a client that is down, Bounce a
live session whose broker enforces exp, KeepAlive one whose broker does not.
Deferring leaves the slot to the backoff ladder, which mints again on its next
attempt.
Covers the reviewer's first four cases. The other two — that a completed SNTP
sync is required, and that time(nullptr) reflects the accepted epoch before
_ntp_synced flips — are inside MQTTBridge.cpp, which the native env does not
compile; locking those down needs a seam around the IDF calls that does not
exist yet.
syncTimeWithNTP() read an epoch over UDP, called configTime(), set _ntp_synced,
and then had the stale-token test and createSlotAuthToken() read time(nullptr) —
without anything having put the accepted epoch there. configTime() restarts SNTP
and returns; the clock lands whenever a packet does.
_rtc->setCurrentTime() looks like it covers this and does not.
AutoDiscoverRTCClock::setCurrentTime() writes a detected DS3231/RV3028/PCF8563/
RX8130CE *instead of* delegating to its fallback, and only that fallback
(ESP32RTCClock) calls settimeofday(). So on every board carrying an RTC chip —
T-Beam Supreme and Station G3 both compile this bridge and both instantiate
AutoDiscoverRTCClock — libc kept the pre-correction time, and the correction path
tested staleness and minted iat claims against exactly the clock it had just
proven wrong. Boards without a chip take the fallback and were unaffected, which
is why the soak rig (Heltec V3/V4, no RTC) never showed it.
settimeofday() with the accepted epoch first, so the invariant downstream code
already assumes actually holds: once _ntp_synced is true, time(nullptr) returns
the epoch we accepted. configTime() still follows, to keep future syncs running.
The fallback configured a server, waited 500 ms, and accepted any plausible
system clock as proof that server had answered. It usually has not answered
that fast — and the device usually already holds valid time, from an earlier
sync or the RTC — so the first server in the list was credited unconditionally,
the walk stopped there, _last_ntp_sync was refreshed, and an unreachable host
was logged as the source. On the `set mqtt.ntp` validation path, where the
single-server walk exists so a typo fails fast, that reported a bad server as OK.
Poll sntp_get_sync_status() for SNTP_SYNC_STATUS_COMPLETED instead, which is the
layer's own statement that a packet arrived. The status is one-shot — reading
COMPLETED clears it — so a result left by an earlier sync would latch on the
first poll; clear it before the loop.
An implausible epoch after a completed sync now moves to the next server rather
than spinning out the remaining attempts against a server that has answered.
esp_mqtt_client_reconnect() is honoured only from MQTT_STATE_WAIT_RECONNECT, so
the post-correction path minted a fresh token, staged it, and then asked a
connected client to reconnect — a request esp-mqtt refuses. The slot kept running
on the token the clock correction had just proven stale, and recovery became
broker-driven rather than the clean reconnect this code intends.
Split the three states the path can find: a stopped or waiting client goes through
reconnectSlotClient() as before, and a live one has its transport closed first.
Only where the broker enforces exp, though — waev leaves live sessions alone past
expiry, so bouncing it would spend the 16 KiB contiguous handshake that the rest of
this branch exists to avoid.
Not a regression: the base branch called client->reconnect() directly at the same
site. On ESP32 the block is reachable from the WiFi-reconnect resync and the CLI
forced sync; the hourly refresh uses refreshNTP(), which does not carry it.
reconnectSlotClient() checks isStarted() and calls connect() instead of
reconnect() when the client was stopped, but only the post-NTP stale-token
path used it. The ordinary backoff ladder and the circuit-breaker probe called
slot.client->reconnect() directly, and esp_mqtt_client_reconnect() is a no-op
on a client that is not started.
Two ways in. connect() sets _started only when esp_mqtt_client_start() returns
ESP_OK while setupSlot() sets initial_connect_done unconditionally, so a start
failure under heap pressure stranded the slot. More routinely, the WiFi-drop
handler calls disconnect() on every connected slot, which clears _started —
after that the ladder issued no-ops forever and the slot never came back.
Not caught by the soaks: the log line the guard prints can only come from the
NTP path, so a stranded slot and a slot that never entered the state produce
identical logs. Observed reconnects were broker-side drops with WiFi up, which
leave the client started.
The renewal-bounce path keeps its own isStarted() branch — it needs
softDisconnect(), which the helper does not do.
Every ordinary backoff reconnect and every circuit-breaker probe minted a
fresh JWT and re-applied credentials, with no check of whether the existing
token was still valid. setCredentials() always dirties the esp-mqtt config, so
reconnect() then called esp_mqtt_set_config() as well. On a flapping broker
that is a signing plus a configuration-copy cycle on every retry, and these
observers see ~38 genuine reconnects/day per slot.
The no-bounce renewal change (27bd05a1) only stopped the proactive renewal
from tearing down a live session; it left this retry path untouched, which is
why a soak shows renewals neither firing nor failing for hours while drops
continue — each reconnect silently re-mints and pushes the expiry out.
Reuse the credentials when their validity is provable and refresh them
otherwise. canReuseJwtForReconnect() lives with the other policy predicates so
it is host-testable, and it establishes current_time < token_expires_at before
subtracting: token_expires_at is unsigned, so an already-expired token would
otherwise wrap to ~4e9 seconds and read as valid for decades. The
>= kMinimumValidEpoch term also rejects the 0 that a failed renewal writes.
Minting stays the default for every uncertain case — unsynced clock, missing or
insane expiry, empty token, or an expiry inside kJwtReconnectSafetyMarginSecs
(60 s), which covers the handshake itself.
Two paths still always mint, deliberately:
- The circuit-breaker probe. It is the recovery of last resort for a slot
that has already failed repeatedly, quite possibly on auth, and it runs
once per 30 minutes — so a fresh token there costs nothing worth counting
against keeping that path guaranteed-clean.
- Any slot whose last error was a broker refusal. Before this change, minting
on every retry accidentally recovered from server-side credential
invalidation: key rotation, revocation, broker clock skew, or an audience
change after a reconfigure. Reuse would have retried a rejected credential
until it neared expiry — up to 24 h for every preset that leaves
token_lifetime at the default. onError already detects
MQTT_ERROR_TYPE_CONNECTION_REFUSED and only logged it; it now also sets a
per-slot force-mint flag, cleared on a successful connect and wherever the
credentials it referred to are blanked. The flag is volatile because the
esp-mqtt callback sets it and the bridge loop consumes it.
The reconnect log line reports the decision and its outcome — REUSE, MINT with
a reason, and OK/FAILED for the mint — because a silently failed mint is the
case most likely to end in an auth refusal. It never prints the token.
Host tests cover the reuse boundary: exact margin, already-expired, expiry 0,
sub-epoch expiry, empty token, unsynced clock, and the force-mint override.
waev's operator confirmed on 2026-08-11 that their servers do not disconnect a
client when its JWT passes exp — a 60-minute token can hold a session open for
hours. The renewal path assumed the opposite, in as many words: the comment at
the bounce called the renewal buffer "the ONLY margin between 'device
re-authenticates' and 'broker enforces exp and FIN-closes the session
mid-stream' — observed on the waev preset".
That premise made waev expensive, because waev is the only preset with a short
token_lifetime (3300 s; every other is 0, meaning the 24 h default). It was
therefore the only slot bouncing often: measured every ~47 minutes, about 30
times a day per device. And the bounce's re-handshake is where contiguous
internal DRAM goes — one renewal traced on hardware took the largest free block
from 27,124 to 16,372 B, below the 16,384 B mbedTLS inbound record buffer, after
which that slot could not re-handshake at all. The teardown and the credential
update cost nothing; the handshake costs everything.
So for a broker that leaves live sessions alone, refresh the credentials in place
and let the next genuine reconnect use them. That path already existed for the
"token renewed but old one still valid" case; this just stops treating imminent
expiry as a reason to tear down a healthy connection.
mqttPresetEnforcesTokenExp() defaults to true and is keyed by preset name rather
than a new struct field: adding a field would mean re-ordering a dozen positional
initialisers, where a mistake is silent, and the wrong default costs an outage
rather than a re-handshake. Custom and audience-only slots have no preset and are
treated as enforcing.
Our own logs already argued against the premise and we had not noticed: across 14
multi-device outages (10 hitting all four devices) the drops landed within ~3 s of
each other, on devices whose independent boot times gave them independent token
issue times. Independent expiries cannot align that tightly, so exp enforcement
was never a good explanation for them.
Unverified on hardware yet — the operator's statement is second-hand. Next: apply
to one board only and confirm the session survives past exp, that a later
reconnect still authenticates, and that the ~47-minute 27,124<->16,372
oscillation stops.
(cherry picked from commit 27bd05a17b9303b158feec7dab60af2fe128f5ce)
Three defects found reviewing the preceding commits.
1. reconnectSlotClient() stranded a STOPPED client, reintroducing the very bug
this branch fixes. It only rebuilt when isStarted() was true and otherwise
fell through to reconnect(), which is a documented no-op on a stopped client
— so nothing restarted it, at any rung, including the breaker probe. The
WiFi-transition teardown reaches exactly this state: it calls the hard
disconnect(), clearing _started while initial_connect_done stays set, so
after WiFi returned the slot could never come back. Now a stopped client is
started with connect() before the rebuild/reuse decision is considered.
The post-NTP credential refresh had the same exposure — it called
reconnect() directly — so it now goes through the helper too, still reusing
the transport since its fault is stale credentials, not the transport.
2. Allocating the neighbors buffer on first use let a stopped bridge allocate.
A neighbour discovery started before a stop can complete after it, and
neither caller rechecks bridge state, so requestPublishNeighbors() would
allocate 4 KB after releaseRuntimeBuffers() had already run and strand
_neighbors_publish_pending with no task to consume it. end() then returns
early on !_initialized, retaining the buffer until a later begin/end or a
reboot. Guarded on isRunning(), the same flag end() checks.
The release/acquire handoff itself was confirmed sound: the allocation and
copy precede the release store, and the task loop reads the pointer only
after its acquire load, so a half-published pointer is not observable.
3. The post-link map check failed open, contradicting the fail-closed claim in
its own commit message. A missing map, an unrecognised map format, or a
partial archive list each warned and passed; and it hardcoded firmware.map
while the post-action target used ${PROGNAME}, so a renamed program could
inspect a stale or absent file and still succeed. All four now fail the
build, and it requires every one of the four archives to appear rather than
at least one.
Rebuilt Heltec_v3_repeater_observer_mqtt, Heltec_v3_repeater and
heltec_v4_repeater_observer_mqtt; the opt-in path still reports all 4 archives
linked from .mbedtls-4k/.
(cherry picked from commit 5b5f076e5e165997e8050f2be061c7c67340fcf7)
allocateRuntimeBuffers() took NEIGHBORS_JSON_BUFFER_SIZE unconditionally on
every board built WITH_MQTT_NEIGHBORS, whether or not mqtt.neighbors was ever
turned on. On a non-PSRAM board that is 4 KB of internal DRAM held for the
bridge's lifetime by a node that may never publish a neighbours snapshot.
Gating the existing allocation on the pref would not work: mqtt.neighbors is
read live by the mesh loop with no bridge restart, so enabling it at runtime
would find no buffer and silently publish nothing. Allocate on first use
instead, in requestPublishNeighbors(), which is reached only when something
actually wants to publish — periodic or a manual discovery.
Publishing the pointer across cores is safe with the existing handshake: the
allocation precedes the release store on _neighbors_publish_pending, and the
task loop reads the pointer only after its matching acquire load, so the
pointer cannot be observed half-published. A failed allocation drops that one
snapshot and retries on the next, rather than disabling neighbours for the
bridge's lifetime as the eager path did.
(cherry picked from commit e6da052a93f8765824d0fb4bd0c704ca3ed3d294)
The scheduled JWT bounce called PsychicMqttClient::disconnect(), which ends
with esp_mqtt_client_stop(). That ends the client task and returns its 6 KiB
stack to the heap at the moment the TLS teardown vacates two 16 KiB mbedTLS
record buffers, so the stack lands in that hole and the next handshake cannot
reuse it. On non-PSRAM boards the largest free block then ratchets down 16 KiB
at a time while total free heap stays flat.
Soak evidence from a Heltec V3 on 8d1a0eb3: 43 of 60 disconnects had no
preceding transport error, i.e. they were this proactive bounce rather than a
broker FIN, and two of the three max_alloc steps landed within 5 s of one.
Losing a whole TLS session later returned exactly 16,384 bytes of contiguity.
softDisconnect() closes the transport without the stop, so the task and its
stack stay put across the handshake. The bounce uses it plus reconnect(), and
falls back to connect() when the client really is stopped, since reconnect()
is a silent no-op in that state.
Also corrects a comment claiming the mbedTLS context survives a transport
close: only the esp-mqtt client object does.
(cherry picked from commit 10cf5cf48fb009e751e25b37fcc1f3d1256ddbbc)
The post-NTP-correction refresh looped to _max_active_slots, which is a count of
activation positions and never an index bound. The indices holding those
positions are not contiguous: a slot passed over by isSlotReady() -- or, since
the demand-driven work, by a failed setup -- leaves a higher index activated. On
a two-position board that meant slots 1 and 2 could be live while only indices 0
and 1 were scanned, so slot 3 kept a JWT issued against the pre-correction clock
until its own expiry or a reconnect regenerated it.
Now bounded by RUNTIME_MQTT_SLOTS. The existing guard already skips disabled,
non-JWT, and clientless slots, so widening the range cannot touch a slot that
was never set up.
Pre-existing (the loop predates the demand-driven work) and kept as its own
commit so it can be picked separately. Audited the other _max_active_slots uses:
all are count comparisons or log arguments, so this was the only misuse.
last_reconnect_attempt starts at zero and teardownSlot() re-zeroes it, so the
retry gate in maintainSlotConnections() reduced to "uptime >=
SLOT_SETUP_RETRY_INTERVAL". Past 60 s of uptime a failed setup was therefore
retried on the next maintenance pass rather than 60 s later -- in the same task
iteration for a live reconfigure, since reconfigure processing runs before
maintenance in the loop.
setupSlot() now stamps last_reconnect_attempt on each failure that represents a
real attempt (client allocation, and both JWT token paths), so all three callers
get the interval measured from the failure. The bounds check and the !enabled
early return are not attempts and stay unstamped. The reconnect ladder reads
this field only for slots with initial_connect_done set, which a failed setup
never sets, so reconnect timing is unaffected.
The retry path's own pre-call stamp is kept as a backstop for any future
false-returning path that does not stamp itself.
reconnect_attempted_this_cycle is computed once before the maintenance loop and
only maintainSlotConnection() was setting it. The deferred-setup retry armed the
15 s cross-slot guard via _last_slot_reconnect_ms but left the local flag false,
so a disconnected slot later in the same pass still saw "no reconnect yet" and
started a second TLS handshake. Two concurrent ~40 KB sessions is exactly the
contention the guard prevents, and it landed in the one situation where internal
heap is already known to be short -- a failed allocation is why the retry runs.
Set on success only: setupSlot() returns true only after client->connect(), so
success means a handshake was launched. A failed retry launches nothing and
continues to spend only setup_retry_this_cycle, which rate-limits the allocation
attempts without consuming the handshake allowance.
[P1] A failed setup no longer strands the slot. setupSlot() returns bool and the
startup loops count only successful activations, so a slot that fails on a client
allocation neither consumes an active-slot position (starving a later healthy
broker on capped hardware) nor sits dead forever: maintainSlotConnections()
previously skipped clientless slots and the reconnect ladder is gated on
initial_connect_done, so nothing retried it. It now retries an enabled but
unactivated slot on a 60 s timer, one per cycle, gated on the same
_slots_setup_done ordering so the NTP-deferred setup sequence is preserved.
[P2] JWT setup no longer proceeds without a usable token. Both the preset and
custom-audience paths returned after ignoring createSlotAuthToken()'s result,
then called connect() and latched initial_connect_done -- so the token-allocation
failure introduced by the previous commit produced an unauthenticated attempt
exactly when memory was exhausted. They now return false and let the retry path
handle it.
[P2] ensureSlotAuthToken() no longer clears an existing token. It cleared
unconditionally, so every renewal wiped the current token before JWTHelper ran;
a renewal that then failed left an empty password where the inline buffer used to
preserve working credentials (JWTHelper writes only on success). Only freshly
allocated buffers are initialised now.
[P2] Raw publications reuse the shared document. buildRawJSON() reached
MQTTPayloadBuilder::buildRawMessage(), which constructed its own default
JsonDocument and therefore malloc'd and freed an internal-heap variant pool per
message -- on the highest-rate topic. The document is threaded through both
builders and the bridge passes _json_scratch_doc.
[P3] The writeTo() guard validates the source fields, not just the destination.
A corrupt payload_len of MAX_PACKET_PAYLOAD + 1 still leaves getRawLength()
inside MAX_TRANS_UNIT, so writeTo() read past packet->payload. Sizing and
validation moved to a pure MQTTWireScratch header with host tests covering the
accept/reject edges, matching the MQTTPacketFilter/MQTTConnectionPolicy pattern.
Two findings fell out: MAX_PATH_SIZE one-byte hops is not encodable (the hop
count is 6 bits, so 64 & 63 == 0; 32 two-byte hops is the widest real path), and
a zero-payload packet serializes but does not survive readFrom() -- pinned as a
test because it constrains any future wire-only queue.
[P3] Corrected the pool-size comment: these targets are 32-bit, so
ARDUINOJSON_SLOT_ID_SIZE is 2 and a pool block is 128 slots / 1024 bytes, not
4096. The 4096 figure came from a pre-existing comment near
NEIGHBORS_DOC_POOL_BUDGET, which is left alone -- its byte measurements are
empirical and still stand, only the block-size attribution is wrong.
Activation is now centralized in activatedSlotCount()/canActivateSlot(), used by
both startup loops, the retry path, and applySlotPreset(). That closes the
pre-existing divergence where a live preset change called setupSlot() without
consulting _max_active_slots, letting a non-PSRAM board reach three concurrent
TLS sessions against a cap of two. BEHAVIOUR CHANGE: a reconfigure that would
exceed the cap now logs and leaves the slot inactive instead of connecting.
Reconfiguring an already-active slot still works, because teardownSlot() releases
its position first.
272/272 native tests pass (5 new); both observer envs and an nRF52 repeater build
clean. Flash 1593249 B non-PSRAM, 1555625 B PSRAM.
ensureSlotAuthToken()/releaseSlotAuthToken() now use the bridge's existing
psram_malloc/psram_free rather than malloc/free, so a JWT slot's 768-byte token
comes out of PSRAM instead of internal DRAM. On PSRAM boards this is the larger
half of the auth_token work: the previous commit reclaims 768 B per slot that
never creates a token, while this reclaims 768 B per slot that does -- up to
3840 B of internal DRAM on a fully configured five-slot board, which is where
the mbedTLS working set is competing for space.
No effect on non-PSRAM boards: psram_malloc falls back to internal DRAM, so the
buffer stays exactly where the previous commit left it. Same for a
BOARD_HAS_PSRAM board whose PSRAM fails to initialise.
Safe to move because every access is a CPU copy on the bridge task, never DMA,
an ISR, or a cache-disabled window: JWTHelper memcpy's the token into this
buffer, and esp-mqtt copies it out of _mqtt_cfg into its own internal-DRAM
storage when connect() applies the config. This is unlike the PSRAM-backed MQTT
task stack that was tried and reverted for resetting Heltec V4 boards, where
the fault was PSRAM execution context rather than a plain buffer read.
Split from the previous commit so it can be reverted alone if hardware soak
shows any PSRAM-related instability on the JWT path.
flash non-PSRAM 1592801 -> 1592865 B (+64)
PSRAM 1555213 -> 1555237 B (+24)
267/267 native tests pass; both observer envs and an nRF52 repeater build clean.
MQTTSlot carried an inline char auth_token[768] -- 768 of its 1192 bytes --
for every runtime slot, whether the slot was JWT, username/password, disabled,
or above the active-slot cap. Now a char* allocated by ensureSlotAuthToken()
from createSlotAuthToken(), the sole writer, which runs only once a slot is
confirmed to have a JWT audience.
Because _slots[] is a fixed member array, this shrinks the bridge object
unconditionally at construction and re-adds only what is used:
MQTTSlot 1192 -> 428 B (-764)
MQTTBridge non-PSRAM 12032 -> 9740 B (-2292)
PSRAM 10080 -> 5496 B (-4584)
The object is allocated as one contiguous block at boot, so a smaller boot-time
chunk is easier to satisfy and leaves a larger contiguous remainder; JWT slots
then take 768 B each as separate blocks. Net steady-state saving is 768 B per
slot that never creates a token -- 768 B on a fully configured board, up to
3840 B on a PSRAM board with one JWT slot. 22 of the 29 presets are
MQTT_AUTH_JWT, so this is mostly reclaiming unconfigured slots rather than
non-JWT ones.
Lifetime rules, which are the whole risk here:
- setCredentials() stores this pointer in _mqtt_cfg rather than copying, and
esp-mqtt re-reads it whenever a later connect() re-applies a dirtied config.
So the buffer is freed only alongside the client, in destroySlotClients(),
after the delete. MQTTSlot::broker_uri already carries an "avoids dangling
pointer" comment from this same hazard with setServer().
- teardownSlot() still clears the token to an empty string but keeps the
buffer: the client survives teardown and does not clear cfg->password (only
setupSlot()'s reconfigure branch does). Freeing there would dangle.
- Never freed per reconnect, preserving the churn-avoidance the inline buffer
was there for.
Allocation goes through MQTTRuntimeBufferLifecycle (already host-tested) using
plain malloc, so the buffer stays in internal DRAM exactly where it was when
inline. Failure propagates through createSlotAuthToken()'s existing bool.
The two call sites that create a token and then test it (preset JWT in
setupSlot, and the custom-slot audience path) now null-check first; the five
other readers are all inside if (createSlotAuthToken(...)) success branches and
need no change. destroySlotClients() releases the token unconditionally rather
than after its client null-check, so a token can never outlive its slot.
flash non-PSRAM 1592697 -> 1592801 B (+104)
PSRAM 1555109 -> 1555213 B (+104)
267/267 native tests pass; both observer envs build clean.
initSlotClients() created a PsychicMqttClient for every one of
RUNTIME_MQTT_SLOTS at begin(), without consulting slot.enabled or the active
cap -- even though presets are applied and _max_active_slots is computed
earlier in the same function. RUNTIME_MQTT_SLOTS is deliberately cap+1
(6/5 with PSRAM, 3/2 without), so at least one client was always unusable,
and a lightly-configured board wasted several.
Replaced with ensureSlotClient(index), called from setupSlot() -- i.e. only
for a slot that is enabled, inside the cap, and ready to connect. A client
that never reaches setupSlot() is completely inert: the reconnect ladder is
gated on initial_connect_done, which only setupSlot() sets. Retained for the
bridge lifetime as before, so reconfigure/reconnect still reuse one mbedTLS
context; that context is built by connect(), not by the constructor, so
deferring costs nothing but the 1284-byte object.
Internal DRAM saved, by configured slot count:
non-PSRAM (3 runtime / 2 cap) 1 configured 2568 B
2-3 1284 B
PSRAM (6 runtime / 5 cap) 1 configured 6420 B
2 5136 B
5-6 1284 B
A BOARD_HAS_PSRAM board whose PSRAM fails to init gets cap 2 against 6
runtime slots, so it saves at least 5136 B -- on the board that just lost its
PSRAM.
Two gates used "client != nullptr" as a proxy for "configured". Under eager
allocation that conjunct was always true and therefore harmless; with lazy
allocation it would have made shouldQueuePacketType() drop every packet
received before the post-NTP-sync slot setup, which is precisely the window
the queue exists to cover. Both now key on slot.enabled, matching the
documented intent above eligiblePacketSlots() that a configured-but-
disconnected broker is still a target.
formatSlotDiagReply() gains the !isSlotReady() -> "wait" branch that
get mqtt.status and getSlotStatusSnapshot() already have; that state
previously fell through to "disc" and read as a network fault when it is
really a missing token/IATA/credential. "no client" now means only what its
name says: the slot is ready but the client could not be allocated. That
state is newly reachable, so the allocation uses new (std::nothrow) -- this
framework enables C++ exceptions, and a throwing new on exhaustion would
panic the node instead of degrading one slot.
flash non-PSRAM 1592573 -> 1592697 B (+124)
PSRAM 1555001 -> 1555109 B (+108)
267/267 native tests pass; both observer envs build clean.
Five independent, behavior-preserving reductions. Measured on
Heltec_v3_repeater_observer_mqtt (non-PSRAM) and
ThinkNode_M7_repeater_observer_mqtt (PSRAM), 267/267 native tests green.
Drop the unused static-task bookkeeping. StaticTask_t _mqtt_task_tcb (344 B)
and StackType_t* _mqtt_task_stack were never used: there is no
xTaskCreateStatic call, the pointer was assigned nullptr immediately before
xTaskCreatePinnedToCore, and the two psram_free() calls on it were dead.
Size the wire-format scratch buffers from the protocol maximum. raw_hex[1024]
at three sites becomes 2*MAX_TRANS_UNIT+1, and raw_buf/reconstructed[512]
become MAX_TRANS_UNIT. Both writeTo() sites now check getRawLength() first:
writeTo() does not bounds-check and returns uint8_t, so the old 512-byte
buffers were the only thing absorbing a malformed payload_len, and the
post-hoc "raw_len > sizeof(buf)" test ran after the overrun.
Pass the already-known serialized length into publishToSlot() instead of
re-running strlen() per destination slot (up to 2 KB per packet per slot, and
NEIGHBORS_JSON_BUFFER_SIZE per neighbor snapshot). Same for the direct
publish in publishStatusToSlot().
Share one JSON buffer and one document across packet, raw, and status. All
publish paths serialize on the bridge task, so the separate status buffer and
document were never concurrent. Status keeps STATUS_JSON_BUFFER_SIZE as its
serialization ceiling, so which oversized status documents get dropped is
unchanged.
Route the document's pools through a PSRAM-preferring allocator. Under
ArduinoJson 7 StaticJsonDocument<N> is a deprecated empty subclass of
JsonDocument whose template argument only feeds capacity(); the object is
64 B and each pool block (4096 B here) came from plain malloc(), i.e. the
internal DRAM the mbedTLS working set needs. Mirrors NeighborsDocAllocator.
The old comment claiming an inline pool has been corrected.
Measured:
sizeof(MQTTBridge) non-PSRAM 13208 -> 12032 B (-1176, internal heap)
PSRAM 10492 -> 10080 B (-412, plus one fewer
768 B PSRAM allocation)
stack, non-PSRAM buildPacketJSON[FromRaw] 1264 -> 736 B
buildRawJSON 1136 -> 608 B
publishPacket 688 -> ~432 B
packetToHex 560 -> ~304 B
deepest publish chain ~2.6 -> ~1.8 KB of 8 KB
flash 1592513 -> 1592573 B (+60)
static RAM 74656 B unchanged -- MQTTBridge is heap-allocated, so
these savings are internal heap, not the linker figure
Deferred from the review: the QueuedPacket wire-only redesign (reward is
1.56 KB on non-PSRAM only, and Packet::readFrom() rejects payload_len == 0),
demand-driven slot clients/JWT tokens, and pool retention across publishes --
JsonDocument::to<T>() always calls clear(), which destroys pools, so
"retain pools by clearing the root object" needs a string-pool lifetime
analysis first.
Neighbors publication was gated on BOARD_HAS_PSRAM. Removing that gate alone
was not enough: the feature built but was inert without PSRAM, because three
allocation sites asked for MALLOC_CAP_SPIRAM (which returns null with no
PSRAM), the bridge's persistent buffer was allocated inside a BOARD_HAS_PSRAM
block, and neighborDiscoverReady() rejected every pass at runtime on
psramFound(). The entry table also did not fit: finishNeighborDiscover put
pubkey_hex[50][65] plus entries[50] on the stack, a 4752-byte frame against
the mesh loop task's 8 KB.
- Gate on MAX_NEIGHBOURS plus PSRAM or an explicit per-variant
MQTT_NEIGHBORS_WITHOUT_PSRAM opt-in.
- Move the entry table and its hex strings into one heap block sized to the
pass; the frame drops from 4752 to 304 bytes.
- Prefer PSRAM and fall back to internal DRAM in the mesh-side allocations
and the ArduinoJson pool; hoist the bridge's persistent buffer out of the
BOARD_HAS_PSRAM block (psram_malloc already falls back).
- Keep the runtime psramFound() check only where the buffers are sized for
PSRAM, so a board whose PSRAM failed to init still refuses.
- Size for internal DRAM without PSRAM: 4 KB text buffer and 20 entries per
publish, keeping the pool to a single block and the peak near 13 KB rather
than ~35 KB. Oversized tables truncate and report total_neighbors as before.
Enabled on the ESP32-S3 observer envs (Heltec V3/WSL3, RAK3112, Heltec
Tracker v1.1/v2). Left off for the classic ESP32 T-LoRa V2.1-1.6, which is
already limited to one active TLS slot.
Costs ~7.4 KB static DRAM on repeaters and ~9.6 KB on room servers. The
prefs layout is unchanged, so this is neutral for existing devices.
MeshRank slots previously took packets only. The broker tolerates status and
neighbors (and its maintainer intends to look at using the neighbors data), so
those now publish under meshrank/uplink/{token}/{device}/{type}, using the same
type suffixes as the MeshCore layout. Raw stays excluded: it is the
highest-volume topic and the broker does not consume it, so `set mqtt.raw on`
has no effect on a MeshRank slot.
All MeshRank gating funnels through the topic router, so relaxing that single
guard was sufficient - publishStatusToSlot, publishStatus, publishRaw,
publishNeighbors, and eligiblePacketSlots already skip slots that cannot form a
topic. A per-slot token is still required. Because eligiblePacketSlots resolves
topic support before serialising, raw JSON is never built for a MeshRank slot
rather than built and discarded.
observer-firmware sends raw to MeshRank; this supersedes that decision, so raw
must stay excluded when the branches merge. MeshRankTakesEveryTypeExceptRaw
fails if a merge re-enables it.
Add support for named packet types in per-slot filters, allowing users
to specify packet types using descriptive names alongside numeric values.
This improves usability and clarity in configuring MQTT slot filters.
Updates include modifications to the parsing logic, WebConfig interface,
and related documentation to reflect the new naming conventions.
Introduce per-slot packet filters to allow users to specify which
packet types are uploaded for each MQTT slot. This feature enhances
the flexibility of the MQTT bridge by enabling users to configure
allowlists for packet types, improving the efficiency of data
transmissions. The implementation includes updates to the WebConfig
interface, internal handling of packet filters, and necessary
modifications to the MQTT preferences structure.
Introduce new methods to manage neighbor discovery JSON budget and
entries in MyMesh. This includes tracking the number of queried and
published neighbors, measuring JSON sizes, and handling truncation
when the buffer limit is reached. These improvements optimize the
neighbor discovery process and ensure efficient JSON message
construction for MQTT communications.
Enhance the MQTT stats reply to include outbox drop statistics,
providing users with better insight into message handling and
memory pressure. This change improves the clarity of the status
message and aids in diagnosing potential issues with message
delivery.
Enhance the command handling in CommonCLI_Observer to clarify the
compilation behavior under different configurations. Additionally,
optimize memory pressure checks in MQTTBridge to reduce unnecessary
heap walks, improving performance during sustained memory pressure.
This change ensures that publishes are skipped only when necessary,
enhancing overall stability.
Eliminate unnecessary calls to gmtime and localtime in the
syncTimeWithNTP function. This cleanup improves code clarity
and reduces potential confusion regarding time handling.
Introduce a mechanism to manage on-connect status publishing for MQTT
slots. This change allows the MQTT task to handle status updates
safely, ensuring that shared resources are accessed in a controlled
manner. The new flag `_status_publish_pending` is set in the onConnect
callback and processed in the main MQTT task loop, preventing race
conditions and improving overall stability.
Replace multiple snprintf calls with a single replyAppendf function to
improve buffer management and prevent overflow. This change enhances
the readability and maintainability of the code while ensuring that
the reply buffer is handled safely across various MQTT status and
diagnostic replies.
Port the periodic-neighbors publication from mqtt-bridge-implementation-flex,
adapted to this branch's structure:
- Add MQTT_PUBLICATION_NEIGHBORS ("neighbors") to the pure MQTTTopicRouter
instead of flex's messageTypeSuffix() helper (this branch already routes
every publication type through mqttBuildPublicationTopic()). Neighbors is a
MeshCore/custom publication type, so it resolves to
meshcore/{iata}/{device}/neighbors and honors custom templates.
- Deliberately do NOT port flex's "all message types to MeshRank" change:
this branch documents and host-tests a packets-only MeshRank contract
(MQTTPresets.h, MQTT_IMPLEMENTATION.md, MeshRankContractIsPacketsOnly). So
neighbors follows status/raw and is rejected on MeshRank slots.
- WITH_MQTT_NEIGHBORS guard (PSRAM + MAX_NEIGHBOURS) gates all new surface.
- MSG_NEIGHBORS message type + enum-drift static_assert.
- Persistent ~10KB PSRAM neighbors buffer allocated/freed via the existing
MQTTRuntimeBufferLifecycle path (allocate/release), not the ctor as flex did.
- Core1->Core0 handoff: requestPublishNeighbors() (mesh) fills the buffer with
a release store; the MQTT task consumes it with an acquire load, publishes
via publishNeighbors() (QoS1, retain = preset->allow_retain, custom=false),
and clears the pending flag. A second snapshot is dropped while one is
in flight.
- setNeighborsSchedule()/NeighborsPhase let the mesh report the timer summary;
formatMqttStatusReply() gains a "nbr: <when>/<last>" field via formatDuration.
Also fix the on-connect status publish (publishStatusToSlot) to honor
preset->allow_retain instead of hardcoding retain=true, matching the periodic
publishStatus() path. Brokers with allow_retain=false (e.g. the waev MeshCore
preset) reject retained publishes, so the on-connect status was being dropped
there. This is flex followup 028a5dca, reconciled to this branch's custom-slot
default of non-retained.
Extends the host topic-router test to cover the neighbors type across all
routes and freezes the new enum value. Bridge itself is on-target only.
UTF-8 em-dashes (U+2014) inside MQTT_DEBUG_PRINTLN / CLI reply strings
render as mojibake ("aEUR"-style, e.g. cooperative stop garbles) on
consoles that don't decode UTF-8. Replace the em-dashes in printed
strings with ASCII '-'. Comments are intentionally left unchanged (they
never reach the console; rewriting them would be needless churn in a
merge-sensitive file).
Verified on V3: the stop message now emits pure ASCII -- zero non-ASCII
bytes in the boot+stop console capture.
Phase 0 hardware characterization showed real mbedTLS/wss client teardown
takes ~5-6s per connected slot, sequentially, so the flat 8s
MQTT_STOP_TIMEOUT_MS force-timed-out healthy multi-slot stops (2-slot
non-PSRAM ~11-12s, 3-slot PSRAM ~16s), which sets the dirty latch and
makes the OTA barrier withhold flashing -- multi-slot nodes could never
ota update.
Replace the flat constant with a slot-scaled budget computed per stop in
end(): 5s base + 8s per enabled slot (~1.5-2x headroom over measured),
applied via new MQTTLifecycle::Coordinator::setStopTimeoutMs() before
requestStop(). Headroom is nearly free: end() returns as soon as the task
acks (checked before the timeout ticks), so a larger bound only lengthens
the wait before force-killing a genuinely wedged task.
Hardware-verified on V3: a 2-slot stop that force-timed-out at 8s pre-fix
now logs "timeout 21000 ms" and acks clean in ~11.7s. Native suite +
both observer firmware builds green.
Close the host-testable gaps named in Phase 6 of STABILITY_TESTABILITY_HANDOFF.md
by moving the last inline decision logic into the pure, host-tested policy seams:
- WiFi STA reconnect backoff: extract the inline ladder + wrap-safe timing from
handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,
wifiReconnectDue,nextWifiBackoffAttempt}. Behavior-preserving (elapsedMs is the
wrap-safe form of the old ULONG_MAX branch); ladder/clamp/attempt-cap unchanged.
- Publication outcome pairing: name the (packet, raw) -> delivered contract as
MQTTPacketQueuePolicy::queuedPacketPublished() and wire both queue-drain sites;
partial success = completed, not retried.
- Freeze MQTTPublicationType enum values in a test (the bridge-side MQTTMessageType
alignment is already enforced by a compile-time static_assert).
Adds host tests for all three (exact boundaries + millis() rollover). Native suite
green (13 dirs); non-PSRAM observer firmware smoke build compiles.
WebConfig batch/reboot/stop state-machine extraction and queue-orchestration
coverage remain open (tracked in the Phase 6 status).
Wire the Phase 4 MQTTLifecycle state machine into MQTTBridge to replace the
blind vTaskDelete teardown that could kill the MQTT task mid-mbedTLS and then
free client buffers on a corrupted heap (the observed OTA teardown panic).
- MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task
(Core 1) from begin()/end(); a nested LifecycleOps binds the host-tested Ops
spec to FreeRTOS/PsychicMqttClient.
- end() requests a cooperative stop; the MQTT task (Core 0) tears down its own
clients where the mbedTLS contexts live, acks via _stop_acked, and
self-terminates. end() waits (bounded) for the ack, then frees queue/buffers.
- Bounded stop timeout -> reviewed fallback (force kill + Core-1 teardown) sets
a dirty latch that withholds OTA flashing.
- begin() gains an idempotent double-call guard and syncs the Coordinator to
Running.
- OTA teardown barrier: simple_repeater's deferred flash aborts/resumes unless
end() reported a clean stop (canFlashAfterStop()).
Scope: minimal cooperative-shutdown unit. The volatile NTP/reconfigure handshake
replacement and the plain-data snapshot / consumer repointing (MQTT_OWNERSHIP.md
sections 1-3) are deferred. MQTT_STOP_TIMEOUT_MS is a Phase-0 placeholder pending
on-hardware characterization.
Native suite green (incl. test_mqtt_lifecycle); both observer firmware smoke
builds compile. Not yet hardware-validated (Phase 7 gate).
Replace direct JSON construction in MQTTMessageBuilder with calls to
MQTTPayloadBuilder for building status, packet, and raw messages.
This change improves code maintainability and reduces duplication by
centralizing message formatting logic. Additionally, update platformio.ini
to include ArduinoJson dependency for JSON handling.
Include detailed instructions for local testing of observer and WiFi
functionality without hardware. Document the use of a mock backend and
Wokwi ESP32-S3 simulation for easier development and testing.
Enhance the MQTT implementation documentation to improve developer
experience and facilitate testing workflows.
Introduce a web configuration portal for easier node management and
provisioning without serial CLI. Enhance MQTT functionality with
improved IATA code validation, dynamic slot management, and
background NTP synchronization. Update web UI elements for better
user experience and security notes regarding open AP usage.
agessaman