The reset was on the wrong side of configTime(). configTime() configures the
server, calls sntp_init(), and returns — the new request is live before it comes
back — so a fast reply could set SNTP_SYNC_STATUS_COMPLETED inside that call, and
the reset immediately after would erase it. The following ten seconds of polling
would then see nothing and reject a server that had in fact answered. On the
`set mqtt.ntp` path that surfaces as a good server failing validation.
Stop any running session first, discard its status, then start the new one, so
the only completion observable is the one being waited for.
Requiring a real SNTP completion took away something the plausible-clock test was
doing by accident. _ntp_synced gates slot setup outright (:1386, :2894), so a
device that cannot reach NTP now brings up no slots at all — and a network that
blocks UDP/123 while allowing 443 is an ordinary firewall configuration, not a
corner case. An RTC-backed observer there used to stay synced and keep minting
JWTs against a perfectly good clock.
Accept the existing clock explicitly when every server has failed, logged as what
it is rather than as a claim about a server that never replied. Excluded from the
`set mqtt.ntp` validation path, where the question is whether that server works
and the clock cannot answer it. configTime() is now called only when a server did
answer, since otherwise there is nothing new to point SNTP at.
The corrected-clock path reconnected a disconnected slot whether or not
createSlotAuthToken() had produced anything, which re-presented the credentials
the correction had just invalidated. Minting fails for recoverable reasons —
allocation pressure is treated as recoverable elsewhere in this file — so the
path is reachable, and the reconnect it spends is one that cannot succeed.
Move the decision into MQTTConnectionPolicy as classifyStaleToken(), where the
four outcomes are named and host-tested rather than spelled out in nested
conditions: Defer on a failed mint, Reconnect a client that is down, Bounce a
live session whose broker enforces exp, KeepAlive one whose broker does not.
Deferring leaves the slot to the backoff ladder, which mints again on its next
attempt.
Covers the reviewer's first four cases. The other two — that a completed SNTP
sync is required, and that time(nullptr) reflects the accepted epoch before
_ntp_synced flips — are inside MQTTBridge.cpp, which the native env does not
compile; locking those down needs a seam around the IDF calls that does not
exist yet.
syncTimeWithNTP() read an epoch over UDP, called configTime(), set _ntp_synced,
and then had the stale-token test and createSlotAuthToken() read time(nullptr) —
without anything having put the accepted epoch there. configTime() restarts SNTP
and returns; the clock lands whenever a packet does.
_rtc->setCurrentTime() looks like it covers this and does not.
AutoDiscoverRTCClock::setCurrentTime() writes a detected DS3231/RV3028/PCF8563/
RX8130CE *instead of* delegating to its fallback, and only that fallback
(ESP32RTCClock) calls settimeofday(). So on every board carrying an RTC chip —
T-Beam Supreme and Station G3 both compile this bridge and both instantiate
AutoDiscoverRTCClock — libc kept the pre-correction time, and the correction path
tested staleness and minted iat claims against exactly the clock it had just
proven wrong. Boards without a chip take the fallback and were unaffected, which
is why the soak rig (Heltec V3/V4, no RTC) never showed it.
settimeofday() with the accepted epoch first, so the invariant downstream code
already assumes actually holds: once _ntp_synced is true, time(nullptr) returns
the epoch we accepted. configTime() still follows, to keep future syncs running.
The fallback configured a server, waited 500 ms, and accepted any plausible
system clock as proof that server had answered. It usually has not answered
that fast — and the device usually already holds valid time, from an earlier
sync or the RTC — so the first server in the list was credited unconditionally,
the walk stopped there, _last_ntp_sync was refreshed, and an unreachable host
was logged as the source. On the `set mqtt.ntp` validation path, where the
single-server walk exists so a typo fails fast, that reported a bad server as OK.
Poll sntp_get_sync_status() for SNTP_SYNC_STATUS_COMPLETED instead, which is the
layer's own statement that a packet arrived. The status is one-shot — reading
COMPLETED clears it — so a result left by an earlier sync would latch on the
first poll; clear it before the loop.
An implausible epoch after a completed sync now moves to the next server rather
than spinning out the remaining attempts against a server that has answered.
The manifest said "rebuild these for every espressif32 platform bump" and nothing
enforced it. mbedtls_4k.py verified the staged archives against the manifest's own
hashes, which proves the pair agrees with itself and nothing more: bump the
platform without rebuilding and every check still passes while the link takes
mbedTLS built against a different IDF. That fails at runtime on struct-layout
drift, not at the link, which is the failure the mechanism claimed to prevent.
Fingerprint the framework's own mbedTLS archives — the ones ours displace — as
stock: lines in the manifest and check them before the build. If the framework
moves, the staged pair is stale by construction and the build stops with the
replacement hashes printed ready to paste. Stronger than comparing a version
string: framework-arduinoespressif32 versions independently of the platform, and
its archives are what actually has to match.
The lib directory is resolved by trying the layouts espressif32 has used rather
than hardcoding one, and failing closed if none holds all four archives. The fetch
script ignores the new lines; its known-arches hint skips them so they cannot be
reported as architectures.
esp_mqtt_client_reconnect() is honoured only from MQTT_STATE_WAIT_RECONNECT, so
the post-correction path minted a fresh token, staged it, and then asked a
connected client to reconnect — a request esp-mqtt refuses. The slot kept running
on the token the clock correction had just proven stale, and recovery became
broker-driven rather than the clean reconnect this code intends.
Split the three states the path can find: a stopped or waiting client goes through
reconnectSlotClient() as before, and a live one has its transport closed first.
Only where the broker enforces exp, though — waev leaves live sessions alone past
expiry, so bouncing it would spend the 16 KiB contiguous handshake that the rest of
this branch exists to avoid.
Not a regression: the base branch called client->reconnect() directly at the same
site. On ESP32 the block is reachable from the WiFi-reconnect resync and the CLI
forced sync; the hourly refresh uses refreshNTP(), which does not carry it.
connect() logged "MQTT client started." unconditionally, so a failing
esp_mqtt_client_start() looked identical to a successful one. That is the one
state a later reconnect() cannot recover from, which made it the worst possible
line to be wrong.
reconnectSlotClient() checks isStarted() and calls connect() instead of
reconnect() when the client was stopped, but only the post-NTP stale-token
path used it. The ordinary backoff ladder and the circuit-breaker probe called
slot.client->reconnect() directly, and esp_mqtt_client_reconnect() is a no-op
on a client that is not started.
Two ways in. connect() sets _started only when esp_mqtt_client_start() returns
ESP_OK while setupSlot() sets initial_connect_done unconditionally, so a start
failure under heap pressure stranded the slot. More routinely, the WiFi-drop
handler calls disconnect() on every connected slot, which clears _started —
after that the ladder issued no-ops forever and the slot never came back.
Not caught by the soaks: the log line the guard prints can only come from the
NTP path, so a stranded slot and a slot that never entered the state produce
identical logs. Observed reconnects were broker-side drops with WiFi up, which
leave the client started.
The renewal-bounce path keeps its own isStarted() branch — it needs
softDisconnect(), which the helper does not do.
Every ordinary backoff reconnect and every circuit-breaker probe minted a
fresh JWT and re-applied credentials, with no check of whether the existing
token was still valid. setCredentials() always dirties the esp-mqtt config, so
reconnect() then called esp_mqtt_set_config() as well. On a flapping broker
that is a signing plus a configuration-copy cycle on every retry, and these
observers see ~38 genuine reconnects/day per slot.
The no-bounce renewal change (27bd05a1) only stopped the proactive renewal
from tearing down a live session; it left this retry path untouched, which is
why a soak shows renewals neither firing nor failing for hours while drops
continue — each reconnect silently re-mints and pushes the expiry out.
Reuse the credentials when their validity is provable and refresh them
otherwise. canReuseJwtForReconnect() lives with the other policy predicates so
it is host-testable, and it establishes current_time < token_expires_at before
subtracting: token_expires_at is unsigned, so an already-expired token would
otherwise wrap to ~4e9 seconds and read as valid for decades. The
>= kMinimumValidEpoch term also rejects the 0 that a failed renewal writes.
Minting stays the default for every uncertain case — unsynced clock, missing or
insane expiry, empty token, or an expiry inside kJwtReconnectSafetyMarginSecs
(60 s), which covers the handshake itself.
Two paths still always mint, deliberately:
- The circuit-breaker probe. It is the recovery of last resort for a slot
that has already failed repeatedly, quite possibly on auth, and it runs
once per 30 minutes — so a fresh token there costs nothing worth counting
against keeping that path guaranteed-clean.
- Any slot whose last error was a broker refusal. Before this change, minting
on every retry accidentally recovered from server-side credential
invalidation: key rotation, revocation, broker clock skew, or an audience
change after a reconfigure. Reuse would have retried a rejected credential
until it neared expiry — up to 24 h for every preset that leaves
token_lifetime at the default. onError already detects
MQTT_ERROR_TYPE_CONNECTION_REFUSED and only logged it; it now also sets a
per-slot force-mint flag, cleared on a successful connect and wherever the
credentials it referred to are blanked. The flag is volatile because the
esp-mqtt callback sets it and the bridge loop consumes it.
The reconnect log line reports the decision and its outcome — REUSE, MINT with
a reason, and OK/FAILED for the mint — because a silently failed mint is the
case most likely to end in an auth refusal. It never prints the token.
Host tests cover the reuse boundary: exact margin, already-expired, expiry 0,
sub-epoch expiry, empty token, unsynced clock, and the force-mint override.
The comment claimed the (WS_BUFFER_SIZE + 1) padding makes an oversized
upgrade response "fail cleanly (Upgrade header not found)". Reading
release/v4.4 transport_ws.c against release/v5.3 shows it does not.
v4.4's response loop is
} while (NULL == strstr(ws->buffer, "\r\n\r\n") && header_len < WS_BUFFER_SIZE);
so it also exits when the buffer fills without the terminator, and the code
then looks for "Sec-WebSocket-Accept:" and returns 0 if it is present. That
header appears early in a response, so an oversized header block yields a
BOGUS SUCCESS rather than a clean failure: the unread remainder stays queued
on the socket and is delivered as the first post-upgrade read, where the
deframer parses HTTP bytes as a WebSocket frame header.
Observed on hardware 2026-08-12 on a Heltec V4: a Cloudflare Page Shield CSP
report-uri header pushed the 101 response past the buffer, and the tail of
that header ("csp-reporting.cloudflare.com/cdn-cgi/script_monitor/report?")
reached the MQTT layer as payload, surfacing as
"Invalid MSG_TYPE response: 3" (0x35 = '5', high nibble 3).
The padding's real and only value is preventing the one-byte overflow of the
heap canary, which is still worth having. Narrow the comment to that claim and
record where the parser fix has to come from: IDF 5.2+ requires the "\r\n\r\n"
delimiter, memmoves the bytes following it, and fails cleanly when the buffer
fills. It cannot be patched here, since transport_ws.c ships precompiled in
libtcp_transport.a on Arduino 2.x.
No functional change.
(cherry picked from commit 9894e65e8ad961704d430a12a065baffda353f50)
waev's operator confirmed on 2026-08-11 that their servers do not disconnect a
client when its JWT passes exp — a 60-minute token can hold a session open for
hours. The renewal path assumed the opposite, in as many words: the comment at
the bounce called the renewal buffer "the ONLY margin between 'device
re-authenticates' and 'broker enforces exp and FIN-closes the session
mid-stream' — observed on the waev preset".
That premise made waev expensive, because waev is the only preset with a short
token_lifetime (3300 s; every other is 0, meaning the 24 h default). It was
therefore the only slot bouncing often: measured every ~47 minutes, about 30
times a day per device. And the bounce's re-handshake is where contiguous
internal DRAM goes — one renewal traced on hardware took the largest free block
from 27,124 to 16,372 B, below the 16,384 B mbedTLS inbound record buffer, after
which that slot could not re-handshake at all. The teardown and the credential
update cost nothing; the handshake costs everything.
So for a broker that leaves live sessions alone, refresh the credentials in place
and let the next genuine reconnect use them. That path already existed for the
"token renewed but old one still valid" case; this just stops treating imminent
expiry as a reason to tear down a healthy connection.
mqttPresetEnforcesTokenExp() defaults to true and is keyed by preset name rather
than a new struct field: adding a field would mean re-ordering a dozen positional
initialisers, where a mistake is silent, and the wrong default costs an outage
rather than a re-handshake. Custom and audience-only slots have no preset and are
treated as enforcing.
Our own logs already argued against the premise and we had not noticed: across 14
multi-device outages (10 hitting all four devices) the drops landed within ~3 s of
each other, on devices whose independent boot times gave them independent token
issue times. Independent expiries cannot align that tightly, so exp enforcement
was never a good explanation for them.
Unverified on hardware yet — the operator's statement is second-hand. Next: apply
to one board only and confirm the session survives past exp, that a later
reconnect still authenticates, and that the ~47-minute 27,124<->16,372
oscillation stops.
(cherry picked from commit 27bd05a17b9303b158feec7dab60af2fe128f5ce)
Three defects found reviewing the preceding commits.
1. reconnectSlotClient() stranded a STOPPED client, reintroducing the very bug
this branch fixes. It only rebuilt when isStarted() was true and otherwise
fell through to reconnect(), which is a documented no-op on a stopped client
— so nothing restarted it, at any rung, including the breaker probe. The
WiFi-transition teardown reaches exactly this state: it calls the hard
disconnect(), clearing _started while initial_connect_done stays set, so
after WiFi returned the slot could never come back. Now a stopped client is
started with connect() before the rebuild/reuse decision is considered.
The post-NTP credential refresh had the same exposure — it called
reconnect() directly — so it now goes through the helper too, still reusing
the transport since its fault is stale credentials, not the transport.
2. Allocating the neighbors buffer on first use let a stopped bridge allocate.
A neighbour discovery started before a stop can complete after it, and
neither caller rechecks bridge state, so requestPublishNeighbors() would
allocate 4 KB after releaseRuntimeBuffers() had already run and strand
_neighbors_publish_pending with no task to consume it. end() then returns
early on !_initialized, retaining the buffer until a later begin/end or a
reboot. Guarded on isRunning(), the same flag end() checks.
The release/acquire handoff itself was confirmed sound: the allocation and
copy precede the release store, and the task loop reads the pointer only
after its acquire load, so a half-published pointer is not observable.
3. The post-link map check failed open, contradicting the fail-closed claim in
its own commit message. A missing map, an unrecognised map format, or a
partial archive list each warned and passed; and it hardcoded firmware.map
while the post-action target used ${PROGNAME}, so a renamed program could
inspect a stale or absent file and still succeed. All four now fail the
build, and it requires every one of the four archives to appear rather than
at least one.
Rebuilt Heltec_v3_repeater_observer_mqtt, Heltec_v3_repeater and
heltec_v4_repeater_observer_mqtt; the opt-in path still reports all 4 archives
linked from .mbedtls-4k/.
(cherry picked from commit 5b5f076e5e165997e8050f2be061c7c67340fcf7)
The reduced-TLS work was validated on hardware but only reachable through
PLATFORMIO_BUILD_FLAGS pointing at an absolute path in a developer's home
directory, so nothing outside that machine could reproduce it.
Distribute the archives as a release asset instead of committing them: ~6 MB
per architecture, and they must be rebuilt for every espressif32 bump, so
committing would grow history permanently and go stale without any signal.
scripts/mbedtls_4k_manifest.txt per-arch sha256 of each archive
scripts/fetch_mbedtls_4k.sh fetch into .mbedtls-4k/<arch>/, verify
scripts/mbedtls_4k.py pre-build wiring and post-link proof
Off by default. The script is attached to esp32_base but returns immediately
unless MESHCORE_REDUCED_TLS=1, so ordinary builds need no artifact and are
byte-for-byte unaffected — confirmed by building with it absent.
Both ways this can fail silently produce a firmware that looks fine and lacks
the change, so the opt-in path refuses to guess:
- a -L at a missing or partial directory: the linker ignores an unusable
search path and resolves mbedTLS from the framework. Now a hard error.
- archives left over from an earlier platform version: now a sha256
mismatch against the manifest, naming both hashes.
- a -L that is present but outranked, leaving the flag inert: after the
link, firmware.map must resolve every libmbed*.a into .mbedtls-4k/, or
the build fails and prints the offending paths.
That last check earned its place immediately — it caught its own first
implementation comparing a relative map path against an absolute one, and an
earlier build flag in this investigation was accepted by the compiler while
no source read it. A flag reaching the compiler proves nothing about the link.
Verified all four paths on Heltec_v3_repeater_observer_mqtt: default build
unaffected; opted in with archives present links all four from .mbedtls-4k/
and says so; archives absent fails with a fetch hint; a single appended byte
fails on sha256.
Also removes platformio.local.ini.hold, which held the superseded approach of
pointing platform_packages at a whole custom framework. That installs over the
shared framework package and changes mbedTLS for every other ESP32 project on
the machine; the -L path keeps the change scoped to one env.
Note the inbound record buffer stays at 16 KiB, so this lowers per-connection
footprint by ~12 KiB but does not move the contiguous allocation a handshake
needs. It buys headroom, not a lower floor.
(cherry picked from commit a87faff6ff170c328fdd0550f4b4dd9089aa2ea0)
allocateRuntimeBuffers() took NEIGHBORS_JSON_BUFFER_SIZE unconditionally on
every board built WITH_MQTT_NEIGHBORS, whether or not mqtt.neighbors was ever
turned on. On a non-PSRAM board that is 4 KB of internal DRAM held for the
bridge's lifetime by a node that may never publish a neighbours snapshot.
Gating the existing allocation on the pref would not work: mqtt.neighbors is
read live by the mesh loop with no bridge restart, so enabling it at runtime
would find no buffer and silently publish nothing. Allocate on first use
instead, in requestPublishNeighbors(), which is reached only when something
actually wants to publish — periodic or a manual discovery.
Publishing the pointer across cores is safe with the existing handshake: the
allocation precedes the release store on _neighbors_publish_pending, and the
task loop reads the pointer only after its matching acquire load, so the
pointer cannot be observed half-published. A failed allocation drops that one
snapshot and retries on the next, rather than disabling neighbours for the
bridge's lifetime as the eager path did.
(cherry picked from commit e6da052a93f8765824d0fb4bd0c704ca3ed3d294)
The scheduled JWT bounce called PsychicMqttClient::disconnect(), which ends
with esp_mqtt_client_stop(). That ends the client task and returns its 6 KiB
stack to the heap at the moment the TLS teardown vacates two 16 KiB mbedTLS
record buffers, so the stack lands in that hole and the next handshake cannot
reuse it. On non-PSRAM boards the largest free block then ratchets down 16 KiB
at a time while total free heap stays flat.
Soak evidence from a Heltec V3 on 8d1a0eb3: 43 of 60 disconnects had no
preceding transport error, i.e. they were this proactive bounce rather than a
broker FIN, and two of the three max_alloc steps landed within 5 s of one.
Losing a whole TLS session later returned exactly 16,384 bytes of contiguity.
softDisconnect() closes the transport without the stop, so the task and its
stack stay put across the handshake. The bounce uses it plus reconnect(), and
falls back to connect() when the client really is stopped, since reconnect()
is a silent no-op in that state.
Also corrects a comment claiming the mbedTLS context survives a transport
close: only the esp-mqtt client object does.
(cherry picked from commit 10cf5cf48fb009e751e25b37fcc1f3d1256ddbbc)
The real Arduino.h includes stdlib.h, so ConfigSerializer.cpp reaches atoi,
atol and atof through it and compiles on device. The mock supplied only
cstdint, cmath and Stream.h, leaving those undeclared — and since the native
env compiles ConfigSerializer.cpp into every suite via build_src_filter, all
21 suites errored rather than just its own.
Fixing the mock keeps src/ identical to upstream and covers any other source
relying on the same transitive include.
pio test -e native: 297 test cases, 297 succeeded.
The rolling-release body is what the flasher dropdown serves as changelog.
Drop the stale v1.16.0 experimental blurb and describe the current observer
surface: MeshCore 1.17.1, web config, in-channel OTA, and neighbors.
Both channels move together so a node comparing its embedded base against
its own channel's manifest does not read as permanently behind.
Resets the per-base build counter to N=1 on both channels. Nodes still on
v1.17.0.N take the diff_base branch in ESP32Board's OTA comparison, so the
counter going backwards does not strand them.
Picks up upstream MeshCore 1.17.1.
Notable upstream content:
- 1.17.1 version/build-date bump in the example MyMesh headers.
- nRF52: combine radio entropy with CC310 RNG.
- Companion FEM prefs: load/save of fem_ properties commented out until they
can be set from the client.
- Scoped reply routing: replies no longer dropped when flood.max.unscoped is
low (RoutingPolicy + unit tests).
- nRF52 unused-pin sweep (T1, T-Echo Lite, MeshPocket).
No conflicts.
Brings in the external FEM gain preferences (fem_txgain, PR #3137 plus the
companion-side port), the AGC reset rxgain fix, the LR2021 preamble/IRQ
timeout logic, and assorted variant fixes (T096, T-Echo Card TCXO, promicro
pinmap, minewsemi, R1 Neo).
Conflict resolutions:
- SH1106Display: both sides fixed T-Beam Supreme startup independently. Kept
our _initialized guard and DISPLAY_ADDRESS_ALT override, took upstream's
SA0-pair fallback and its unconditional display.begin() so the frame buffer
is allocated even when no panel answers.
- MyMesh/SensorMesh/CommonCLI: took upstream's fem_txgain default and wiring,
kept our comments and the observer-side prefs layout.
Also fixes CustomLLCC68Wrapper, which upstream missed when sx126xResetAGC
gained its rx_boost_gain parameter. No variant builds that wrapper today, so
neither tree failed to compile.
SPI pins that are set to -1 cause OOB reads on NRF52. The unused Serial2 pin definitions were removed to avoid potential issues with the Uart framework.
Setting SPI pins to values that are OOB of the g_ADigitalPinMap[] array causes OOB reads. This variant.h has pin 0 as 0xFF which passes through as NRFX_SPIM_PIN_NOT_USED.
Documents why the largest allocatable block on a non-PSRAM observer walks down in
16 KiB steps at every TLS reconnect, and what to do about it.
The framework builds mbedTLS with the symmetric buffer configuration
(CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN 16384), so each broker slot holds two 16 KiB
record buffers in internal DRAM and two slots cost 64 KiB on a board with about
80 KiB free. Confirmed against the sdkconfig the Heltec env actually links, not
the IDF 5.3 libs package that belongs to the esp32c6 env.
Rebuilding only the mbedTLS archives with an asymmetric outbound buffer of 4 KiB
is ABI-safe, because in_buf/out_buf are heap pointers sized in ssl_setup() and no
public struct embeds a CONTENT_LEN-sized array. Dropping KEEP_PEER_CERTIFICATE is
not, since it changes mbedtls_ssl_session layout, so it is excluded.
Also records two traps worth not rediscovering: esp32-arduino-lib-builder's
release/v4.4 clones its dependencies at master and no longer resolves, and
pointing platform_packages at a file:// framework copy overwrites the shared
package for every other project on the machine.
Introduced consistent preferences for external LoRa FEM RX and TX gain settings in NodePrefs. Updated companion MyMesh to apply these settings during initialization and transmission. Added unit tests to verify the round-trip serialization of these new preferences.
agessaman