Implement logic to determine the maximum number of active MQTT slots
based on available memory. This change allows the system to warn users
when attempting to connect more slots than supported by the hardware,
improving user experience and preventing connection issues. The logic
is centralized in the new getMaxActiveSlots() method, ensuring
consistency across the codebase.
Enhance the documentation for the `start ota` command to clarify its
behavior when connected to a Wi-Fi network versus when using the
`start ota ap` option. This provides users with better guidance on
how to initiate OTA updates under different network conditions.
(cherry picked from commit 2019a8c0ca)
Introduced a new MQTT preset for the meshtexas connection, allowing
users to connect to mqtt.meshtexas.org with JWT authentication over WSS.
This addition increases the number of built-in presets to 27, enhancing
the flexibility of the MQTT implementation.
publishStatusToSlot hardcoded the retain flag to true, ignoring the preset's
allow_retain field that the packets and neighbors paths already respect. The
waev preset (allow_retain=false, MeshCore topic style) has therefore been
publishing retained status to a broker that rejects the retain flag; the same
would now apply to meshrank, which just gained status publishing. Use
slot.preset->allow_retain, defaulting to true for custom slots to preserve
their existing behavior.
MeshRank slots now publish every message type (status/packets/raw/neighbors)
under meshrank/uplink/{token}/{device}/, matching the standard MeshCore topic
layout, instead of packets only. The four duplicated topic-suffix ternaries are
folded into messageTypeSuffix().
Fix the discover.neighbors -> discover.scopes race: a scopes request issued
while a zero-hop discovery is still collecting responses (from discover.neighbors
or the periodic timer) now queues behind that 60-second window and runs against
the refreshed table, reporting the wait in its reply. A queued one-shot request
is no longer cancelled by `mqtt.neighbors off`; only the periodic timer's own
refresh is. Precondition checks are shared via neighborDiscoverReady() so queuing
cannot report OK for a request that will fail once the window closes.
Add the periodic-neighbors schedule to `get mqtt.status` as a trailing
nbr:<next>/<last> field (time to next publish + last result). The mesh owns the
timer and reports a summary to the bridge across the core boundary via atomics.
Also clamp the running offset in formatMqttStatusReply: snprintf returns the
untruncated length, so a full 6-slot line could push the offset past the buffer
and underflow the remaining size on the next append. Reachable today with real
preset-name combinations at the 160-byte reply size.
Adjusted the valid range for the `mqtt.neighbors.interval` setting from
12-8760 hours to 12-336 hours to reflect new constraints. Updated
related documentation and CLI commands to ensure consistency across
the codebase. This change improves clarity and prevents potential
misconfigurations in neighbor publishing intervals.
The 30s "MQTT: Memory" line was useful during the outbox/sync-publish
investigation but is spam for production. Gate the periodic logMemoryStatus()
call in the MQTT task loop behind MQTT_MEMORY_DEBUG (a dedicated diagnostics
flag, not enabled by plain MQTT_DEBUG or production builds) and stop the
heltec_v3 variant from force-enabling MQTT_MEMORY_DEBUG on its observer_mqtt
envs (now commented out to match heltec_v4). logMemoryStatus() itself is kept
intact for opt-in debugging.
Expose the same data on demand via a new `get mqtt.stats` CLI command backed by
MQTTBridge::formatMqttStatsReply(): free/max heap, queue depth, outbox total,
and per-slot publish ok/err counts (1-based, matching the msgs: line). Fits the
160-byte reply buffer at 6 slots; returns "(bridge not running)" when down.
The esp-mqtt task drains only one QUEUED outbox item per loop iteration, and
each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on esp_transport_poll_read.
With little inbound traffic that caps throughput at ~1 message/second per
connection, so even a light packet rate (~1.2/s) outruns the drain: the outbox
pins at its cap and ~20-30% of QoS0 packets are dropped as backpressure. The
poll timeout is a compile-time constant baked into the precompiled esp-mqtt lib,
so the async drain rate cannot be raised on the Arduino/IDF 4.4 toolchain.
Route QoS0 packet publishes through esp_mqtt_client_publish() (async=false) so
they write straight to the socket, bypassing the outbox drain entirely — QoS0 no
longer touches the outbox. QoS1 status keeps the async/outbox + retransmit path.
The esp-mqtt task releases its API lock before the poll, so a synchronous publish
from the (Core-0, prio-1) MQTT task acquires the lock and writes immediately; a
stalled socket blocks only that task (mesh RX on Core 1 and the WiFi/TCP stack
are unaffected), bounded by a new setNetworkTimeout() lowered to 2500ms so a
first stall fails fast and flips the slot to disconnected.
The outbox cap from the previous commit stays as a dormant safety net. Retools
the MQTT_DEBUG diagnostic from outbox size/drops (now always ~0) to per-slot
publish ok/err counts, the live signal for delivery health, with 1-based slot
numbering to match the status line.
QoS0 packet/raw publishes are forced into the esp-mqtt outbox (store=true,
async) so packet topics keep flowing, but the outbox has no size bound of its
own — esp-mqtt frees entries only on send-ack or ~30s expiry. On a stalled or
slow uplink (socket still "connected") QoS0 frames accumulate on internal heap
without limit, driving the heap exhaustion/fragmentation seen in the field.
Cap the outbox at the application level: PsychicMqttClient::setOutboxLimit()
records a per-client byte cap, and publish() drops a QoS0 message (returns -2)
when esp_mqtt_client_get_outbox_size() is already at/over the cap, before
enqueuing. The bridge's existing processPacketQueue retry/drop path handles the
-2 as backpressure. Caps: 16 KiB PSRAM / 8 KiB non-PSRAM (outbox lives on
internal heap, so non-PSRAM is the fragmentation-sensitive case).
Portable across IDF 4.4 and 5 via esp_mqtt_client_get_outbox_size(); esp-mqtt's
own outbox.limit config is not used (its enqueue path does not reliably enforce
it for QoS0, and the app-level guard fires before enqueue regardless).
Adds getOutboxSize()/getOutboxLimit()/getOutboxDrops() and surfaces per-slot
outbox size/cap/drops via a throttled logMemoryStatus() in the MQTT task loop
(MQTT_DEBUG-gated) to confirm the bound on-target.
Reverts merge ae045395 (feat/flex-ipv6, PR #25). Enabling IPv6 joins the
device into IPv6 multicast/ND processing; on multicast-heavy LANs (e.g.
with a Thread/Matter border router advertising a ULA prefix) inbound
bursts land in dynamic WiFi RX buffers in internal heap. Measured on a
Station G2 with 4 WSS brokers: min-free floor dropped 36 KB -> 18 KB,
largest free block pinned below the 16 KB publish threshold for minutes
at a time, 523 dropped publishes in 15 minutes. With IPv6 disabled the
floor and max-alloc recovered and publish skips stopped.
The feature only fed the wifi.status display line — no transport uses
IPv6 (all brokers connect over IPv4), so the fleet risk (network-
dependent degradation on unknown home LANs) buys nothing. Revisit after
the Arduino core 3.x / IDF 5.x move if IPv6 transport is ever needed;
the branch remains at feat/flex-ipv6.
Kept from the merge: the wifi.status uptime append now computes the
actual remaining space in the 160-byte reply buffer instead of assuming
a hardcoded 128, fixing a latent overflow of the snprintf bound.
- Introduced commands for periodic neighbors publishing: `get mqtt.neighbors`, `set mqtt.neighbors on|off`, `get mqtt.neighbors.interval`, and `set mqtt.neighbors.interval <hours>`.
- Updated MQTT topics to include a new neighbors topic for cached zero-hop repeater neighbors.
- Added JSON message format for neighbors, including fields for SNR, last-heard age, and status.
- Enhanced CLI documentation to reflect new commands and settings.
- Implemented neighbor discovery logic in MyMesh class, including handling responses and publishing neighbor data.
- Adjusted MQTTBridge to manage neighbors JSON buffer and publish logic.
This update enhances the MQTT implementation by allowing devices to share neighbor information, improving network awareness and connectivity.
The token's exp claim and the renewal schedule derive from the same
value, so the flat 60 s RENEWAL_BUFFER was the entire margin between
proactive re-auth and the broker enforcing exp on the live session -
one failed renewal attempt (60 s throttle) or a minute of clock skew
lost the race, seen as clean-FIN disconnects (tls=0x8008) on the waev
preset, whose 55-minute tokens are the only ones short enough to hit
enforcement. Buffer is now lifetime/10 clamped to [60 s, 300 s], and
the disconnect-now threshold uses the same value so every renewal is
a proactive reconnect on the device's schedule; waev re-auths 10 min
before its real 60-minute TTL with ~5 retry windows.
Document why waev's preset claims 3300 s against the broker's real
3600 s TTL: the 5-minute claim-side gap protects token acceptance
against fast device clocks, which the renewal buffer cannot do.
A CONNACK alone reset the backoff ladder, so a flapping broker (accepts
then drops within seconds) retried at the 10 s rung forever - each
attempt a full ~40 KB TLS session alloc/free on internal heap, a known
fragmentation driver. The ladder now clears only after the connection
survives 2 minutes (at least one 75 s keepalive round-trip); flapping
endpoints degrade to the 300 s rung and then the existing 30-minute
circuit-breaker probes, and recover automatically once stable.
The precompiled IDF 4.4 WebSocket transport (libtcp_transport.a) has an
off-by-one in ws_connect(): when a wss:// endpoint answers the upgrade
request with >=1024 bytes of HTTP response before the blank-line
terminator (typical of a down broker behind a proxy serving a large
error page), it writes a NUL one byte past the 1024-byte ws->buffer.
Heap poisoning catches the clobbered tail canary (0xbaad5678 ->
0xbaad5600) only when the block is freed in ws_destroy() during
esp_mqtt_client_destroy() - i.e. MQTTBridge::end() - so a single down
broker made every deferred 'ota update' panic and reboot at teardown,
before the download started. Decoded from a Heltec V3 crash backtrace
on v1.16.0.11; line numbers match ESP-IDF release/v4.4 exactly.
The transport code ships precompiled, so patch at link time instead:
[esp32_base] wraps esp_transport_ws_init and the wrapper swaps the
fresh buffer for a (WS_BUFFER_SIZE + 1)-byte allocation, making the
out-of-bounds index land on owned memory. The oversized handshake then
fails cleanly instead of corrupting the heap. Pass-through on IDF 5.x,
where upstream already fixed it; delete with the Arduino core 3.x move.
Verified: wrap resolves from ESP32WsTransportFix.cpp.o in the observer
firmware.map; observer, room-server observer and plain repeater ESP32
targets build. RAK_4631_repeater failure is pre-existing (reproduced
on the merge base without these changes).
esp-mqtt's default message_retransmit_timeout is 1000 ms: any unacked QoS 1
PUBLISH is resent (byte-identical, DUP=1) every second until the PUBACK
arrives or the outbox entry expires (30 s). Status messages are the only
QoS 1 publishes; on a congested or recovering uplink where broker acks take
several seconds, each 5-minute /status was delivered ~6 times, ~1 s apart,
as exact copies (same timestamp and stats). Downstream observers flagged
excessive_packet_copies and at least one broker treats it as abuse.
Expose message_retransmit_timeout via PsychicMqttClient and set it to 15 s
in optimizeMqttClientConfig: one retry still fits inside the 30 s outbox
expiry, preserving at-least-once delivery while capping duplicates at one.
/packets paths are QoS 0 and were never affected.
Device testing at 'set dutycycle 1' on a busy mesh showed the node becoming
un-administrable within ~2 minutes: the shed policy dropped its own CLI
responses along with repeats, and parked retransmissions (which never expire)
absorbed every budget refill.
RxReservePacketManager now sheds by priority below the RX reserve — only
pri > 1 outbound (multi-hop flood repeats, adverts, trace) is refused, so the
node's own responses/ACKs (pri 0) and login/PATH replies (pri 1) still queue;
below an emergency floor (reserve/2) everything is shed to protect capture.
Queued packets untransmitted 30 s past their scheduled time are expired at
dequeue via a pointer-keyed age table (the pool is a fixed set of packets, so
pool_size slots cover every key). Under normal load the queue drains in
milliseconds and neither policy triggers.
Load-testing the restored token bucket at 'set dutycycle 1' showed MQTT
capture dropping to exactly the TX rate. Queued retransmissions hold static-
pool packets with no expiry, so throttling parks the whole pool in the send
queue; Dispatcher::checkRecv() then discards received packets before logRx()
ever feeds the bridge — each completed TX frees exactly one packet for
exactly one more RX.
Observer builds now use RxReservePacketManager (fork-owned header): once the
free pool drops below a quarter of the pool, outbound packets are refused and
freed, so RX allocation and MQTT capture continue at full rate while the node
sheds repeat load it has no TX budget for anyway. Non-observer builds keep
upstream pool behavior via the same factory; StaticPoolPacketManager stays
byte-identical to upstream.
performChannelScan was restored as protected non-virtual but upstream declares
it public virtual — match upstream verbatim so the hunk disappears from the
merge surface. Also replace the bare 'extra > 5' tail threshold with
COM_PREFS_TAIL_BYTES, tied by comment to the trailing writes in savePrefs(),
so the next upstream field append updates one named constant.
The unknown-version path kept defaults at boot but any later savePrefs()
(every CLI set command) rewrote /mqtt_prefs as v1 with defaults, destroying
the newer config after a firmware downgrade. Latch _mqtt_prefs_hold when an
unsupported version is seen and refuse to write while it is set — checked
before the NRF52/STM32 open path, which deletes the file first.
Also pin the frozen legacy /mqtt_prefs layouts (472/1464/2904 bytes + 8-byte
header) with static_asserts so every target build re-verifies the deployed
fleet's file offsets, and null-check _obs in AlertReporter::onLoop.
Completes the FEM RX-gain restoration begun in the CAD/prefs change, which
persisted radio_fem_rxgain but didn't yet drive the hardware. Also dropped
by the 22eb9b87 revert; restored to match upstream.
- MainBoard: setLoRaFemLnaEnabled()/canControlLoRaFemLna()/isLoRaFemLnaEnabled()
virtuals (default: can't control — non-FEM boards report unsupported)
- heltec_v4: board overrides driving loRaFEMControl; LoRaFEMControl gains the
isLNAEnabled() getter (it already tracked lna_enabled and drove the FEM)
- CLI: `set radio.fem.rxgain on/off` / `get radio.fem.rxgain` (guarded by
canControlLoRaFemLna, so it reports "unsupported" on non-FEM boards)
- app startup applies the persisted pref: board.setLoRaFemLnaEnabled(
_prefs.radio_fem_rxgain), beside setRxBoostedGainMode
Default is ON (upstream), so on FEM boards the LNA is enabled after upgrade —
a real reception behavior change to confirm on hardware. The other FEM
variants (heltec_t096/tower_v2/tracker_v2) need the same small board-override
addition; until then `radio.fem.rxgain` reports unsupported there (no
regression — status quo).
Builds: heltec_v4 repeater-observer + room-observer (FEM board), Heltec_v3
repeater (non-FEM, base virtuals no-op). NEEDS on-device validation on a
Heltec V4.
Continues restoring features dropped by the 22eb9b87 revert. Both were
upstream-tested code, not intentional fork removals.
CAD (hardware Channel Activity Detection / listen-before-talk before TX),
fully restored and functional:
- NodePrefs.cad_enabled + `set cad on/off` / `get cad` CLI (default off)
- RadioLibWrapper: _cad_enabled + setCADEnabled() + the scanChannel()/CAD
branch in isChannelActive() (Phase 1 already restored the Dispatcher hook)
- getCADEnabled() overrides in the repeater/room/sensor apps (return the
pref) and companion (always on, matching upstream)
radio_fem_rxgain: the NodePrefs field + /com_prefs persistence are restored
here at upstream's exact offsets (293 fem, 294 cad), which makes /com_prefs
byte-identical to upstream through the tail. The field is persisted and
defaults on (upstream default), but the per-board LNA *driving* + the
`radio.fem.rxgain` CLI are deferred to the FEM-hardware change (they depend
on board methods and want per-board bench testing).
The new-format /com_prefs tail grows from 3 to 5 bytes; the old-format
detection threshold and the host migration harness are updated accordingly
(all scenarios pass, incl. the non-MQTT-build variant).
Builds: Heltec_v3 repeater, repeater-observer, room-observer. (sensor /
plain-room / companion fail only on the pre-existing Timezone.h include
issue, unrelated to these changes.)
Commit 22eb9b87 ("Revert 'Merge remote-tracking branch origin/dev...'")
reverted an entire upstream merge to escape a bad merge state, dropping
860 lines across 66 files. Among the collateral never reconciled on a
later re-merge was eb4fa032's token-bucket duty-cycle enforcement — the
mechanism that keeps nodes under a configured airtime budget (and EU
868 MHz nodes under the legally-mandated duty cycle). The fork had fallen
back to fixed per-packet spacing (getAirtimeBudgetFactor reverted to 2.0),
losing the windowed enforcement.
This was never an intentional design choice, so restoring it re-aligns the
fork with upstream and REDUCES the merge-conflict surface: Dispatcher.{h,cpp}
now diverge from upstream by watchdog additions only (77 insertions, 0
deletions) instead of rewriting checkSend()/loop().
Restored from upstream: updateTxBudget/tx_budget_ms/duty_cycle_window_ms/
getRemainingTxBudget/getDutyCycleWindowMs and the windowed budget logic in
Dispatcher; getOutboundTotal() and the 0xFFFFFFFF count-all sentinel in
StaticPoolPacketManager; the getOutboundTotal() call in StatsFormatHelper.
Re-applied the fork's MQTT radio-watchdog on top as pure additions
(#ifdef WITH_MQTT_BRIDGE), keeping formatRadioDiag.
Stored airtime_factor settings keep their meaning: fork's t*factor spacing
and upstream's 1/(1+factor) windowed budget yield the same steady-state
duty cycle; upstream additionally allows short bursts within the window.
Phase 2 (CAD / radio_fem_rxgain, which touch NodePrefs persistence and
per-board FEM wiring) is documented in RESTORE_UPSTREAM_NOTES.md, not done
here. Builds: Heltec_v3 observer + plain repeater. NEEDS ON-DEVICE
duty-cycle validation before merge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The /mqtt_prefs format was detected purely by struct size, which is fragile
(size collisions across eras) and forced the vestigial `_legacy_*` fields to
be retained forever to preserve byte offsets. With ~several thousand observer
devices deployed from the rolling `observer-mqtt-latest` flasher, this is the
last safe moment to fix it before the observer-settings split ships.
/mqtt_prefs now leads with an 8-byte MQTTPrefsHeader (magic {0xF5,'M','Q','P'},
version, payload_len) followed by the raw payload. The magic's non-ASCII lead
byte cannot collide with a legacy file (whose payload starts with the
mqtt_origin string), so versioned and headerless files are cleanly separable.
An unrecognized (newer) version leaves the file untouched and falls back to
defaults rather than misreading it.
MQTTPrefs is compacted: the six `_legacy_*` fields are removed. Every deployed
headerless layout — pre-slot (OldMQTTPrefs), 3-slot (ThreeSlotMQTTPrefs), and
the shipped 6-slot flex layout (new Legacy6SlotMQTTPrefs) — is field-copied
into the compact struct and re-saved with the header once, on first boot.
Future fields append to the payload and stay backward compatible.
Verified with a host harness that generates a byte-exact deployed flex
/mqtt_prefs (the flex MQTTPrefs is confirmed identical to Legacy6SlotMQTTPrefs,
2904 bytes) and round-trips it through the new load path, plus pre-slot/3-slot
migration, unknown-version safety, magic/origin collision-safety, and the
combined /com_prefs observer-tail recovery. Observer + room-server + no-MQTT
ESP32 targets build.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The NodePrefs->MQTTPrefs split (2eb41bae) left the promised one-time
migration of the old /com_prefs trailing block unimplemented, so users
upgrading an observer node silently lost SNMP, radio-watchdog, and
fault-alert configuration (alerts reset to off; PSK/hashtag/region wiped).
loadPrefsInt now detects an old-format /com_prefs by its size, skips the
legacy zero-filled MQTT gap (6-slot or 3-slot era), and recovers the
trailing observer block into a LegacyObserverTail (reusing the old
firmware's byte291/292 heuristic and per-field availability guards).
loadMQTTPrefs applies those values when the loaded /mqtt_prefs predates
the appended observer fields, and both files are rewritten once in the
current layout. rx_boosted_gain/flood_max_* are also recovered from the
correct offsets (previously read from inside the old gap and reset).
Verified with a host-side harness that round-trips the real old-firmware
savePrefs (from 2eb41bae^) through the new load path across upgrade,
fresh-install, upstream-format, 3-slot-era, truncated, and legacy-variant
cases, plus a non-MQTT-build variant. Updates the stale migration comments
and documents the CommonCLI_Observer seam in MQTT_IMPLEMENTATION.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Updated MQTT configuration to succeed when port is defined in mqttN.server but not explicitly set in mqttN.port. Updated documentation to specify that when a full
URL with a scheme is provided, the port setting is optional. Added an
example for local development using a plain WebSocket URL, enhancing
clarity for users configuring custom MQTT brokers.
Refactored the handling of observer-related settings by moving them from
NodePrefs to a new MQTTPrefs structure. This change centralizes MQTT,
WiFi, timezone, SNMP, and alert configurations, improving code organization
and maintainability. The new structure allows for better separation of
concerns and prepares the codebase for future enhancements.
Refactored the CommonCLI class to separate observer-related command
handling into CommonCLI_Observer.cpp. This change improves code
organization and maintainability by isolating MQTT, WiFi, and other
observer-specific commands from the main CLI logic.
Improved error handling in the MQTT client to log specific reasons
for connection refusals, including detailed return codes. This change
ensures that users are informed of authentication issues and server
availability problems, enhancing the debugging experience.
Remove unused MQTTMessageBuilder members (getPacketTypeString,
formatTimestamp/Time/Date stubs, JSON_BUFFER_SIZE constant) for a
small flash saving with no behavior change.
Replace the per-byte snprintf("%02X") in bytesToHex with a nibble
lookup table, avoiding a format-string parse up to ~512x per publish
on the MQTT task. Output is byte-for-byte identical uppercase hex.
Implemented new commands for configuring and diagnosing NTP server
settings in the MQTT bridge. Users can now set a custom NTP server
and probe connectivity to configured servers. This enhancement
improves time synchronization reliability for JWT authentication
and provides better diagnostics for NTP connectivity issues.
Implemented functionality to generate and compare partition-table
signatures during OTA updates. This enhancement ensures that the
target build's partition layout matches the device's actual layout,
improving the reliability of OTA updates and preventing issues
related to partition changes.
Updated the otaFromManifestImpl method to enhance the display of
available and current firmware versions. The changes ensure that
the short commit hash is included in the status messages,
providing clearer versioning information during OTA checks.
Improved the otaFromManifestImpl method to handle dry run scenarios
by allowing HTTP fetches without TLS, reducing heap usage on no-PSRAM
boards. This change ensures better compatibility and reliability during
OTA checks while maintaining security for actual updates.
Enhanced the firmware versioning system by appending a build number
suffix when available, allowing for better tracking of published builds.
This change improves the OTA update process by providing clearer
versioning information in the embedded firmware string.
agessaman