A plain `git rebase` refuses to run when the checkout has unstaged changes, so
the retry path could fail instead of retrying. The beta workflow stages a scoped
subset on purpose (production's files must never ride along), which leaves the
rest of the checkout dirty and hits this exactly; keep all three flasher pushes
identical so it cannot resurface here either. Unstaged content stays unstaged,
so nothing extra gets published.
build-observer-firmwares.yml and sync-flasher-content.yml shared the
`flasher-publish` concurrency group. A mixed code+docs commit queues both, and
GitHub cancels any run left *pending* in an occupied group — cancel-in-progress:
false only shields runs already in progress. On 8fc96f68 the build lost that
race and was cancelled in 1s with zero jobs, so nothing was published.
Give each workflow its own group, and make the flasher push tolerate the
concurrent writes the shared group was guarding against: rebase onto the
sibling's commit and retry, up to 3 attempts. Both workflows generate
byte-identical docs and changelog content, so the loser's commit normally
rebases away to nothing and the push reports up-to-date.
Stamp FIRMWARE_BUILD_NUMBER into FIRMWARE_VERSION_STRING (the output
filename), so production assets become <env>-v1.16.0.N-<hash>.bin and the
web flasher's Version dropdown (parsed from the asset name by the /releases
Worker) shows the true published build instead of the bare base version.
The build-number suffix is computed once, up front, and now feeds BOTH the
filename and the embedded -DFIRMWARE_VERSION (byte-for-byte unchanged, it
already carried .N). Local builds without the counter still produce
3-segment names, so OTA and every filename parser stay backward-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement SNMP functionality in MyMesh to provide radio diagnostics
and statistics updates. This includes a new method for formatting
radio diagnostic replies and periodic updates to the SNMP agent
with radio stats, enhancing monitoring capabilities for users
with SNMP enabled.
(cherry picked from commit fbe608d6c0)
Add functionality to schedule deferred OTA updates in the MyMesh class.
This includes a new member variable to track the update time and a method
to initiate the update process, allowing for a smoother OTA experience
by managing timing and state transitions effectively.
Implement logic to determine the maximum number of active MQTT slots
based on available memory. This change allows the system to warn users
when attempting to connect more slots than supported by the hardware,
improving user experience and preventing connection issues. The logic
is centralized in the new getMaxActiveSlots() method, ensuring
consistency across the codebase.
Enhance the documentation for the `start ota` command to clarify its
behavior when connected to a Wi-Fi network versus when using the
`start ota ap` option. This provides users with better guidance on
how to initiate OTA updates under different network conditions.
(cherry picked from commit 2019a8c0ca)
Added functionality for neighbor discovery, including the ability to
store and manage neighbor information. This includes methods for
sending discovery requests, handling responses, and updating the
neighbor table. The changes enhance the mesh networking capabilities
by allowing devices to discover and interact with nearby peers more
effectively. Updated relevant structures and methods in MyMesh.cpp
and MyMesh.h to support these features.
The flasher's versions become feed-driven (/releases on the firmware
proxy Worker): config.json's observer entries turn into github release
defs with no embedded filenames, so update-firmware.py would find zero
matches and exit nonzero — remove the call before the config conversion
lands. Changelog notes now ride the release body via a non-fatal
'gh release edit --notes-file firmware-notes.html' in the publish step.
MUST land before the flasher config.json conversion, or the next build
goes red at Update Flasher Config.
Workflow-only commit: pushing this triggers no build.
The rename to observer-firmware is verified: the rename's own push event
ran both workflows on the new name (branch creation bypasses path
filters), sync-flasher-content pushed to the flasher repo, and the
production publish completed on observer-firmware. The transition dual
trigger is no longer needed.
Also update the MQTT_INTERNALS.md prose reference to the old branch name.
Workflow+docs-only commit: pushing this triggers no build.
Flip gen-slim-manifests.py to its new --bin-dir mode (flasher repo PR #1):
the slim OTA manifests are now generated from out/ — the assets actually
uploaded to the release — instead of the config.json entries, which the
/releases feed migration will convert to github release defs. STATIC_PATH
joins the workflow env; it must stay consistent with config.json's
staticPath.
Also add observer-firmware alongside mqtt-bridge-implementation-flex in
both push triggers, ahead of renaming the branch: GitHub does not rewrite
branch filters on rename, and a rename with only the old name in the
filter silently stops production publishing. The old name is dropped once
a publish is verified on the new one.
Workflow-only commit (.github/** is in paths-ignore): pushing this
triggers no build.
Introduced a new MQTT preset for the meshtexas connection, allowing
users to connect to mqtt.meshtexas.org with JWT authentication over WSS.
This addition increases the number of built-in presets to 27, enhancing
the flexibility of the MQTT implementation.
publishStatusToSlot hardcoded the retain flag to true, ignoring the preset's
allow_retain field that the packets and neighbors paths already respect. The
waev preset (allow_retain=false, MeshCore topic style) has therefore been
publishing retained status to a broker that rejects the retain flag; the same
would now apply to meshrank, which just gained status publishing. Use
slot.preset->allow_retain, defaulting to true for custom slots to preserve
their existing behavior.
MeshRank slots now publish every message type (status/packets/raw/neighbors)
under meshrank/uplink/{token}/{device}/, matching the standard MeshCore topic
layout, instead of packets only. The four duplicated topic-suffix ternaries are
folded into messageTypeSuffix().
Fix the discover.neighbors -> discover.scopes race: a scopes request issued
while a zero-hop discovery is still collecting responses (from discover.neighbors
or the periodic timer) now queues behind that 60-second window and runs against
the refreshed table, reporting the wait in its reply. A queued one-shot request
is no longer cancelled by `mqtt.neighbors off`; only the periodic timer's own
refresh is. Precondition checks are shared via neighborDiscoverReady() so queuing
cannot report OK for a request that will fail once the window closes.
Add the periodic-neighbors schedule to `get mqtt.status` as a trailing
nbr:<next>/<last> field (time to next publish + last result). The mesh owns the
timer and reports a summary to the bridge across the core boundary via atomics.
Also clamp the running offset in formatMqttStatusReply: snprintf returns the
untruncated length, so a full 6-slot line could push the offset past the buffer
and underflow the remaining size on the next append. Reachable today with real
preset-name combinations at the 160-byte reply size.
Updated the MQTT neighbors functionality to include a two-stage
refresh process for periodic publishing. The first stage performs
a 60-second zero-hop neighbor refresh, followed by querying the
refreshed table for scopes before publishing. This change improves
the accuracy of neighbor data and ensures timely updates in the
network. Documentation has been updated to reflect these changes.
Adjusted the valid range for the `mqtt.neighbors.interval` setting from
12-8760 hours to 12-336 hours to reflect new constraints. Updated
related documentation and CLI commands to ensure consistency across
the codebase. This change improves clarity and prevents potential
misconfigurations in neighbor publishing intervals.
Credit: @yellowcooln (PR #12). Adds Wireless Tracker v1.1 board
def and MQTT observer envs for v1.1/v2; gate FEM control so v1.1
builds without KCT8103L pins.
The 30s "MQTT: Memory" line was useful during the outbox/sync-publish
investigation but is spam for production. Gate the periodic logMemoryStatus()
call in the MQTT task loop behind MQTT_MEMORY_DEBUG (a dedicated diagnostics
flag, not enabled by plain MQTT_DEBUG or production builds) and stop the
heltec_v3 variant from force-enabling MQTT_MEMORY_DEBUG on its observer_mqtt
envs (now commented out to match heltec_v4). logMemoryStatus() itself is kept
intact for opt-in debugging.
Expose the same data on demand via a new `get mqtt.stats` CLI command backed by
MQTTBridge::formatMqttStatsReply(): free/max heap, queue depth, outbox total,
and per-slot publish ok/err counts (1-based, matching the msgs: line). Fits the
160-byte reply buffer at 6 slots; returns "(bridge not running)" when down.
The esp-mqtt task drains only one QUEUED outbox item per loop iteration, and
each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on esp_transport_poll_read.
With little inbound traffic that caps throughput at ~1 message/second per
connection, so even a light packet rate (~1.2/s) outruns the drain: the outbox
pins at its cap and ~20-30% of QoS0 packets are dropped as backpressure. The
poll timeout is a compile-time constant baked into the precompiled esp-mqtt lib,
so the async drain rate cannot be raised on the Arduino/IDF 4.4 toolchain.
Route QoS0 packet publishes through esp_mqtt_client_publish() (async=false) so
they write straight to the socket, bypassing the outbox drain entirely — QoS0 no
longer touches the outbox. QoS1 status keeps the async/outbox + retransmit path.
The esp-mqtt task releases its API lock before the poll, so a synchronous publish
from the (Core-0, prio-1) MQTT task acquires the lock and writes immediately; a
stalled socket blocks only that task (mesh RX on Core 1 and the WiFi/TCP stack
are unaffected), bounded by a new setNetworkTimeout() lowered to 2500ms so a
first stall fails fast and flips the slot to disconnected.
The outbox cap from the previous commit stays as a dormant safety net. Retools
the MQTT_DEBUG diagnostic from outbox size/drops (now always ~0) to per-slot
publish ok/err counts, the live signal for delivery health, with 1-based slot
numbering to match the status line.
QoS0 packet/raw publishes are forced into the esp-mqtt outbox (store=true,
async) so packet topics keep flowing, but the outbox has no size bound of its
own — esp-mqtt frees entries only on send-ack or ~30s expiry. On a stalled or
slow uplink (socket still "connected") QoS0 frames accumulate on internal heap
without limit, driving the heap exhaustion/fragmentation seen in the field.
Cap the outbox at the application level: PsychicMqttClient::setOutboxLimit()
records a per-client byte cap, and publish() drops a QoS0 message (returns -2)
when esp_mqtt_client_get_outbox_size() is already at/over the cap, before
enqueuing. The bridge's existing processPacketQueue retry/drop path handles the
-2 as backpressure. Caps: 16 KiB PSRAM / 8 KiB non-PSRAM (outbox lives on
internal heap, so non-PSRAM is the fragmentation-sensitive case).
Portable across IDF 4.4 and 5 via esp_mqtt_client_get_outbox_size(); esp-mqtt's
own outbox.limit config is not used (its enqueue path does not reliably enforce
it for QoS0, and the app-level guard fires before enqueue regardless).
Adds getOutboxSize()/getOutboxLimit()/getOutboxDrops() and surfaces per-slot
outbox size/cap/drops via a throttled logMemoryStatus() in the MQTT task loop
(MQTT_DEBUG-gated) to confirm the bound on-target.
Reverts merge ae045395 (feat/flex-ipv6, PR #25). Enabling IPv6 joins the
device into IPv6 multicast/ND processing; on multicast-heavy LANs (e.g.
with a Thread/Matter border router advertising a ULA prefix) inbound
bursts land in dynamic WiFi RX buffers in internal heap. Measured on a
Station G2 with 4 WSS brokers: min-free floor dropped 36 KB -> 18 KB,
largest free block pinned below the 16 KB publish threshold for minutes
at a time, 523 dropped publishes in 15 minutes. With IPv6 disabled the
floor and max-alloc recovered and publish skips stopped.
The feature only fed the wifi.status display line — no transport uses
IPv6 (all brokers connect over IPv4), so the fleet risk (network-
dependent degradation on unknown home LANs) buys nothing. Revisit after
the Arduino core 3.x / IDF 5.x move if IPv6 transport is ever needed;
the branch remains at feat/flex-ipv6.
Kept from the merge: the wifi.status uptime append now computes the
actual remaining space in the 160-byte reply buffer instead of assuming
a hardcoded 128, fixing a latent overflow of the snprintf bound.
- Introduced commands for periodic neighbors publishing: `get mqtt.neighbors`, `set mqtt.neighbors on|off`, `get mqtt.neighbors.interval`, and `set mqtt.neighbors.interval <hours>`.
- Updated MQTT topics to include a new neighbors topic for cached zero-hop repeater neighbors.
- Added JSON message format for neighbors, including fields for SNR, last-heard age, and status.
- Enhanced CLI documentation to reflect new commands and settings.
- Implemented neighbor discovery logic in MyMesh class, including handling responses and publishing neighbor data.
- Adjusted MQTTBridge to manage neighbors JSON buffer and publish logic.
This update enhances the MQTT implementation by allowing devices to share neighbor information, improving network awareness and connectivity.
The token's exp claim and the renewal schedule derive from the same
value, so the flat 60 s RENEWAL_BUFFER was the entire margin between
proactive re-auth and the broker enforcing exp on the live session -
one failed renewal attempt (60 s throttle) or a minute of clock skew
lost the race, seen as clean-FIN disconnects (tls=0x8008) on the waev
preset, whose 55-minute tokens are the only ones short enough to hit
enforcement. Buffer is now lifetime/10 clamped to [60 s, 300 s], and
the disconnect-now threshold uses the same value so every renewal is
a proactive reconnect on the device's schedule; waev re-auths 10 min
before its real 60-minute TTL with ~5 retry windows.
Document why waev's preset claims 3300 s against the broker's real
3600 s TTL: the 5-minute claim-side gap protects token acceptance
against fast device clocks, which the renewal buffer cannot do.
A CONNACK alone reset the backoff ladder, so a flapping broker (accepts
then drops within seconds) retried at the 10 s rung forever - each
attempt a full ~40 KB TLS session alloc/free on internal heap, a known
fragmentation driver. The ladder now clears only after the connection
survives 2 minutes (at least one 75 s keepalive round-trip); flapping
endpoints degrade to the 300 s rung and then the existing 30-minute
circuit-breaker probes, and recover automatically once stable.
The precompiled IDF 4.4 WebSocket transport (libtcp_transport.a) has an
off-by-one in ws_connect(): when a wss:// endpoint answers the upgrade
request with >=1024 bytes of HTTP response before the blank-line
terminator (typical of a down broker behind a proxy serving a large
error page), it writes a NUL one byte past the 1024-byte ws->buffer.
Heap poisoning catches the clobbered tail canary (0xbaad5678 ->
0xbaad5600) only when the block is freed in ws_destroy() during
esp_mqtt_client_destroy() - i.e. MQTTBridge::end() - so a single down
broker made every deferred 'ota update' panic and reboot at teardown,
before the download started. Decoded from a Heltec V3 crash backtrace
on v1.16.0.11; line numbers match ESP-IDF release/v4.4 exactly.
The transport code ships precompiled, so patch at link time instead:
[esp32_base] wraps esp_transport_ws_init and the wrapper swaps the
fresh buffer for a (WS_BUFFER_SIZE + 1)-byte allocation, making the
out-of-bounds index land on owned memory. The oversized handshake then
fails cleanly instead of corrupting the heap. Pass-through on IDF 5.x,
where upstream already fixed it; delete with the Arduino core 3.x move.
Verified: wrap resolves from ESP32WsTransportFix.cpp.o in the observer
firmware.map; observer, room-server observer and plain repeater ESP32
targets build. RAK_4631_repeater failure is pre-existing (reproduced
on the merge base without these changes).
esp-mqtt's default message_retransmit_timeout is 1000 ms: any unacked QoS 1 PUBLISH is resent (byte-identical, DUP=1) every second until the PUBACK arrives or the outbox entry expires (30 s). Status messages are the only QoS 1 publishes; on a congested or recovering uplink where broker acks take several seconds, each 5-minute /status was delivered ~6 times, ~1 s apart, as exact copies (same timestamp and stats). Downstream observers flagged excessive_packet_copies and at least one broker treats it as abuse.
Expose message_retransmit_timeout via PsychicMqttClient and set it to 15 s in optimizeMqttClientConfig: one retry still fits inside the 30 s outbox expiry, preserving at-least-once delivery while capping duplicates at one.
/packets paths are QoS 0 and were never affected.
esp-mqtt's default message_retransmit_timeout is 1000 ms: any unacked QoS 1
PUBLISH is resent (byte-identical, DUP=1) every second until the PUBACK
arrives or the outbox entry expires (30 s). Status messages are the only
QoS 1 publishes; on a congested or recovering uplink where broker acks take
several seconds, each 5-minute /status was delivered ~6 times, ~1 s apart,
as exact copies (same timestamp and stats). Downstream observers flagged
excessive_packet_copies and at least one broker treats it as abuse.
Expose message_retransmit_timeout via PsychicMqttClient and set it to 15 s
in optimizeMqttClientConfig: one retry still fits inside the 30 s outbox
expiry, preserving at-least-once delivery while capping duplicates at one.
/packets paths are QoS 0 and were never affected.
Merges three stacked PRs in one push so the rolling release only builds
the final state:
#20 — Complete observer settings migration and upstream-seam cleanup
- Implement the missing one-time /com_prefs -> /mqtt_prefs observer
settings migration (silent data loss on upgrade before this).
- Harden /mqtt_prefs into a versioned format (magic + version +
payload_len header); legacy headerless layouts detected by size and
migrated once. Drop vestigial _legacy_* fields.
- Deployed flex layout pinned by static_assert (Legacy6SlotMQTTPrefs ==
2904 bytes); verified with the host migration harness and on-device
(Heltec V4: presets, WiFi, and origin survive upgrade and reboot).
#21 — Restore upstream duty-cycle enforcement and CAD
- Restore Dispatcher/StaticPoolPacketManager verbatim from upstream/dev
(token-bucket duty cycle, CAD interface); re-apply the MQTT radio
watchdog as pure additions behind WITH_MQTT_BRIDGE.
- Restore cad_enabled and radio_fem_rxgain persistence at upstream's
/com_prefs offsets (byte-parity with upstream, tail 3 -> 5 bytes).
- Add RxReservePacketManager for observer builds: priority-aware shed +
30s stale expiry so heavy throttling can't park the packet pool and
make the node un-administrable.
- Device-confirmed on Heltec V4.2: throttling holds at set dutycycle 1
on a busy mesh, CAD persists across boots, capture continues at full
rate under throttle.
#22 — Drive LoRa FEM RX gain from radio_fem_rxgain
- MainBoard gains setLoRaFemLnaEnabled/canControlLoRaFemLna/
isLoRaFemLnaEnabled; wired for heltec_v4, heltec_t096, and
tracker_v2. radio.fem.rxgain CLI gated on board capability.
- Persisted pref applied at app startup; defaults ON (upstream parity),
so LNA-capable boards gain RX boost on upgrade.
- Guard path device-confirmed on V4.2 (GC1109, PA-only: reports
unsupported). LNA toggle on V4.3/KCT8103L pending hardware.
Multi-hour soak on the full stack: no memory issues or regressions.
Duty-cycle throttling, CAD-under-load, capture decoupling, remote admin under
throttle, and radio-watchdog fire/recover all device-confirmed on a Heltec
V4.2 (busy mesh + off-frequency bench). Notes the watchdog observability
gotchas found during testing: silent on release builds (check stats-radio-diag
err_flags bit 8) and armed only after first radio activity.
Device testing at 'set dutycycle 1' on a busy mesh showed the node becoming
un-administrable within ~2 minutes: the shed policy dropped its own CLI
responses along with repeats, and parked retransmissions (which never expire)
absorbed every budget refill.
RxReservePacketManager now sheds by priority below the RX reserve — only
pri > 1 outbound (multi-hop flood repeats, adverts, trace) is refused, so the
node's own responses/ACKs (pri 0) and login/PATH replies (pri 1) still queue;
below an emergency floor (reserve/2) everything is shed to protect capture.
Queued packets untransmitted 30 s past their scheduled time are expired at
dequeue via a pointer-keyed age table (the pool is a fixed set of packets, so
pool_size slots cover every key). Under normal load the queue drains in
milliseconds and neither policy triggers.
Load-testing the restored token bucket at 'set dutycycle 1' showed MQTT
capture dropping to exactly the TX rate. Queued retransmissions hold static-
pool packets with no expiry, so throttling parks the whole pool in the send
queue; Dispatcher::checkRecv() then discards received packets before logRx()
ever feeds the bridge — each completed TX frees exactly one packet for
exactly one more RX.
Observer builds now use RxReservePacketManager (fork-owned header): once the
free pool drops below a quarter of the pool, outbound packets are refused and
freed, so RX allocation and MQTT capture continue at full rate while the node
sheds repeat load it has no TX budget for anyway. Non-observer builds keep
upstream pool behavior via the same factory; StaticPoolPacketManager stays
byte-identical to upstream.
companion_radio has its own NodePrefs without radio_fem_rxgain (matching
upstream, which doesn't wire companion either), so the startup call added in
e905451d broke every companion build:
MyMesh.cpp:973: error: 'struct NodePrefs' has no member named 'radio_fem_rxgain'
It went unnoticed behind the pre-existing companion build failures.
Also port upstream's FEM LNA overrides for heltec_t096 and heltec_tracker_v2
verbatim (board overrides + isLNAEnabled/const getters), so all three boards
upstream wires now match instead of reporting unsupported. Both variant files
compile; those targets still fail overall on the pre-existing non-observer
MQTT-source build-config issue (unchanged with this work stashed).
performChannelScan was restored as protected non-virtual but upstream declares
it public virtual — match upstream verbatim so the hunk disappears from the
merge surface. Also replace the bare 'extra > 5' tail threshold with
COM_PREFS_TAIL_BYTES, tied by comment to the trailing writes in savePrefs(),
so the next upstream field append updates one named constant.
The unknown-version path kept defaults at boot but any later savePrefs()
(every CLI set command) rewrote /mqtt_prefs as v1 with defaults, destroying
the newer config after a firmware downgrade. Latch _mqtt_prefs_hold when an
unsupported version is seen and refuse to write while it is set — checked
before the NRF52/STM32 open path, which deletes the file first.
Also pin the frozen legacy /mqtt_prefs layouts (472/1464/2904 bytes + 8-byte
header) with static_asserts so every target build re-verifies the deployed
fleet's file offsets, and null-check _obs in AlertReporter::onLoop.
agessaman