Adds Station_G3_ESP32_repeater_observer_mqtt and
Station_G3_ESP32_room_server_observer_mqtt, mirroring the known-good
Station G2 observer envs. The two boards are the same family — ESP32-S3,
qio_opi, PSRAM, 16 MB flash, SH1106 display — so the overlay ports across
unchanged: default_16MB.csv partitions, adafruit-full cert bundle,
MQTT bridge + SNMP, and the same lib_deps pinning.
Verified the G3 partition signature is byte-identical to the G2 observer's,
so the OTA partition-compatibility gate treats them the same. Both envs
build clean (1.67 MB of the 6.25 MB app slot, 28% RAM) and produce the full
CI artifact set via build.sh: .bin, -merged.bin and .partsig. Observer env
discovery now finds 32 envs (was 30) and the ArduinoJson pin check covers
both new sections.
Docs: added the G3 build commands and the two partition-table rows to
MQTT_IMPLEMENTATION.md.
The [Xiao_S3_WIO] base block lost its sensor wiring when upstream
3dd6dc02 ("xiao_s3: use environment sensor manager and add sensor
role") was resolved to our side in a later merge. Only the .ini half
was dropped -- target.h/target.cpp kept the EnvironmentSensorManager,
so builds still compiled helpers/sensors with an empty SENSOR_TABLE:
no ENV_INCLUDE_* macros meant the table held only its sentinel, so
begin() scanned the I2C bus and initialized nothing and querySensors()
emitted nothing above channel 1.
Reported as an INA226 missing from channel 2 on a MQTT Observer build,
working on stock firmware. Affected every Xiao_S3_WIO env, including
the dedicated Xiao_S3_WIO_sensor role.
Restores sensor_base build_flags/lib_deps, -UENV_INCLUDE_GPS and the
PIN_BOARD_SDA/SCL defines, making the base block identical to upstream
again. Also picks up the commented rs232 pin relocation from the same
upstream commit, since 5,6 is now the I2C bus.
Flash on Xiao_S3_WIO_repeater_observer_mqtt: 42.7% -> 44.3%.
It was committed as a gitlink (mode 160000) with no .gitmodules entry, so every
CI checkout's cleanup step ended with "No url found for submodule path
'.build-wt-028a5dca'" and a git exit-128 warning, and `git submodule` commands
failed in fresh clones. The worktree itself is removed locally; commit 028a5dca
stays reachable from this branch, so no history is affected.
Ignore the pattern so a throwaway build worktree cannot be committed again, and
add it to paths-ignore since such a directory can never change the binaries —
that also keeps this removal from triggering a pointless full rebuild.
A plain `git rebase` refuses to run when the checkout has unstaged changes, so
the retry path could fail instead of retrying. The beta workflow stages a scoped
subset on purpose (production's files must never ride along), which leaves the
rest of the checkout dirty and hits this exactly; keep all three flasher pushes
identical so it cannot resurface here either. Unstaged content stays unstaged,
so nothing extra gets published.
build-observer-firmwares.yml and sync-flasher-content.yml shared the
`flasher-publish` concurrency group. A mixed code+docs commit queues both, and
GitHub cancels any run left *pending* in an occupied group — cancel-in-progress:
false only shields runs already in progress. On 8fc96f68 the build lost that
race and was cancelled in 1s with zero jobs, so nothing was published.
Give each workflow its own group, and make the flasher push tolerate the
concurrent writes the shared group was guarding against: rebase onto the
sibling's commit and retry, up to 3 attempts. Both workflows generate
byte-identical docs and changelog content, so the loser's commit normally
rebases away to nothing and the push reports up-to-date.
Stamp FIRMWARE_BUILD_NUMBER into FIRMWARE_VERSION_STRING (the output
filename), so production assets become <env>-v1.16.0.N-<hash>.bin and the
web flasher's Version dropdown (parsed from the asset name by the /releases
Worker) shows the true published build instead of the bare base version.
The build-number suffix is computed once, up front, and now feeds BOTH the
filename and the embedded -DFIRMWARE_VERSION (byte-for-byte unchanged, it
already carried .N). Local builds without the counter still produce
3-segment names, so OTA and every filename parser stay backward-compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement SNMP functionality in MyMesh to provide radio diagnostics
and statistics updates. This includes a new method for formatting
radio diagnostic replies and periodic updates to the SNMP agent
with radio stats, enhancing monitoring capabilities for users
with SNMP enabled.
(cherry picked from commit fbe608d6c0)
Add functionality to schedule deferred OTA updates in the MyMesh class.
This includes a new member variable to track the update time and a method
to initiate the update process, allowing for a smoother OTA experience
by managing timing and state transitions effectively.
Implement logic to determine the maximum number of active MQTT slots
based on available memory. This change allows the system to warn users
when attempting to connect more slots than supported by the hardware,
improving user experience and preventing connection issues. The logic
is centralized in the new getMaxActiveSlots() method, ensuring
consistency across the codebase.
Enhance the documentation for the `start ota` command to clarify its
behavior when connected to a Wi-Fi network versus when using the
`start ota ap` option. This provides users with better guidance on
how to initiate OTA updates under different network conditions.
(cherry picked from commit 2019a8c0ca)
Added functionality for neighbor discovery, including the ability to
store and manage neighbor information. This includes methods for
sending discovery requests, handling responses, and updating the
neighbor table. The changes enhance the mesh networking capabilities
by allowing devices to discover and interact with nearby peers more
effectively. Updated relevant structures and methods in MyMesh.cpp
and MyMesh.h to support these features.
The flasher's versions become feed-driven (/releases on the firmware
proxy Worker): config.json's observer entries turn into github release
defs with no embedded filenames, so update-firmware.py would find zero
matches and exit nonzero — remove the call before the config conversion
lands. Changelog notes now ride the release body via a non-fatal
'gh release edit --notes-file firmware-notes.html' in the publish step.
MUST land before the flasher config.json conversion, or the next build
goes red at Update Flasher Config.
Workflow-only commit: pushing this triggers no build.
The rename to observer-firmware is verified: the rename's own push event
ran both workflows on the new name (branch creation bypasses path
filters), sync-flasher-content pushed to the flasher repo, and the
production publish completed on observer-firmware. The transition dual
trigger is no longer needed.
Also update the MQTT_INTERNALS.md prose reference to the old branch name.
Workflow+docs-only commit: pushing this triggers no build.
Flip gen-slim-manifests.py to its new --bin-dir mode (flasher repo PR #1):
the slim OTA manifests are now generated from out/ — the assets actually
uploaded to the release — instead of the config.json entries, which the
/releases feed migration will convert to github release defs. STATIC_PATH
joins the workflow env; it must stay consistent with config.json's
staticPath.
Also add observer-firmware alongside mqtt-bridge-implementation-flex in
both push triggers, ahead of renaming the branch: GitHub does not rewrite
branch filters on rename, and a rename with only the old name in the
filter silently stops production publishing. The old name is dropped once
a publish is verified on the new one.
Workflow-only commit (.github/** is in paths-ignore): pushing this
triggers no build.
Introduced a new MQTT preset for the meshtexas connection, allowing
users to connect to mqtt.meshtexas.org with JWT authentication over WSS.
This addition increases the number of built-in presets to 27, enhancing
the flexibility of the MQTT implementation.
publishStatusToSlot hardcoded the retain flag to true, ignoring the preset's
allow_retain field that the packets and neighbors paths already respect. The
waev preset (allow_retain=false, MeshCore topic style) has therefore been
publishing retained status to a broker that rejects the retain flag; the same
would now apply to meshrank, which just gained status publishing. Use
slot.preset->allow_retain, defaulting to true for custom slots to preserve
their existing behavior.
MeshRank slots now publish every message type (status/packets/raw/neighbors)
under meshrank/uplink/{token}/{device}/, matching the standard MeshCore topic
layout, instead of packets only. The four duplicated topic-suffix ternaries are
folded into messageTypeSuffix().
Fix the discover.neighbors -> discover.scopes race: a scopes request issued
while a zero-hop discovery is still collecting responses (from discover.neighbors
or the periodic timer) now queues behind that 60-second window and runs against
the refreshed table, reporting the wait in its reply. A queued one-shot request
is no longer cancelled by `mqtt.neighbors off`; only the periodic timer's own
refresh is. Precondition checks are shared via neighborDiscoverReady() so queuing
cannot report OK for a request that will fail once the window closes.
Add the periodic-neighbors schedule to `get mqtt.status` as a trailing
nbr:<next>/<last> field (time to next publish + last result). The mesh owns the
timer and reports a summary to the bridge across the core boundary via atomics.
Also clamp the running offset in formatMqttStatusReply: snprintf returns the
untruncated length, so a full 6-slot line could push the offset past the buffer
and underflow the remaining size on the next append. Reachable today with real
preset-name combinations at the 160-byte reply size.
Updated the MQTT neighbors functionality to include a two-stage
refresh process for periodic publishing. The first stage performs
a 60-second zero-hop neighbor refresh, followed by querying the
refreshed table for scopes before publishing. This change improves
the accuracy of neighbor data and ensures timely updates in the
network. Documentation has been updated to reflect these changes.
Adjusted the valid range for the `mqtt.neighbors.interval` setting from
12-8760 hours to 12-336 hours to reflect new constraints. Updated
related documentation and CLI commands to ensure consistency across
the codebase. This change improves clarity and prevents potential
misconfigurations in neighbor publishing intervals.
Credit: @yellowcooln (PR #12). Adds Wireless Tracker v1.1 board
def and MQTT observer envs for v1.1/v2; gate FEM control so v1.1
builds without KCT8103L pins.
The 30s "MQTT: Memory" line was useful during the outbox/sync-publish
investigation but is spam for production. Gate the periodic logMemoryStatus()
call in the MQTT task loop behind MQTT_MEMORY_DEBUG (a dedicated diagnostics
flag, not enabled by plain MQTT_DEBUG or production builds) and stop the
heltec_v3 variant from force-enabling MQTT_MEMORY_DEBUG on its observer_mqtt
envs (now commented out to match heltec_v4). logMemoryStatus() itself is kept
intact for opt-in debugging.
Expose the same data on demand via a new `get mqtt.stats` CLI command backed by
MQTTBridge::formatMqttStatsReply(): free/max heap, queue depth, outbox total,
and per-slot publish ok/err counts (1-based, matching the msgs: line). Fits the
160-byte reply buffer at 6 slots; returns "(bridge not running)" when down.
The esp-mqtt task drains only one QUEUED outbox item per loop iteration, and
each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on esp_transport_poll_read.
With little inbound traffic that caps throughput at ~1 message/second per
connection, so even a light packet rate (~1.2/s) outruns the drain: the outbox
pins at its cap and ~20-30% of QoS0 packets are dropped as backpressure. The
poll timeout is a compile-time constant baked into the precompiled esp-mqtt lib,
so the async drain rate cannot be raised on the Arduino/IDF 4.4 toolchain.
Route QoS0 packet publishes through esp_mqtt_client_publish() (async=false) so
they write straight to the socket, bypassing the outbox drain entirely — QoS0 no
longer touches the outbox. QoS1 status keeps the async/outbox + retransmit path.
The esp-mqtt task releases its API lock before the poll, so a synchronous publish
from the (Core-0, prio-1) MQTT task acquires the lock and writes immediately; a
stalled socket blocks only that task (mesh RX on Core 1 and the WiFi/TCP stack
are unaffected), bounded by a new setNetworkTimeout() lowered to 2500ms so a
first stall fails fast and flips the slot to disconnected.
The outbox cap from the previous commit stays as a dormant safety net. Retools
the MQTT_DEBUG diagnostic from outbox size/drops (now always ~0) to per-slot
publish ok/err counts, the live signal for delivery health, with 1-based slot
numbering to match the status line.
QoS0 packet/raw publishes are forced into the esp-mqtt outbox (store=true,
async) so packet topics keep flowing, but the outbox has no size bound of its
own — esp-mqtt frees entries only on send-ack or ~30s expiry. On a stalled or
slow uplink (socket still "connected") QoS0 frames accumulate on internal heap
without limit, driving the heap exhaustion/fragmentation seen in the field.
Cap the outbox at the application level: PsychicMqttClient::setOutboxLimit()
records a per-client byte cap, and publish() drops a QoS0 message (returns -2)
when esp_mqtt_client_get_outbox_size() is already at/over the cap, before
enqueuing. The bridge's existing processPacketQueue retry/drop path handles the
-2 as backpressure. Caps: 16 KiB PSRAM / 8 KiB non-PSRAM (outbox lives on
internal heap, so non-PSRAM is the fragmentation-sensitive case).
Portable across IDF 4.4 and 5 via esp_mqtt_client_get_outbox_size(); esp-mqtt's
own outbox.limit config is not used (its enqueue path does not reliably enforce
it for QoS0, and the app-level guard fires before enqueue regardless).
Adds getOutboxSize()/getOutboxLimit()/getOutboxDrops() and surfaces per-slot
outbox size/cap/drops via a throttled logMemoryStatus() in the MQTT task loop
(MQTT_DEBUG-gated) to confirm the bound on-target.
Reverts merge ae045395 (feat/flex-ipv6, PR #25). Enabling IPv6 joins the
device into IPv6 multicast/ND processing; on multicast-heavy LANs (e.g.
with a Thread/Matter border router advertising a ULA prefix) inbound
bursts land in dynamic WiFi RX buffers in internal heap. Measured on a
Station G2 with 4 WSS brokers: min-free floor dropped 36 KB -> 18 KB,
largest free block pinned below the 16 KB publish threshold for minutes
at a time, 523 dropped publishes in 15 minutes. With IPv6 disabled the
floor and max-alloc recovered and publish skips stopped.
The feature only fed the wifi.status display line — no transport uses
IPv6 (all brokers connect over IPv4), so the fleet risk (network-
dependent degradation on unknown home LANs) buys nothing. Revisit after
the Arduino core 3.x / IDF 5.x move if IPv6 transport is ever needed;
the branch remains at feat/flex-ipv6.
Kept from the merge: the wifi.status uptime append now computes the
actual remaining space in the 160-byte reply buffer instead of assuming
a hardcoded 128, fixing a latent overflow of the snprintf bound.
- Introduced commands for periodic neighbors publishing: `get mqtt.neighbors`, `set mqtt.neighbors on|off`, `get mqtt.neighbors.interval`, and `set mqtt.neighbors.interval <hours>`.
- Updated MQTT topics to include a new neighbors topic for cached zero-hop repeater neighbors.
- Added JSON message format for neighbors, including fields for SNR, last-heard age, and status.
- Enhanced CLI documentation to reflect new commands and settings.
- Implemented neighbor discovery logic in MyMesh class, including handling responses and publishing neighbor data.
- Adjusted MQTTBridge to manage neighbors JSON buffer and publish logic.
This update enhances the MQTT implementation by allowing devices to share neighbor information, improving network awareness and connectivity.
The token's exp claim and the renewal schedule derive from the same
value, so the flat 60 s RENEWAL_BUFFER was the entire margin between
proactive re-auth and the broker enforcing exp on the live session -
one failed renewal attempt (60 s throttle) or a minute of clock skew
lost the race, seen as clean-FIN disconnects (tls=0x8008) on the waev
preset, whose 55-minute tokens are the only ones short enough to hit
enforcement. Buffer is now lifetime/10 clamped to [60 s, 300 s], and
the disconnect-now threshold uses the same value so every renewal is
a proactive reconnect on the device's schedule; waev re-auths 10 min
before its real 60-minute TTL with ~5 retry windows.
Document why waev's preset claims 3300 s against the broker's real
3600 s TTL: the 5-minute claim-side gap protects token acceptance
against fast device clocks, which the renewal buffer cannot do.
A CONNACK alone reset the backoff ladder, so a flapping broker (accepts
then drops within seconds) retried at the 10 s rung forever - each
attempt a full ~40 KB TLS session alloc/free on internal heap, a known
fragmentation driver. The ladder now clears only after the connection
survives 2 minutes (at least one 75 s keepalive round-trip); flapping
endpoints degrade to the 300 s rung and then the existing 30-minute
circuit-breaker probes, and recover automatically once stable.
The precompiled IDF 4.4 WebSocket transport (libtcp_transport.a) has an
off-by-one in ws_connect(): when a wss:// endpoint answers the upgrade
request with >=1024 bytes of HTTP response before the blank-line
terminator (typical of a down broker behind a proxy serving a large
error page), it writes a NUL one byte past the 1024-byte ws->buffer.
Heap poisoning catches the clobbered tail canary (0xbaad5678 ->
0xbaad5600) only when the block is freed in ws_destroy() during
esp_mqtt_client_destroy() - i.e. MQTTBridge::end() - so a single down
broker made every deferred 'ota update' panic and reboot at teardown,
before the download started. Decoded from a Heltec V3 crash backtrace
on v1.16.0.11; line numbers match ESP-IDF release/v4.4 exactly.
The transport code ships precompiled, so patch at link time instead:
[esp32_base] wraps esp_transport_ws_init and the wrapper swaps the
fresh buffer for a (WS_BUFFER_SIZE + 1)-byte allocation, making the
out-of-bounds index land on owned memory. The oversized handshake then
fails cleanly instead of corrupting the heap. Pass-through on IDF 5.x,
where upstream already fixed it; delete with the Arduino core 3.x move.
Verified: wrap resolves from ESP32WsTransportFix.cpp.o in the observer
firmware.map; observer, room-server observer and plain repeater ESP32
targets build. RAK_4631_repeater failure is pre-existing (reproduced
on the merge base without these changes).
agessaman