- Allow mqtt.neighbors and mqtt.neighbors.interval in the WebConfigKeys set-key
allowlist (the CLI enforces the PSRAM guard; the stub reply handles non-PSRAM).
- Emit neighbors + neighbors_interval (hours) in the WebConfigServer config JSON.
- Add a "Publish neighbors" toggle and a "Neighbors interval (hours)" field
(12-336) to the Publishing card, with getVal() cases in webui/index.html.
- Cover both keys in test_webconfig_keys.
WebConfigHtml.h is a gitignored build artifact regenerated by the pre-build
hook from index.html, so it is not committed. Verified: test_webconfig_keys
passes and the T_Beam_S3_Supreme observer_mqtt firmware builds [SUCCESS].
Port the neighbors CLI from mqtt-bridge-implementation-flex:
- set mqtt.neighbors on|off - enable/disable periodic publishing
- set mqtt.neighbors.interval <h> - 12-336 hours (rejected, not clamped), stored ms
- get mqtt.neighbors - "on"/"off"
- get mqtt.neighbors.interval - "> <hours> hours (<ms> ms)" (ceiling division)
Both gated on WITH_MQTT_NEIGHBORS with a WITH_MQTT_BRIDGE stub replying
"Err - not supported (requires PSRAM)" on non-PSRAM builds. Handlers only write
prefs + savePrefs() — the mesh loop reads them live, so no bridge restart and no
direct bridge/MyMesh call (matches flex).
Token ordering preserved: SET tokens keep their trailing space so the shorter
"mqtt.neighbors " can precede "mqtt.neighbors.interval "; GET tokens have no
trailing space so the longer ".interval" is tested first.
Port the periodic-neighbors publication from mqtt-bridge-implementation-flex,
adapted to this branch's structure:
- Add MQTT_PUBLICATION_NEIGHBORS ("neighbors") to the pure MQTTTopicRouter
instead of flex's messageTypeSuffix() helper (this branch already routes
every publication type through mqttBuildPublicationTopic()). Neighbors is a
MeshCore/custom publication type, so it resolves to
meshcore/{iata}/{device}/neighbors and honors custom templates.
- Deliberately do NOT port flex's "all message types to MeshRank" change:
this branch documents and host-tests a packets-only MeshRank contract
(MQTTPresets.h, MQTT_IMPLEMENTATION.md, MeshRankContractIsPacketsOnly). So
neighbors follows status/raw and is rejected on MeshRank slots.
- WITH_MQTT_NEIGHBORS guard (PSRAM + MAX_NEIGHBOURS) gates all new surface.
- MSG_NEIGHBORS message type + enum-drift static_assert.
- Persistent ~10KB PSRAM neighbors buffer allocated/freed via the existing
MQTTRuntimeBufferLifecycle path (allocate/release), not the ctor as flex did.
- Core1->Core0 handoff: requestPublishNeighbors() (mesh) fills the buffer with
a release store; the MQTT task consumes it with an acquire load, publishes
via publishNeighbors() (QoS1, retain = preset->allow_retain, custom=false),
and clears the pending flag. A second snapshot is dropped while one is
in flight.
- setNeighborsSchedule()/NeighborsPhase let the mesh report the timer summary;
formatMqttStatusReply() gains a "nbr: <when>/<last>" field via formatDuration.
Also fix the on-connect status publish (publishStatusToSlot) to honor
preset->allow_retain instead of hardcoding retain=true, matching the periodic
publishStatus() path. Brokers with allow_retain=false (e.g. the waev MeshCore
preset) reject retained publishes, so the on-connect status was being dropped
there. This is flex followup 028a5dca, reconciled to this branch's custom-slot
default of non-retained.
Extends the host topic-router test to cover the neighbors type across all
routes and freezes the new enum value. Bridge itself is on-target only.
Add buildNeighborsMessage to the pure MQTTPayloadBuilder core and a thin
delegating wrapper + NeighborsMessageEntry alias on MQTTMessageBuilder, so
the neighbors topic is built by the same firmware-facing API as status/
packet/raw while the layout logic stays exercisable by native tests.
The document is bounded to the publish buffer: entries arrive ordered most-
to least-useful and the tail is dropped once the next entry would overflow,
so a fixed PSRAM buffer can never be handed truncated JSON.
Uses ArduinoJson v7 idioms (.to<JsonObject>()/.add<JsonObject>()) to stay
warning-clean under -Werror, unlike the deprecated createNested* forms.
Adds three test_mqtt_payload_builder cases: self+entry round-trip, empty
table / null scopes, and bounded-growth tail-drop under a tight buffer.
Append mqtt_neighbors_enabled(u8) + mqtt_neighbors_interval(u32) to the
observer tail of MQTTPrefs. The layout is kept byte-identical to the flex
neighbors build: the enable flag lands in the old struct's zeroed trailing
padding (offset 2857) and the interval begins exactly at the former baseline
(2860), so sizeof grows 2860 -> 2864 (net +4 bytes). offsetof static_asserts
lock the layout so a mismatch fails the build.
The codec now accepts three v1 payload sizes: register 2860 as a
"pre-neighbors" Current payload so an in-lineage upgrade reads its existing
/mqtt_prefs and defaults the neighbors tail (off / 24h). Because 2864 is the
shared Current baseline, a /mqtt_prefs written by either the flex build or this
firmware is interchangeable.
Add the 12/24/336h interval constants, neighbors defaults, and a load-time
interval clamp that keeps persisted values inside the signed-delta millis()
scheduling window. Extend the host codec suite with a pre-neighbors migration
case and neighbors round-trip coverage.
Three real regressions from the 2026-07-19 upstream merge, all invisible to the
two prescribed smoke builds:
- heltec_tracker_v2/HeltecTrackerV2Board.cpp: the FEM trio
(setLoRaFemLnaEnabled/canControlLoRaFemLna/isLoRaFemLnaEnabled) was duplicated
verbatim by auto-merge, and upstream's new powerOff() (35f654ce) used
P_LORA_PA_POWER unguarded — a macro defined only for the tracker_v2 envs, while
heltec_tracker_v1_1 compiles the same board file. Guarded it the same way
LoRaFEMControl.cpp already guards that macro.
- SimpleMeshTables.h: the tracker variants pull in TFT_eSPI, whose
TFT_eSPI_ESP32_S3.h defines FS_NO_GLOBALS. That suppresses FS.h's own
'using fs::File', so File never reached global scope and every TU routed
through it failed with "'File' has not been declared" — here and at
simple_repeater/MyMesh.h:158. Restore the using when FS_NO_GLOBALS is set.
Explicit fs::File is not an option: File is also the global type on the
nRF52/RP2040 paths, which have no fs namespace.
- ST7735Display.cpp: upstream's HSPI fix (d30d8ed7) guarded on
HELTEC_LORA_V3 || HELTEC_TRACKER_V2. heltec_tracker_v1_1 matches neither and
fell through to &SPI1, which is not instantiated on ESP32. Added it to the guard.
Also parks LilyGo_TLora_V2_1_1_6_{repeater,room_server}_observer_mqtt with a
trailing underscore (the nibble_screen_connect convention from b8f1fad6), which
also excludes them from the workflows' enumeration regex. That board does NOT
fit and never did: 2,069,397 / 1,966,080 = 105.3% on flex, with no webconfig and
no upstream merge. It has been failing on production all along — the release
ships 30 envs, not 32 — hidden because build.sh does not propagate pio's exit
code. Dropping webconfig would recover ~46 KB of a ~101 KB deficit, so that is
not a fix. Rationale and options in .scratch/tlora-v2-oversize.md.
Saving observer prefs logged an ESP32 error line on every save:
[E][vfs_api.cpp:182] remove(): /mqtt_prefs.tmp does not exists or is directory
MQTTPrefsFileStore::begin() cleared a stale transaction with an unconditional
remove("/mqtt_prefs.tmp"). On the normal path there is no stale tmp - commit()
renames it away - so the remove always failed and the ESP32 VFS layer logged it
at [E] level. The save itself succeeded; the noise just reads as a fault in the
serial log at exactly the moment an operator is watching a config change.
Guard each remove on exists(), at all four sites: begin() and abort() for both
the /mqtt_prefs and /com_prefs stores. Semantics are unchanged - a genuinely
stale tmp is still cleared, and a failure to clear it still aborts the
transaction - it just stops issuing a syscall that can only fail.
Fixed here on webconfig (the 1.16.0-based line that carries the atomic store)
so it flows to flex with the rest of that work. NOT applicable to
mqtt-bridge-implementation-flex today: flex has no .tmp/rename handling at all,
so the code path does not exist there.
Verified: Heltec_v3_repeater_observer_mqtt builds; hardware confirmation of the
silenced log pending.
First upstream merge since the 2026-06-06 base (191 upstream commits). 14 files
conflicted; resolutions below.
Fleet-critical check (Constraint 1): upstream reordered NodePrefs members
(rx_boosted_gain / path_hash_mode moved to the struct tail) but did NOT change
/com_prefs. Persistence is written field-by-field at explicit offsets, so member
order is in-memory only. Verified the fork's writeCommonPrefsImage() is
byte-identical to upstream's inline writer at every offset (79 pad, 121, 122,
290-294). No migration needed.
Resolutions:
- CommonCLI.h: kept the fork's NodePrefs (superset) and adopted upstream's
setRxBoostedGain(bool)->bool signature change, which CommonCLI.cpp now uses to
report unsupported. Corrected a stale comment claiming rx_boosted_gain lives at
offset 79 (it is a pad; the field is at 290).
- CommonCLI.cpp: kept the fork's legacy /com_prefs migration and the extracted
writeCommonPrefsImage() call.
- UITask.cpp: three-way merge - upstream's drawTextCentered + powering-off
screen, plus the fork's WITH_WEBCONFIG portal/reboot screens.
- ESP32Board.cpp, MeshCore.h, platformio.ini: kept both sides (fork OTA additions
alongside upstream powerOff/enterDeepSleep and Packet.cpp).
- MicroNMEALocationProvider.h: took upstream's claim/release and added the
_claims member they depend on.
- MyMesh.cpp/.h (repeater + room server): kept the fork's superset defaults.
- Removed duplicate declarations auto-merge produced: RadioLibWrapper::_cad_enabled
and MyMesh::getCADEnabled().
Verification: native suite 15/15 (incl. upstream's new test_mesh_tables), both
MQTT smoke builds green, ArduinoJson pin check passes. Hardware validation next.
The pure batch/reboot/stop state machine in WebConfigBatch.h was host-tested
but not referenced by production, so the real logic in WebConfigServer.cpp was
untested and the two could drift silently.
Repoint the production decision points at the spec: POST classification and
replay-state naming, drain pacing/all-ok/finish, reboot scheduling and firing,
result classification, confirm-reboot arming, and stop gating. MAX_BATCH and
STOP_WARN_MS now alias kMaxBatch/kStopWarnMs so the constants cannot drift.
Behavior-preserving. Two asymmetries are deliberate and documented in the
header: finishRebootAt()'s 0 return must not be assigned unconditionally
(the manual /api/reboot route also owns _reboot_at and could be cancelled),
and classifyPost() is consulted in two phases because the change count is
only known after parsing, which must not precede the Replay/Busy answer.
Native suite (14 suites) and both MQTT smoke builds green.
UTF-8 em-dashes (U+2014) inside MQTT_DEBUG_PRINTLN / CLI reply strings
render as mojibake ("aEUR"-style, e.g. cooperative stop garbles) on
consoles that don't decode UTF-8. Replace the em-dashes in printed
strings with ASCII '-'. Comments are intentionally left unchanged (they
never reach the console; rewriting them would be needless churn in a
merge-sensitive file).
Verified on V3: the stop message now emits pure ASCII -- zero non-ASCII
bytes in the boot+stop console capture.
Phase 0 hardware characterization showed real mbedTLS/wss client teardown
takes ~5-6s per connected slot, sequentially, so the flat 8s
MQTT_STOP_TIMEOUT_MS force-timed-out healthy multi-slot stops (2-slot
non-PSRAM ~11-12s, 3-slot PSRAM ~16s), which sets the dirty latch and
makes the OTA barrier withhold flashing -- multi-slot nodes could never
ota update.
Replace the flat constant with a slot-scaled budget computed per stop in
end(): 5s base + 8s per enabled slot (~1.5-2x headroom over measured),
applied via new MQTTLifecycle::Coordinator::setStopTimeoutMs() before
requestStop(). Headroom is nearly free: end() returns as soon as the task
acks (checked before the timeout ticks), so a larger bound only lengthens
the wait before force-killing a genuinely wedged task.
Hardware-verified on V3: a 2-slot stop that force-timed-out at 8s pre-fix
now logs "timeout 21000 ms" and acks clean in ~11.7s. Native suite +
both observer firmware builds green.
The WebConfig POST/result/reboot/stop batch state machine was the largest
remaining Phase 6 coverage gap (all inline in WebConfigServer.cpp, coupled to
AsyncWebServer/ArduinoJson and untestable on host). Extract its decision + timing
CORE into a pure, dependency-free spec mirroring MQTTLifecycle.h:
- src/helpers/WebConfigBatch.h: classifyPost (replay/busy/accept/no-changes with
the DONE-vs-PENDING reqid asymmetry), drain pacing (signed 25 ms gate, sticky
all_ok, 30 s reboot fallback), result classification + arm-once 3 s reboot,
signed-wrap-safe reboot-due / isRebootPending, and stop gating (finalize when
refs==0, warn-once, never force teardown). Constants verbatim from the source.
- test/test_webconfig_batch/: full host coverage incl. exact boundaries and
millis() rollover.
Spec-first, exactly like Phase 4's MQTTLifecycle.h: this is NOT yet wired into
WebConfigServer.cpp. That server is hardware-tuned (debugged against real iOS
captive-portal + HTTP-caching + route-ordering behavior), so making the spec
load-bearing is a deliberately separate, hardware-validated follow-up.
Faithfulness independently reviewed against WebConfigServer.cpp; native suite
green (14 dirs). No production behavior change.
Close the host-testable gaps named in Phase 6 of STABILITY_TESTABILITY_HANDOFF.md
by moving the last inline decision logic into the pure, host-tested policy seams:
- WiFi STA reconnect backoff: extract the inline ladder + wrap-safe timing from
handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,
wifiReconnectDue,nextWifiBackoffAttempt}. Behavior-preserving (elapsedMs is the
wrap-safe form of the old ULONG_MAX branch); ladder/clamp/attempt-cap unchanged.
- Publication outcome pairing: name the (packet, raw) -> delivered contract as
MQTTPacketQueuePolicy::queuedPacketPublished() and wire both queue-drain sites;
partial success = completed, not retried.
- Freeze MQTTPublicationType enum values in a test (the bridge-side MQTTMessageType
alignment is already enforced by a compile-time static_assert).
Adds host tests for all three (exact boundaries + millis() rollover). Native suite
green (13 dirs); non-PSRAM observer firmware smoke build compiles.
WebConfig batch/reboot/stop state-machine extraction and queue-orchestration
coverage remain open (tracked in the Phase 6 status).
Wire the Phase 4 MQTTLifecycle state machine into MQTTBridge to replace the
blind vTaskDelete teardown that could kill the MQTT task mid-mbedTLS and then
free client buffers on a corrupted heap (the observed OTA teardown panic).
- MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task
(Core 1) from begin()/end(); a nested LifecycleOps binds the host-tested Ops
spec to FreeRTOS/PsychicMqttClient.
- end() requests a cooperative stop; the MQTT task (Core 0) tears down its own
clients where the mbedTLS contexts live, acks via _stop_acked, and
self-terminates. end() waits (bounded) for the ack, then frees queue/buffers.
- Bounded stop timeout -> reviewed fallback (force kill + Core-1 teardown) sets
a dirty latch that withholds OTA flashing.
- begin() gains an idempotent double-call guard and syncs the Coordinator to
Running.
- OTA teardown barrier: simple_repeater's deferred flash aborts/resumes unless
end() reported a clean stop (canFlashAfterStop()).
Scope: minimal cooperative-shutdown unit. The volatile NTP/reconfigure handshake
replacement and the plain-data snapshot / consumer repointing (MQTT_OWNERSHIP.md
sections 1-3) are deferred. MQTT_STOP_TIMEOUT_MS is a Phase-0 placeholder pending
on-hardware characterization.
Native suite green (incl. test_mqtt_lifecycle); both observer firmware smoke
builds compile. Not yet hardware-validated (Phase 7 gate).
Fork-owned, host-tested MQTT bridge lifecycle state machine and the narrow
dependency seam to drive it deterministically, plus the cross-core ownership
model. This is the Phase 4 "ownership and teardown test seams" safety net that
must land before the Phase 5 cooperative-shutdown refactor.
- src/helpers/MQTTLifecycle.h: pure state machine
(Stopped->Starting->Running->StopRequested->Stopping->Stopped) + injected
Ops seam (clock / task / resource owner / OTA barrier) + Coordinator with a
bounded stop timeout. No Arduino/FreeRTOS/WiFi deps.
- test/test_mqtt_lifecycle/: 18 GoogleTest cases covering the phase's teardown
matrix (stop during every activity, callback timing, duplicate/early stop,
restart, timeout fallback, no-access-after-release) and the OTA-barrier
scenarios.
- MQTT_OWNERSHIP.md: one owner per mutable runtime domain, current cross-core
hazards with file:line references, target primitives, and Phase-0-pending
(hardware-characterization) items.
- STABILITY_TESTABILITY_HANDOFF.md: Phase 4 status and verified premise
refinements.
Scope: MQTTBridge.cpp is intentionally untouched. The production rewiring
(plain-data snapshot publication, volatile-handshake replacement, cooperative
shutdown) is deferred to Phase 5.
Rework the /mqtt_prefs load/save path so preference migrations are
crash-safe and, for the first time, unit-testable on the host.
Most of this is extraction. The multi-format migration that previously
lived inline in CommonCLI.cpp (and could only run on-device) is moved
into three dependency-free headers so it can be exercised without
Arduino, a filesystem, or the radio stack:
- MQTTPrefsStorage.h frozen layout structs for every /mqtt_prefs
format ever shipped, with static_asserts that
fail the build if any on-flash offset changes.
- MQTTPrefsCodec.h pure format classification, field-copy
migration, and plausibility validation.
- MQTTPrefsAtomicStore.h transactional writer plus the power-cut
upgrade gate, both host-testable.
New behavior, beyond the refactor:
- Atomic writes: /mqtt_prefs is written to /mqtt_prefs.tmp, verified,
then published with an atomic rename; the writer never removes the
existing file. A failed or interrupted save leaves the current
config intact.
- Power-cut ordering: LegacyUpgradeGate guarantees /mqtt_prefs is
durably committed before the legacy /com_prefs (or /node_prefs)
carrying the observer tail is compacted or removed, so an
interrupted two-file upgrade retries on the next boot without
losing settings.
- Corrupt, unsupported-version, and newer-than-known files are
preserved and the device boots on in-RAM defaults, rather than
overwriting a file this firmware cannot fully decode.
- Headerless legacy formats are validated for plausibility before
they are trusted and rewritten (raw prefs carry no checksum).
The full historical format matrix is migrated forward to the versioned
v1 layout: pre-slot (including pre-wifi-power), 3-slot (base and
token/topic tails), and headerless 6-slot (base, audience, rx, ntp).
Scope note: only /mqtt_prefs and the one-time /node_prefs -> /com_prefs
name migration use the atomic path. Ordinary /com_prefs saves remain a
direct rewrite, unchanged by this commit.
Tests: adds two host GoogleTest suites (pio test -e native).
- test_mqtt_prefs_codec: format classification, migration fixtures,
v1 header integrity, downgrade preservation.
- test_mqtt_prefs_atomic_store: transactional writes, short-write
detection, begin/finish/rename failure cleanup, original-file
preservation.
Replace direct JSON construction in MQTTMessageBuilder with calls to
MQTTPayloadBuilder for building status, packet, and raw messages.
This change improves code maintainability and reduces duplication by
centralizing message formatting logic. Additionally, update platformio.ini
to include ArduinoJson dependency for JSON handling.
Improve the mqttSubstituteTopic function to ensure that it reports
overflow when the output buffer is full, preventing silent truncation
of topics. Update related tests to verify the new behavior for
literal overflow and exact fit scenarios, enhancing robustness and
correctness of topic handling.
Add validation for request IDs in the web configuration server to ensure
they conform to the expected format. Enhance error responses for invalid
or unknown request IDs, improving the robustness of request handling.
Update the web UI to reflect these changes, ensuring that clients can
properly handle errors related to request ID mismatches.
Include detailed instructions for local testing of observer and WiFi
functionality without hardware. Document the use of a mock backend and
Wokwi ESP32-S3 simulation for easier development and testing.
Enhance the MQTT implementation documentation to improve developer
experience and facilitate testing workflows.
Enhance the WebConfigServer to track in-flight requests, ensuring that
the server is only freed once all requests have completed or a hard
timeout has been reached. This prevents crashes due to live connections
during server deletion. Update route handlers to log requests and
manage their lifecycle more effectively, improving stability and
performance of the web configuration portal.
Introduce a web configuration portal for easier node management and
provisioning without serial CLI. Enhance MQTT functionality with
improved IATA code validation, dynamic slot management, and
background NTP synchronization. Update web UI elements for better
user experience and security notes regarding open AP usage.
Add new configuration options for flood traffic management and loop
detection in the web interface. This includes parameters for maximum
flood hops, maximum advert hops, and loop detection modes, improving
the control over mesh network behavior.
next_check and next_gps_update stored a future millis() value in a signed
long and compared with a naive '>'. After the ~24.8-day millis() sign flip
the deadline sits above the wrapped millis(), so the block never runs again
and GPS->RTC time-sync (and the location cache refresh) stall permanently
until reboot. Switch to unsigned deadlines with the wrap-safe signed-
difference compare '(long)(millis() - deadline) > 0', matching the idiom in
Dispatcher::millisHasNowPassed.
Also: reorder the MicroNMEALocationProvider ctor init-list to declaration
order (silences -Wreorder) and drop the always-true 'if (_claims > 0)' guard
in claim() (claim() always runs after _claims++, so it is >= 1).
agessaman