The 2026-07-19 upstream merge auto-merged two additions into the same class
bodies without conflicting, producing duplicate declarations:
examples/simple_room_server/MyMesh.h - getCADEnabled()
variants/heltec_v4/HeltecV4Board.h - setLoRaFemLnaEnabled(),
canControlLoRaFemLna(),
isLoRaFemLnaEnabled()
Same class of breakage as the RadioLibWrapper::_cad_enabled and
MyMesh::getCADEnabled() duplicates already fixed in the merge commit. These
survived because the merge was validated with the two prescribed MQTT smoke
builds, and neither of them compiles simple_room_server or the heltec_v4
variant - so CI was the first thing to touch the broken files.
Now verified across ALL 32 observer envs locally, not a sample: 32/32 build.
Saving observer prefs logged an ESP32 error line on every save:
[E][vfs_api.cpp:182] remove(): /mqtt_prefs.tmp does not exists or is directory
MQTTPrefsFileStore::begin() cleared a stale transaction with an unconditional
remove("/mqtt_prefs.tmp"). On the normal path there is no stale tmp - commit()
renames it away - so the remove always failed and the ESP32 VFS layer logged it
at [E] level. The save itself succeeded; the noise just reads as a fault in the
serial log at exactly the moment an operator is watching a config change.
Guard each remove on exists(), at all four sites: begin() and abort() for both
the /mqtt_prefs and /com_prefs stores. Semantics are unchanged - a genuinely
stale tmp is still cleared, and a failure to clear it still aborts the
transaction - it just stops issuing a syscall that can only fail.
Fixed here on webconfig (the 1.16.0-based line that carries the atomic store)
so it flows to flex with the rest of that work. NOT applicable to
mqtt-bridge-implementation-flex today: flex has no .tmp/rename handling at all,
so the code path does not exist there.
Verified: Heltec_v3_repeater_observer_mqtt builds; hardware confirmation of the
silenced log pending.
The build step redeclared OTA_MANIFEST_BASE_URL and OTA_CHANNEL_TAG as
${{ env.X }}, referencing the very variables it was setting. Workflow-level
env: is already inherited by every step, so this was redundant; had the
self-reference resolved empty it would have silently blanked the channel and
produced firmware with no manifest base. The verify step would have caught it,
but the risk is unnecessary.
Dispatch-only does not work in this repo. The fork's default branch is `dev`
(an upstream mirror carrying none of the observer workflows), and GitHub only
surfaces workflow_dispatch for workflows present on the DEFAULT branch — so the
beta workflow would never have appeared in the Actions UI and could not have
been run at all.
Adding fork-specific workflows to `dev` would pollute the upstream mirror and
conflict on every upstream sync, so the push trigger is the right mechanism: it
runs from the file on the pushed branch, which is exactly how the production
observer workflow already works.
workflow_dispatch is retained (harmless, and starts working if the default
branch ever changes). paths-ignore mirrors production so docs/CI-only commits
do not rebuild firmware.
Trade-off now explicit in the file: every push to observer-firmware-dev
publishes a dev build. Stage on a side branch and fast-forward when you intend
to release.
The branch was named as a one-off dated merge (merge/upstream-dev-20260719),
but it is actually the standing development line: upstream merges land here and
the dev/beta firmware channel is built from it. Rename accordingly and reframe
the handoff so future upstream merges land ON this branch rather than spawning a
new dated branch each time.
Adds a "Branch and Release Channels" section with the full production vs
dev/beta separation table (branch, workflow, release tag, manifest base,
download host, flasher config, embedded version), and restates why both channels
share FIRMWARE_VERSION: the OTA logic treats a differing base as "always an
update", so channels must separate by manifest URL, never by base version.
Also corrects the beta workflow's dispatch-only rationale, which cited a
short-lived branch name that no longer applies. Dispatch-only still stands, for
the better reason: publishing firmware that real nodes pull over the air should
be explicit, not a side effect of every commit to a dev branch.
- OTA_CHANNEL_TAG is now 'beta-dev', so the embedded version carries the
channel AND its provenance: v1.16.0.N-observer-beta-dev-<hash>. This channel
is built from the upstream-dev-merged line, so 'dev' is visible in `ver`,
the MQTT firmware_version, and SNMP rather than inferred from a branch name.
Verified on a real build; OTA version parsing is unaffected.
- config-beta.json is now written into the flasher checkout and committed,
reversing the earlier ephemeral approach: the flasher SPA loads it directly
for ?config=config-beta, so it has to be served. It is still DERIVED from
config.json on every beta build rather than hand-maintained, so the dev/beta
device list cannot drift from production.
Publishes a parallel observer firmware channel that cannot cross-contaminate
production. Manual dispatch only, so the branch is chosen in the Actions UI
rather than hardcoded here.
Channel separation (each of these is load-bearing, not cosmetic):
- OTA_MANIFEST_BASE_URL -> beta nodes only ever read beta manifests. This is
the one that actually keeps devices on-channel.
- Separate RELEASE_TAG: the publish step prunes all but KEEP_BUILDS hashes
WITHIN its tag, so a shared tag would make each channel delete the other's
assets.
- Separate build counter: shared counters would interleave and make OTA's
"N behind" comparison meaningless.
- Separate staticPath via a derived config-beta.json.
FIRMWARE_VERSION deliberately matches production: the OTA logic treats a
different base version as "always an update", so channels must be separated by
manifest URL, not base version. OTA_CHANNEL_TAG marks the embedded version
instead (v1.16.0.N-observer-beta-<hash>) so `ver` identifies the channel.
config-beta.json is derived per build rather than committed - a checked-in copy
would be a 56-entry duplicate of config.json that goes stale as devices are
added. Deriving keeps the beta device list identical by construction.
Two verify steps fail the build rather than publish firmware that would OTA
itself onto production: one checks the beta URL is baked into a binary (and the
production URL is not), one checks the generated manifests use the beta host.
Production's changelog and docs sync steps are omitted - those rewrite site-wide
content the production channel owns. The flasher commit is scoped to the beta
manifest dir and counter for the same reason.
Also adds OTA_CHANNEL_TAG support to build.sh. Safe for OTA version parsing:
ota_parseVersion() reads to the first '-' and ota_extractHash() takes the token
after the last, so an extra tag between them changes neither. Verified on a
real build: v1.16.0.7-observer-beta-36831271.
The observer fetches its manifest from <OTA_MANIFEST_BASE>/<OTA_VARIANT>.json,
so that URL IS the release channel. It was hardcoded to the production channel
in 28 identical places across variants/*/platformio.ini, which made a parallel
(e.g. beta) channel impossible: both channels build the same env names, so beta
devices would read the production manifest and silently flash themselves onto
production firmware.
Inject it from build.sh instead, symmetric with OTA_VARIANT (which no .ini
declares), defaulting to the production URL. Set OTA_MANIFEST_BASE_URL to
publish a parallel channel.
Removed from the .ini files rather than overridden: PLATFORMIO_BUILD_FLAGS
cannot reliably override a -D from build_flags, because SCons reorders -U/-D
and the -U can land after both -Ds, leaving the macro undefined. Verified
empirically before choosing this approach.
A default in a header was deliberately NOT added: leaving both macros undefined
on a plain 'pio run' is what keeps ESP32Board.cpp's 'ERR: OTA not configured
(build via build.sh)' guard firing, so locally built firmware is never OTA-armed.
Verified on Heltec_v3_repeater_observer_mqtt:
- plain 'pio run' -> OTA disarmed, no manifest URL in the binary
- build.sh (default) -> https://observer.gessaman.com/v
- OTA_MANIFEST_BASE_URL set -> beta URL only, production URL absent
Roadmap status was stale in two ways: Phase 6 still described WebConfigBatch.h
as spec-only/not-wired, and there was no record of the upstream merge.
- Phase 6 / Current Baseline: record that WebConfigBatch is wired and
load-bearing, document the two deliberate spec/caller asymmetries, and state
plainly what is NOT verified (the POST/drain/result/reboot sequence over real
HTTP) so the next person does not mistake green host tests for coverage.
- New "Upstream Merge Record" section: the /com_prefs finding (offset-addressed,
so NodePrefs reordering is safe - with the warning not to generalize that to
the versioned /mqtt_prefs), every conflict resolution, the setRxBoostedGain
signature change, and the cost signal that the fork is the churn source and so
merges should follow each phase rather than batch.
- Forward plan replaced with the actual remaining work in execution order.
- Merge discipline: note that Git resolved two duplicate declarations silently
and only the compiler caught them.
First upstream merge since the 2026-06-06 base (191 upstream commits). 14 files
conflicted; resolutions below.
Fleet-critical check (Constraint 1): upstream reordered NodePrefs members
(rx_boosted_gain / path_hash_mode moved to the struct tail) but did NOT change
/com_prefs. Persistence is written field-by-field at explicit offsets, so member
order is in-memory only. Verified the fork's writeCommonPrefsImage() is
byte-identical to upstream's inline writer at every offset (79 pad, 121, 122,
290-294). No migration needed.
Resolutions:
- CommonCLI.h: kept the fork's NodePrefs (superset) and adopted upstream's
setRxBoostedGain(bool)->bool signature change, which CommonCLI.cpp now uses to
report unsupported. Corrected a stale comment claiming rx_boosted_gain lives at
offset 79 (it is a pad; the field is at 290).
- CommonCLI.cpp: kept the fork's legacy /com_prefs migration and the extracted
writeCommonPrefsImage() call.
- UITask.cpp: three-way merge - upstream's drawTextCentered + powering-off
screen, plus the fork's WITH_WEBCONFIG portal/reboot screens.
- ESP32Board.cpp, MeshCore.h, platformio.ini: kept both sides (fork OTA additions
alongside upstream powerOff/enterDeepSleep and Packet.cpp).
- MicroNMEALocationProvider.h: took upstream's claim/release and added the
_claims member they depend on.
- MyMesh.cpp/.h (repeater + room server): kept the fork's superset defaults.
- Removed duplicate declarations auto-merge produced: RadioLibWrapper::_cad_enabled
and MyMesh::getCADEnabled().
Verification: native suite 15/15 (incl. upstream's new test_mesh_tables), both
MQTT smoke builds green, ArduinoJson pin check passes. Hardware validation next.
The pure batch/reboot/stop state machine in WebConfigBatch.h was host-tested
but not referenced by production, so the real logic in WebConfigServer.cpp was
untested and the two could drift silently.
Repoint the production decision points at the spec: POST classification and
replay-state naming, drain pacing/all-ok/finish, reboot scheduling and firing,
result classification, confirm-reboot arming, and stop gating. MAX_BATCH and
STOP_WARN_MS now alias kMaxBatch/kStopWarnMs so the constants cannot drift.
Behavior-preserving. Two asymmetries are deliberate and documented in the
header: finishRebootAt()'s 0 return must not be assigned unconditionally
(the manual /api/reboot route also owns _reboot_at and could be cancelled),
and classifyPost() is consulted in two phases because the change count is
only known after parsing, which must not precede the Replay/Busy answer.
Native suite (14 suites) and both MQTT smoke builds green.
UTF-8 em-dashes (U+2014) inside MQTT_DEBUG_PRINTLN / CLI reply strings
render as mojibake ("aEUR"-style, e.g. cooperative stop garbles) on
consoles that don't decode UTF-8. Replace the em-dashes in printed
strings with ASCII '-'. Comments are intentionally left unchanged (they
never reach the console; rewriting them would be needless churn in a
merge-sensitive file).
Verified on V3: the stop message now emits pure ASCII -- zero non-ASCII
bytes in the boot+stop console capture.
An operator who triggers `ota update` via remote management only sees the
command's reply; the actual OTA runs ~2.5s later and reboots on success,
so the start/fail notifications land outside the reply window. Mirror the
key OTA milestones onto the configured alert channel (in addition to the
Serial log):
- START ("OTA update starting") from beginDeferredOtaUpdate(), i.e. while
the loop still runs -- a flood queued at the deferred fire could never
transmit before the flash blocks the loop / reboots on success.
- FAIL ("OTA aborted: ...") at both abort points (teardown barrier
withheld flashing; preflight/download error).
Success has no message: a successful flash reboots into the new image, so
the node returning on the new version is the signal.
New MyMesh::otaAlert() gates on the `alert on/off` master switch and rides
the configured alert scope (AlertReporter::sendText -> sendChannel ->
resolveAlertScope); no-op when alerts are off or no channel is set. Only
these start/fail milestones -- routine slot connect/disconnect is
unaffected (stays in AlertReporter's fault logic). Documented in ALERTS.md.
Both observer firmwares build.
3-slot PSRAM stop (V4) that force-timed-out at 8s pre-fix now logs
timeout 29000 ms and acks clean in ~16.4s -- confirms the slot-scaled
timeout on both memory paths (V3 non-PSRAM 2-slot + V4 PSRAM 3-slot).
Phase 0 hardware characterization showed real mbedTLS/wss client teardown
takes ~5-6s per connected slot, sequentially, so the flat 8s
MQTT_STOP_TIMEOUT_MS force-timed-out healthy multi-slot stops (2-slot
non-PSRAM ~11-12s, 3-slot PSRAM ~16s), which sets the dirty latch and
makes the OTA barrier withhold flashing -- multi-slot nodes could never
ota update.
Replace the flat constant with a slot-scaled budget computed per stop in
end(): 5s base + 8s per enabled slot (~1.5-2x headroom over measured),
applied via new MQTTLifecycle::Coordinator::setStopTimeoutMs() before
requestStop(). Headroom is nearly free: end() returns as soon as the task
acks (checked before the timeout ticks), so a larger bound only lengthens
the wait before force-killing a genuinely wedged task.
Hardware-verified on V3: a 2-slot stop that force-timed-out at 8s pre-fix
now logs "timeout 21000 ms" and acks clean in ~11.7s. Native suite +
both observer firmware builds green.
Hardware run of the outstanding Phase 0 (teardown timing) and Phase 7
(fault-injection) items on V3 (non-PSRAM, 1 wss slot) and V4 (PSRAM,
3 wss slots) observer nodes, both flashed with this branch.
Primary finding (release-gating): MQTT_STOP_TIMEOUT_MS=8000 is too
small. Per-wss-slot teardown is ~5-6s sequential, so healthy multi-slot
stops (2 slots at the non-PSRAM max, 3+ on PSRAM) exceed 8s, trip the
dirty/timeout fallback, and the OTA barrier withholds flashing -- so
multi-slot nodes could never ota update, and the forced path also stalls
the loop task ~15-27s. Recommend raising the timeout (slot-count-aware
preferred, e.g. 4s + 6s*active_slots) before Phase 5 ships.
Phase 7: no heap leak or crash across the representative fault-injection
matrix on either board (incl. repeated forced teardowns); non-PSRAM
reconfigure-churn fragmentation is bounded and fully reboot-recoverable;
PSRAM largest-block rock-stable. OTA barrier latch validated on hardware
in both clean and dirty states.
The WebConfig POST/result/reboot/stop batch state machine was the largest
remaining Phase 6 coverage gap (all inline in WebConfigServer.cpp, coupled to
AsyncWebServer/ArduinoJson and untestable on host). Extract its decision + timing
CORE into a pure, dependency-free spec mirroring MQTTLifecycle.h:
- src/helpers/WebConfigBatch.h: classifyPost (replay/busy/accept/no-changes with
the DONE-vs-PENDING reqid asymmetry), drain pacing (signed 25 ms gate, sticky
all_ok, 30 s reboot fallback), result classification + arm-once 3 s reboot,
signed-wrap-safe reboot-due / isRebootPending, and stop gating (finalize when
refs==0, warn-once, never force teardown). Constants verbatim from the source.
- test/test_webconfig_batch/: full host coverage incl. exact boundaries and
millis() rollover.
Spec-first, exactly like Phase 4's MQTTLifecycle.h: this is NOT yet wired into
WebConfigServer.cpp. That server is hardware-tuned (debugged against real iOS
captive-portal + HTTP-caching + route-ordering behavior), so making the spec
load-bearing is a deliberately separate, hardware-validated follow-up.
Faithfulness independently reviewed against WebConfigServer.cpp; native suite
green (14 dirs). No production behavior change.
Close the host-testable gaps named in Phase 6 of STABILITY_TESTABILITY_HANDOFF.md
by moving the last inline decision logic into the pure, host-tested policy seams:
- WiFi STA reconnect backoff: extract the inline ladder + wrap-safe timing from
handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,
wifiReconnectDue,nextWifiBackoffAttempt}. Behavior-preserving (elapsedMs is the
wrap-safe form of the old ULONG_MAX branch); ladder/clamp/attempt-cap unchanged.
- Publication outcome pairing: name the (packet, raw) -> delivered contract as
MQTTPacketQueuePolicy::queuedPacketPublished() and wire both queue-drain sites;
partial success = completed, not retried.
- Freeze MQTTPublicationType enum values in a test (the bridge-side MQTTMessageType
alignment is already enforced by a compile-time static_assert).
Adds host tests for all three (exact boundaries + millis() rollover). Native suite
green (13 dirs); non-PSRAM observer firmware smoke build compiles.
WebConfig batch/reboot/stop state-machine extraction and queue-orchestration
coverage remain open (tracked in the Phase 6 status).
Wire the Phase 4 MQTTLifecycle state machine into MQTTBridge to replace the
blind vTaskDelete teardown that could kill the MQTT task mid-mbedTLS and then
free client buffers on a corrupted heap (the observed OTA teardown panic).
- MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task
(Core 1) from begin()/end(); a nested LifecycleOps binds the host-tested Ops
spec to FreeRTOS/PsychicMqttClient.
- end() requests a cooperative stop; the MQTT task (Core 0) tears down its own
clients where the mbedTLS contexts live, acks via _stop_acked, and
self-terminates. end() waits (bounded) for the ack, then frees queue/buffers.
- Bounded stop timeout -> reviewed fallback (force kill + Core-1 teardown) sets
a dirty latch that withholds OTA flashing.
- begin() gains an idempotent double-call guard and syncs the Coordinator to
Running.
- OTA teardown barrier: simple_repeater's deferred flash aborts/resumes unless
end() reported a clean stop (canFlashAfterStop()).
Scope: minimal cooperative-shutdown unit. The volatile NTP/reconfigure handshake
replacement and the plain-data snapshot / consumer repointing (MQTT_OWNERSHIP.md
sections 1-3) are deferred. MQTT_STOP_TIMEOUT_MS is a Phase-0 placeholder pending
on-hardware characterization.
Native suite green (incl. test_mqtt_lifecycle); both observer firmware smoke
builds compile. Not yet hardware-validated (Phase 7 gate).
Fork-owned, host-tested MQTT bridge lifecycle state machine and the narrow
dependency seam to drive it deterministically, plus the cross-core ownership
model. This is the Phase 4 "ownership and teardown test seams" safety net that
must land before the Phase 5 cooperative-shutdown refactor.
- src/helpers/MQTTLifecycle.h: pure state machine
(Stopped->Starting->Running->StopRequested->Stopping->Stopped) + injected
Ops seam (clock / task / resource owner / OTA barrier) + Coordinator with a
bounded stop timeout. No Arduino/FreeRTOS/WiFi deps.
- test/test_mqtt_lifecycle/: 18 GoogleTest cases covering the phase's teardown
matrix (stop during every activity, callback timing, duplicate/early stop,
restart, timeout fallback, no-access-after-release) and the OTA-barrier
scenarios.
- MQTT_OWNERSHIP.md: one owner per mutable runtime domain, current cross-core
hazards with file:line references, target primitives, and Phase-0-pending
(hardware-characterization) items.
- STABILITY_TESTABILITY_HANDOFF.md: Phase 4 status and verified premise
refinements.
Scope: MQTTBridge.cpp is intentionally untouched. The production rewiring
(plain-data snapshot publication, volatile-handshake replacement, cooperative
shutdown) is deferred to Phase 5.
The handoff plan had drifted from the branch. Phases 1-3 are already
implemented, so this updates the plan of record to match reality and adds
a change-control section governing how agents extend the work.
Re-baseline:
- Add a Roadmap Status table and an explicit forward-plan execution order
(Phase 0 -> 4 -> 5 + OTA barrier -> 6 -> 7).
- Rewrite Current Baseline to include the landed CI, PSRAM buffer symmetry,
and versioned /mqtt_prefs migration work.
- Mark Phases 1-3 Complete with their residuals (build-size gate + ASan
pending; filesystem prefs adapter still in CommonCLI).
- Replace Phase 2's now-false premise (buffers are allocated in begin() /
released in end() / reallocated on restart, per MQTTRuntimeBufferLifecycle).
- Annotate Phases 0/4/5/6/7 with verified status, including the on-demand
getSlotStatusSnapshot() naming and the begin() double-call leak.
- Reframe the OTA teardown barrier as the fix for a known shipping heap panic.
Add "Change-Control Discipline (Stop-and-Ask)": agents must stop and ask
before implementing out-of-plan refactors or fixing newly discovered bugs,
report with evidence, and keep fixes single-purpose.
Rework the /mqtt_prefs load/save path so preference migrations are
crash-safe and, for the first time, unit-testable on the host.
Most of this is extraction. The multi-format migration that previously
lived inline in CommonCLI.cpp (and could only run on-device) is moved
into three dependency-free headers so it can be exercised without
Arduino, a filesystem, or the radio stack:
- MQTTPrefsStorage.h frozen layout structs for every /mqtt_prefs
format ever shipped, with static_asserts that
fail the build if any on-flash offset changes.
- MQTTPrefsCodec.h pure format classification, field-copy
migration, and plausibility validation.
- MQTTPrefsAtomicStore.h transactional writer plus the power-cut
upgrade gate, both host-testable.
New behavior, beyond the refactor:
- Atomic writes: /mqtt_prefs is written to /mqtt_prefs.tmp, verified,
then published with an atomic rename; the writer never removes the
existing file. A failed or interrupted save leaves the current
config intact.
- Power-cut ordering: LegacyUpgradeGate guarantees /mqtt_prefs is
durably committed before the legacy /com_prefs (or /node_prefs)
carrying the observer tail is compacted or removed, so an
interrupted two-file upgrade retries on the next boot without
losing settings.
- Corrupt, unsupported-version, and newer-than-known files are
preserved and the device boots on in-RAM defaults, rather than
overwriting a file this firmware cannot fully decode.
- Headerless legacy formats are validated for plausibility before
they are trusted and rewritten (raw prefs carry no checksum).
The full historical format matrix is migrated forward to the versioned
v1 layout: pre-slot (including pre-wifi-power), 3-slot (base and
token/topic tails), and headerless 6-slot (base, audience, rx, ntp).
Scope note: only /mqtt_prefs and the one-time /node_prefs -> /com_prefs
name migration use the atomic path. Ordinary /com_prefs saves remain a
direct rewrite, unchanged by this commit.
Tests: adds two host GoogleTest suites (pio test -e native).
- test_mqtt_prefs_codec: format classification, migration fixtures,
v1 header integrity, downgrade preservation.
- test_mqtt_prefs_atomic_store: transactional writes, short-write
detection, begin/finish/rename failure cleanup, original-file
preservation.
Replace direct JSON construction in MQTTMessageBuilder with calls to
MQTTPayloadBuilder for building status, packet, and raw messages.
This change improves code maintainability and reduces duplication by
centralizing message formatting logic. Additionally, update platformio.ini
to include ArduinoJson dependency for JSON handling.
Improve the mqttSubstituteTopic function to ensure that it reports
overflow when the output buffer is full, preventing silent truncation
of topics. Update related tests to verify the new behavior for
literal overflow and exact fit scenarios, enhancing robustness and
correctness of topic handling.
Add validation for request IDs in the web configuration server to ensure
they conform to the expected format. Enhance error responses for invalid
or unknown request IDs, improving the robustness of request handling.
Update the web UI to reflect these changes, ensuring that clients can
properly handle errors related to request ID mismatches.
Include detailed instructions for local testing of observer and WiFi
functionality without hardware. Document the use of a mock backend and
Wokwi ESP32-S3 simulation for easier development and testing.
Enhance the MQTT implementation documentation to improve developer
experience and facilitate testing workflows.
Enhance the WebConfigServer to track in-flight requests, ensuring that
the server is only freed once all requests have completed or a hard
timeout has been reached. This prevents crashes due to live connections
during server deletion. Update route handlers to log requests and
manage their lifecycle more effectively, improving stability and
performance of the web configuration portal.
Introduce a web configuration portal for easier node management and
provisioning without serial CLI. Enhance MQTT functionality with
improved IATA code validation, dynamic slot management, and
background NTP synchronization. Update web UI elements for better
user experience and security notes regarding open AP usage.
Add new configuration options for flood traffic management and loop
detection in the web interface. This includes parameters for maximum
flood hops, maximum advert hops, and loop detection modes, improving
the control over mesh network behavior.
agessaman