HaloKeymind/STABILITY_TESTABILITY_HANDOFF.md

Stability, Testability, and Upstream-Merge Handoff

Purpose

This document is the plan of record for refining the fork-owned WebConfig and MQTT observer code after the initial policy extraction and host-test work. The goal is to improve runtime stability, long-uptime confidence, and serviceability without broad rewrites of upstream-heavy files or creating unnecessary merge conflicts.

The order is deliberate: cheap CI and persistence guardrails land first, then tests and ownership boundaries needed to make lifecycle work safe, and only then the cooperative-shutdown refactor. Hardware soak testing validates the result; it is not the first line of defense for the riskiest change.

Roadmap Status

The guardrail phases have already landed on this branch; the remaining work is the lifecycle refactor and its safety net. Phases are intentionally not renumbered so cross-references and the completed acceptance criteria stay stable; each phase below carries an explicit status line, and this table is the quick index.

Phase	Scope	Status
1	PR CI smoke builds + ArduinoJson pin enforcement	Done (build-size gate and ASan/UBSan still pending)
2	PSRAM restart resource symmetry	Done
3	MQTT preference migration fixtures	Done (filesystem adapter still lives in CommonCLI)
0	Pre-change lifecycle characterization	Hardware run 2026-07-19 (see "Hardware Characterization Results"): teardown timing measured on V3+V4. Finding: flat MQTT_STOP_TIMEOUT_MS=8000 too small -> FIXED with slot-scaled timeout (5s + 8sxslots), hardware-verified (2-slot stop now clean)
4	Ownership and teardown test seams	Seams + ownership doc + teardown tests done; production rewiring deferred to Phase 5
5	Cooperative MQTT shutdown	Minimal cooperative end() + begin() guard + OTA barrier implemented on branch phase5/cooperative-mqtt-shutdown (native green, firmware smoke build green); NOT hardware-validated. Volatile-handshake replacement + snapshot-consumer repointing deferred
--	OTA teardown barrier	Implemented -- flash gated on a clean MQTT stop in simple_repeater; not hardware-validated
6	Request/queue/connection/publication integration tests	Partial: WiFi-backoff + publish-outcome + enum-alignment gaps extracted and host-tested; WebConfig batch/reboot/stop spec (WebConfigBatch.h) now wired into WebConfigServer.cpp (2026-07-19) so the host tests cover production; queue-orchestration coverage still open, and the wired path is not yet exercised over real HTTP
7	Uptime, memory, and fault-injection gates	Representative HW matrix run 2026-07-19: V3 non-PSRAM + V4 PSRAM done (no leak/crash; forced-path OTA-withhold + ~15-27 s loop stall observed). Multi-day soak + stack-HWM build pending
--	Upstream merge	upstream/dev merged 2026-07-19 on observer-firmware-dev (191 commits, 14 conflicted files). See "Upstream Merge Record". Not yet promoted to webconfig
--	Dev release channel	observer-firmware-dev publishes the dev/beta firmware channel (see "Release Channels"). Manual dispatch; separate from production

Phases 0, 4, and 5 (with the OTA teardown barrier) are landed and hardware-validated. Remaining work, in execution order:

1. Exercise the wired WebConfig batch machine over real HTTP (Phase 6) -- the only untested part of a change that is already in the branch.
2. Drive the live ota update deferred-flash path on a bench node (OTA barrier action; the latch input is already hardware-verified).
3. Close the remaining Phase 6 queue-orchestration coverage.
4. Phase 7 multi-day soak + task-stack-HWM build.
5. Promote observer-firmware-dev into webconfig once the items above are met, then keep merging upstream after each phase rather than batching (see "Upstream Merge Record").

Do not reopen a "Done" phase without a deliberate reason (see "Change-Control Discipline").

Hardware Characterization Results (Phase 0 & Phase 7) -- 2026-07-19

Hardware run of the outstanding Phase 0 (pre-change/cooperative teardown timing) and Phase 7 (uptime/memory/fault-injection) items against two live observer nodes. Not yet committed as a plan change -- this section records measured results and a release-gating recommendation for review.

Setup

• V3 -- Heltec WiFi LoRa 32 V3, non-PSRAM (max 2 active slots), env Heltec_v3_repeater_observer_mqtt, 1 configured slot (meshmapper, wss).
• V4 -- Heltec WiFi LoRa 32 V4, 2 MB PSRAM (max 5 active slots), env heltec_v4_repeater_observer_mqtt, 3 configured slots (analyzer-us, cascadiamesh, waev, all wss). mbedTLS + JSON/raw buffers allocate in PSRAM.
• Both flashed with this branch (phase5/cooperative-mqtt-shutdown) via pio run -t upload (app-slot flash only; nvs+spiffs preserved, no erase). Note: FIRMWARE_VERSION/FIRMWARE_BUILD_DATE are hardcoded in MyMesh.h and were not bumped, so ver reads "v1.16.0 (6 Jun 2026)" on both old and new builds -- confirm the flash by behavior (the (clean) / cooperative-stop log lines), not by ver.
• Driven over the serial CLI (set bridge.enabled off/on = end()/begin(); set mqttN.* = slot reconfigure; get mqtt.stats = Free/Max(largest block)/queue/outbox/per-slot counters). Host-side timestamped log parsing.

PRIMARY FINDING -- MQTT_STOP_TIMEOUT_MS = 8000 is too small (release-gating)

Per-wss-slot teardown costs ~5-6 s, applied sequentially (destroySlotClients(): disconnect() -> 50 ms -> esp_mqtt_client_destroy(); the ~5 s is the esp-mqtt task/network close, not the 50 ms settle). This is unchanged from the pre-change path (same teardown code), so the cooperative end() ack time scales with the number of connected slots:

Config	Slots	Measured teardown	vs 8 s timeout	Result
V3 non-PSRAM	1 wss	ack ~5.3-6.2 s (avg 5.8)	under	clean PASS
V3 non-PSRAM (design max)	2 wss	~11-12 s (cut off at 8 s)	over	forced/dirty -> OTA withheld FAIL
V4 PSRAM (normal config)	3 wss	~16 s (cut off at 8 s; per-slot disc @2.2/7.8/10.6 s)	over	forced/dirty -> OTA withheld FAIL
V4 PSRAM (design max)	5 wss	~27-30 s projected	far over	forced/dirty -> OTA withheld FAIL

Pre-change (v1.16.0) reference teardown (same nodes, blocking end()): 1 slot ~6.1 s (V3), 3 slots ~16.4 s (V4) -- consistent with the above.

Consequence: at the non-PSRAM board's designed maximum of 2 wss slots, and on a normal healthy 3-slot PSRAM node, an ordinary cooperative shutdown exceeds 8 s -> the coordinator fires StopTimedOut -> sets the dirty latch -> force-kills the MQTT task (the exact mbedTLS-mid-teardown path Phase 5 exists to avoid) and canFlashAfterStop() returns false. Because the OTA barrier only flashes after a clean stop, this means any multi-slot device would have ota update permanently withheld with the current timeout -- the barrier misclassifies healthy stops as dirty. This inverts the barrier's intent and must be fixed before Phase 5 ships.

FIX APPLIED (slot-count-aware timeout) -- verified on hardware.

The flat MQTT_STOP_TIMEOUT_MS = 8000 is replaced by a slot-scaled budget computed per stop in MQTTBridge::end():

timeout = MQTT_STOP_TIMEOUT_BASE_MS (5000) + MQTT_STOP_TIMEOUT_PER_SLOT_MS (8000) x enabled_slots

-> 1 slot 13 s, 2 slots 21 s, 3 slots 29 s, 5 slots 45 s -- ~1.5-2x headroom over the measured ~5-6 s/slot teardown. end() counts enabled slots and calls Coordinator::setStopTimeoutMs() before requestStop(). Headroom is nearly free because end() returns as soon as the task acks (_stop_acked is checked before the timeout ticks), so a larger bound does not slow a healthy stop -- it only lengthens the wait before force-killing a genuinely wedged task. (Files: MQTTBridge.cpp constants + end(); MQTTLifecycle.h setStopTimeoutMs().)

Hardware verification (both memory paths): the same configs that force-timed- out at 8 s pre-fix now ack cleanly -- V3 non-PSRAM 2-slot logs timeout 21000 ms and acks in ~11.7 s; V4 PSRAM 3-slot logs timeout 29000 ms and acks in ~16.4 s -> MQTT Bridge stopped (clean) (OTA no longer withheld). Native suite + both firmware builds green. Complementary future option (not done): shorten the per-slot esp_mqtt_client_destroy() wait to reduce absolute teardown time.

Phase 0 -- cooperative lifecycle (V3, non-PSRAM, this branch)

• 6x and 10x begin->connect->end->restart cycles: all clean, ack 5.3-6.2 s, end() loop-task block <=6.9 s, restart->connect ~5.4 s.
• No leak across full stop/start cycles: free heap flat (~137-142 k, varies only with the meshmapper outbox), largest free block stable (~115-129 k). This satisfies Phase 5's "no downward heap/largest-block trend across start/stop cycles" acceptance criterion on the non-PSRAM path.

OTA teardown barrier

• Barrier input validated on hardware in both states: clean stops -> canFlashAfterStop() true; forced/timeout stops log OTA blocked / OTA flashing withheld (= canFlashAfterStop() false). Confirmed mayBeginFlash() == (Stopped && !_stop_timed_out); a fresh start clears the latch; a dirty stop still permits restart.
• Barrier action (simple_repeater aborts+resumes vs. proceeds) is a one-line gate on that latch (MyMesh.cpp deferred-OTA site) -- validated by code + the Phase 4 host lifecycle tests.
• Not exercised end-to-end on hardware: the live ota update deferred-flash path. A plain pio run build reports ERR: OTA not configured (build via build.sh), so otaFromManifest bails before scheduling; driving it needs a build.sh firmware + a controlled manifest, and was not improvised on working fleet nodes (risk of flashing a real fleet build). Recommend a dedicated bench run for this.

Phase 7 -- fault-injection matrix (V3 non-PSRAM; representative, not the soak)

Matrix: 10 clean stop/start + 5 forced 2-slot teardowns + 8 slot-reconfigure + 4 down-broker (wss://192.0.2.1) flaps. Time series of free/largest-block/ queue/outbox recorded per step; reboot/crash detection on.

• No crash across the entire matrix (incl. 5 repeated force-kill fallbacks) -- the reviewed dirty-stop fallback is robust on non-PSRAM.
• Forced teardown reproducible: every 2-slot stop timed out (acks 7.4-8.2 s), reinforcing the primary finding.
• Bounded fragmentation, fully recoverable: slot-level reconfigure/flap churn (which keeps a disabled slot's persistent client until a full bridge restart) dropped the largest free block from ~125 k to ~72 k and held ~17 k of free heap, but it plateaued (stable across 12 churn iterations, not an unbounded leak) and a reboot fully restored ~141 k free / ~125 k largest block. This is pre-existing bridge behavior (TLS-context lifecycle), not a Phase 5 regression.

Phase 7 -- forced-teardown stress (V4 PSRAM, 3 wss slots, this branch)

8x end()/begin() cycles at the node's normal 3-slot config (every stop exceeds 8 s -> forced/dirty):

• All 8 forced, no leak, no crash. Largest free block rock-stable at 139 252 across all 8 cycles (mbedTLS is PSRAM-allocated, so internal-heap fragmentation is absent on the PSRAM path); free internal heap flat. The force-kill fallback is robust on PSRAM too.
• Loop-task stall is severe on the forced path: end() blocked the loop task ~15-27 s per stop, because the forced path waits the full 8 s timeout and then runs a complete redundant teardown on Core 1 (~ 8 s + ~16 s). During that window the repeater's mesh/CLI/radio servicing is stalled. Sizing the timeout so the cooperative (Core-0) teardown completes cleanly removes both the OTA-withhold and this double-teardown stall.
• Both nodes left restored to their original config, bridge on, heap healthy.

Limitations / not covered

• Task stack high-water mark and explicit task/client counts are not exposed by the CLI; a dedicated MQTT_MEMORY_DEBUG build is needed for those (heap stability was used as a proxy leak signal, and it held).
• The 72 h / 7-day soaks were not run (bounded representative cycles per agreed scope); WiFi loss/recovery, TLS-handshake failure, queue saturation, JWT/NTP failure, and millis()-rollover scenarios need external network control / time injection and were not driven over serial.
• Both devices left restored to their original slot config with the bridge on.

Current Baseline

The current branch has:

• Native tests (GoogleTest, [env:native]) for MQTT presets, validation, topic templates and routing, connection policy, packet-queue policy, payload construction, WebConfig keys, the WebConfig batch state machine, the MQTT lifecycle/teardown seam, the /mqtt_prefs codec, the atomic prefs store, the runtime-buffer lifecycle, and upstream Utils::toHex and mesh-table behavior. 15 suites as of the 2026-07-19 upstream merge.
• ArduinoJson pinned to 7.4.3 across the native and all firmware environments, enforced in CI by scripts/check_arduinojson_pin.py.
• PR CI (.github/workflows/) that runs the native suite and compiles both representative MQTT observer smoke builds: Heltec_v3_repeater_observer_mqtt (non-PSRAM) and T_Beam_S3_Supreme_SX1262_repeater_observer_mqtt (PSRAM).
• Symmetric PSRAM runtime buffers: begin() allocates through allocateRuntimeBuffers() and end() frees and nulls through releaseRuntimeBuffers(), so raw-data caching and the PSRAM JSON buffers are restored after a restart (MQTTRuntimeBufferLifecycle.h).
• A versioned /mqtt_prefs loader extracted into fork-owned, host-tested seams (MQTTPrefsCodec.h, MQTTPrefsStorage.h, MQTTPrefsAtomicStore.h) that classifies every deployed layout, preserves an unknown or newer file without overwriting it, and rejects corrupt input without out-of-bounds reads.
• WebConfig request/result correlation, failed-batch reboot gating, and a process-lifetime HTTP listener that avoids deleting an object still referenced by an asynchronous request.
• Pure MQTT policy helpers that reduce decision duplication while leaving the production bridge as the integration point.
• A wired WebConfig batch/reboot/stop state machine: WebConfigServer.cpp calls WebConfigBatch.h directly for POST classification, drain pacing and all-ok accumulation, reboot scheduling/firing, result classification, confirm-reboot arming, and stop gating; MAX_BATCH/STOP_WARN_MS alias the spec constants. The host tests therefore cover production, not a parallel copy.

This is a solid guardrail and unit-test foundation, but it does not yet validate cross-core state ownership beyond the Phase 5 stop path, the WebConfig batch machine over real HTTP, or long-running heap behavior. Those are the remaining phases.

Constraints

1. MQTT preference-file layouts are fleet-critical. Unknown newer formats must not be overwritten, and every supported old layout must remain recoverable.
2. Upstream MeshCore changes are merged regularly. Prefer additive fork-owned helpers and small adapters over reorganizing upstream-heavy files.
3. The non-PSRAM profile is the memory-pressure baseline; the PSRAM profile is a separate allocation/lifecycle path and must also be tested.
4. Risky lifecycle changes require fast deterministic tests before hardware soak testing.
5. Preserve observable behavior unless a behavior change is explicitly named, reviewed, and tested.

Change-Control Discipline (Stop-and-Ask)

This roadmap is executed incrementally, often by an agent working one phase at a time. Each unit of work is scoped to its assigned phase. When work uncovers something outside that scope, the correct action is to stop and ask, not to fix forward. Silent scope expansion is the primary way a targeted stability change turns into an unreviewed refactor or a needless merge-conflict surface. This section is binding on anyone -- human or agent -- executing the plan.

Stop and ask before writing code when any of these is true

• A refactor larger than the current phase authorizes appears necessary -- for example touching task lifecycle, client lifetime, cross-core ownership, or the queue backends when the phase did not name them.
• A bug is discovered that is not described in the plan, especially one affecting persistence, teardown, OTA, or heap behavior.
• A phase premise turns out to be false or already implemented (for example, a described defect that is already fixed). Report the discrepancy and get the plan re-baselined before writing code against a stale assumption.
• A change would alter observable behavior the phase did not explicitly name, review, and test (Constraint 5).
• Work would touch upstream-heavy files (MQTTBridge.cpp, CommonCLI, the role-specific MyMesh files) beyond a small, additive adapter (Constraint 2).
• A change touches fleet-critical persistence layouts, or could overwrite or discard an unknown or newer /mqtt_prefs file (Constraint 1).
• Two phases would be combined in one change, or a "Explicitly Deferred Debt" item would be pulled forward.
• A fix spans multiple files, changes a public seam, or would surprise a reviewer expecting only the phase's stated work.

What "stop and ask" means in practice

• Do not implement the out-of-scope change in the same pass. Record it.
• Report it with evidence: file and line references, a concrete reproduction or failure scenario, and the specific invariant or constraint at risk.
• Present options with tradeoffs and a recommendation, then wait for an explicit decision on scope before proceeding.
• File a newly found bug as its own item. Do not fold an opportunistic fix into an unrelated phase's commit -- single-purpose commits are required, and an unexpected fix deserves its own review.

What may proceed without asking

• Work squarely inside the assigned phase and its acceptance criteria.
• Small, behavior-preserving fixes fully contained in fork-owned helper files with matching host tests, where a reviewer would expect them as part of the phase.

When uncertain whether a change is in scope, treat it as out of scope and ask. The cost of a question is a round trip; the cost of an unreviewed lifecycle or persistence change across a fleet of thousands of devices is not.

Sequenced Work

Phase 0: Record the pre-change lifecycle characterization

Status: Not started -- the next actionable step, and prerequisite for Phases 4 and 5.

Capture current behavior before changing shutdown mechanics. This provides a reference for the lifecycle fakes and lets the later state-machine refactor prove equivalence rather than relying on memory or a multi-hour soak.

Characterize at least:

• Normal begin() -> connect -> end() ordering.
• end() while disconnected, connecting, connected, publishing, retrying, and applying a slot reconfiguration.
• Which callbacks can arrive during and after end().
• Queue disposition and connection/status counters across stop and restart.
• Heap, largest allocatable internal block, task stack high-water mark, and client/task counts before start, after start, after stop, and after restart.
• Partial initialization failures: queue allocation, task creation, client allocation, and PSRAM-buffer allocation.

Use instrumented hardware logging where necessary, but encode every behavior that can be represented deterministically into the lifecycle fake tests in Phase 4. Store representative logs as CI or test artifacts rather than enabling permanent high-volume production logging.

Phase 1: Put MQTT/WebConfig firmware smoke builds in PR CI

Status: Complete on this branch. PR CI already compiles both smoke builds and runs the native suite and the ArduinoJson pin check. Still pending against the acceptance criteria: build-size artifact/threshold reporting (criterion 4) and the optional ASan/UBSan native job. The description below is retained as the record of intent.

The PR build matrix historically did not compile an MQTT observer target; it now compiles two required smoke builds for changes touching firmware, variants, PlatformIO configuration, MQTT/WebConfig helpers, or their tests:

• Heltec_v3_repeater_observer_mqtt for the constrained non-PSRAM path.
• T_Beam_S3_Supreme_SX1262_repeater_observer_mqtt for the PSRAM path.

Keep the native suite required. Add ASan/UBSan to a separate native job if the PlatformIO native toolchain supports it reliably. Record static RAM and flash usage and fail only on reviewed limits with enough headroom to avoid noisy one-byte regressions.

Acceptance criteria:

• Pull requests cannot merge when native tests or either representative MQTT build fails.
• The build log shows ArduinoJson 7.4.3 for both firmware profiles.
• CI checks that every ArduinoJson declaration remains pinned to 7.4.3.
• Build-size output is retained as an artifact or job summary.

Phase 2: Fix PSRAM restart resource symmetry

Status: Complete on this branch. The asymmetry described in the original plan has been fixed; this section now documents the intended design and the tests that guard it, not outstanding work.

PSRAM-backed raw-data and JSON buffers are allocated in begin() through allocateRuntimeBuffers() (idempotent, allocate-if-missing), freed and nulled by end() through releaseRuntimeBuffers(), and reallocated by a later begin(). Raw-data caching and the PSRAM JSON buffers therefore survive a restart; the task-stack buffer is used only when a live allocation actually fails. The pure helpers live in MQTTRuntimeBufferLifecycle.h, covered by test_mqtt_runtime_buffer_lifecycle.

The design keeps symmetric runtime-resource operations:

• allocateRuntimeBuffers() called by begin() or a shared initialization path.
• releaseRuntimeBuffers() called by end() and initialization rollback.

The operations must be idempotent, handle partial allocation, and preserve the existing graceful fallback when PSRAM allocation fails. Avoid changing client or task shutdown behavior in this phase.

Tests and validation:

• Allocation success, partial failure, repeated allocation, and repeated release using a narrow allocator fake.
• Multiple begin()/end() cycles do not lose the raw-data buffer or permanently move JSON serialization onto the task stack.
• Initialization failure releases only resources owned by that attempt.
• Representative PSRAM and non-PSRAM firmware builds pass.
• A short hardware restart loop shows stable free heap and largest free block.

Phase 3: Add binary MQTT preference migration fixtures

Status: Complete on this branch. The versioned loader, the frozen-layout static_asserts, and the migration/fixture coverage described below are implemented in MQTTPrefsCodec.h, MQTTPrefsStorage.h, and MQTTPrefsAtomicStore.h, and host-tested by test_mqtt_prefs_codec and test_mqtt_prefs_atomic_store. Unknown-newer files are preserved and corrupt input is rejected without overwrite. Residual: the concrete filesystem adapter and the load/saveMQTTPrefs() orchestration still live in CommonCLI.cpp (platform-coupled by design); keep that adapter small. The description below is retained as the record of intent.

Build deterministic tests around the versioned /mqtt_prefs loader before upstream merges change CommonCLI, filesystem behavior, or preference structs. Use checked-in, non-secret binary fixtures or byte arrays representing each deployed layout.

Required cases:

• Pre-slot layout.
• Three-slot layout.
• Legacy headerless six-slot layout.
• Shorter version-1 payload with newer fields defaulted.
• Current version round trip.
• Truncated header and truncated payload.
• Invalid magic and implausible payload length.
• Unsupported newer version: run with safe defaults and preserve the original file without overwriting it.
• Migration failure or interrupted save leaves a recoverable source file.
• Existing credentials, slot ordering, and publish flags survive migration.

Prefer extracting a fork-owned serializer/decoder seam over host-compiling all of CommonCLI. Keep the production adapter small and retain the frozen-layout static_asserts.

Acceptance criteria:

• All known deployed layouts have a fixture and field-by-field expected result.
• Corrupt input cannot cause an out-of-bounds read or silent overwrite.
• A future MQTTPrefs layout change fails tests until its migration and fixture are added deliberately.

Phase 4: Establish ownership and teardown test seams

Status: Seams, ownership doc, and teardown tests landed; production rewiring deferred to Phase 5 by explicit decision (scope: "seams + tests only"). The fork-owned pure lifecycle state machine and narrow dependency seam (src/helpers/MQTTLifecycle.h), the teardown-focused test matrix (test/test_mqtt_lifecycle/), and the ownership model (MQTT_OWNERSHIP.md) are in place. MQTTBridge.cpp was intentionally left untouched to keep the merge-sensitive file free of churn until the Phase 5 change lands as one reviewable unit. The invasive production work -- publishing the plain-data snapshot and repointing consumers at it, replacing the volatile handshakes with a command queue / task notifications, and the cooperative-shutdown end() rewrite with a begin() double-call guard -- is carried into Phase 5.

Verified premise (with Phase 4 refinements): the loop task, WebConfig, CLI, and AlertReporter currently read the MQTT task's live, mutable slot objects and client counters cross-core without a lock or a published snapshot. getSlotStatusSnapshot() is built on demand from live state despite its name -- the refactor must actually publish a plain-data snapshot, not assume one exists. Refinements found while mapping the code (see MQTT_OWNERSHIP.md for file:line references):

• The snapshot's name/state const char*s point at static rodata, so they are not dangling; the real hazards are reading the mutable slot.preset pointer value (Core 0 can null/reassign it) and slot.client->getPublishOk() (a client pointer Core 0 can delete during teardown).
• The live-bridge web reads are in the MyMesh.cpp app layer (buildStatsJson), not WebConfigServer.cpp, which reads only a compile-time constant.
• end() clears only the s_mqtt_bridge_instance singleton; the app's bridge pointer and AlertReporter::_bridge are not cleared, so instance reads can touch a torn-down bridge.

This phase is the safety net for cooperative shutdown. It must land before the shutdown state machine.

Ownership model

Document one owner for each mutable runtime domain:

• MQTT task: clients, slot connection state, NTP client operations, publish counters, and the packet drain path.
• Loop task: CLI execution, preference persistence, WebConfig batch draining, and bridge lifecycle requests.
• Producer/radio context: packet staging before queue handoff.
• Async TCP context: request parsing and immutable response handoff only.

Replace cross-core volatile handshakes with an appropriate primitive:

• Task notifications or a command queue for one-way lifecycle/reconfigure/NTP requests.
• Atomics only for truly independent scalar state.
• Immutable published snapshots for WebConfig, CLI diagnostics, and alerting.
• A mutex only where ownership transfer or snapshot publication cannot express the operation cleanly.

In particular, loop/WebConfig code must not directly inspect mutable MQTT slot objects or client counters. The MQTT task should publish a plain-data status snapshot.

Narrow lifecycle fakes

Introduce interfaces or callbacks for only the dependencies needed to drive the lifecycle deterministically:

• Clock/timer.
• Task start, stop request, acknowledgment, and timeout.
• MQTT client connect/disconnect and delayed callbacks.
• Queue depth/pop/requeue behavior.
• Runtime allocator and heap measurements.
• OTA coordinator/barrier.

Required teardown-focused tests:

• Stop while connecting, connected, publishing, retrying, renewing a token, running NTP, and applying a slot change.
• Callback delivered before stop, during stop, after disconnect, and after the stop acknowledgment.
• Duplicate stop, stop before full initialization, and restart after stop.
• Timeout/fallback behavior when the MQTT task or client does not acknowledge.
• No client, queue, buffer, or task access after its owner releases it.
• Current queue and diagnostic behavior is preserved unless a change is explicitly approved.

OTA Teardown Barrier: release-critical scenario

Status: Not started. This is the fix for a known shipping crash, not a hypothetical hardening target. With a broker down over wss, the abrupt vTaskDelete in end() can kill the MQTT task inside mbedTLS; destroySlotClients() then frees client buffers on a possibly-corrupted heap and OTA begins flashing with no barrier -- the observed teardown heap panic. There is no coordination today between MQTT shutdown and flash writing beyond straight-line ordering on the loop task.

Treat OTA teardown as a first-class test target throughout Phases 0, 2, 4, 5, and 7. It is not merely another restart case.

The required invariant is: firmware erase/write must not begin until MQTT shutdown has reached a safe acknowledgment point, and the bridge must not be restarted while flash writing is active.

Required scenarios:

• OTA requested while MQTT is connecting, publishing, draining retries, or handling a callback.
• MQTT stop succeeds and OTA begins only after the teardown barrier.
• MQTT stop times out: OTA aborts safely rather than writing under uncertain ownership.
• OTA preflight or download aborts before flashing: the bridge restarts once and returns to the prior configured behavior.
• Successful OTA: no bridge restart is attempted before the device reboots.
• Power loss/reset at the platform-supported OTA boundaries retains a bootable partition; this portion requires hardware/platform validation.
• Repeated failed OTA attempts do not leak heap, duplicate WiFi callbacks, or leave the bridge permanently stopped.

Retain the prior teardown/heap-panic reproduction as a regression artifact if available. OTA-related lifecycle tests are release gates for any future change to bridge teardown, OTA sequencing, MQTT client lifetime, or task ownership.

Phase 5: Implement cooperative MQTT shutdown

Status: Minimal cooperative shutdown implemented on branch phase5/cooperative-mqtt-shutdown (scope: "the smallest change that fixes the OTA teardown panic as one reviewable unit"). Native suite green; the non-PSRAM observer firmware smoke build compiles. NOT yet hardware-validated -- that is the Phase 7 gate -- and the stop timeout is a Phase-0 placeholder (see below).

What landed (wiring the Phase 4 MQTTLifecycle state machine into the bridge):

• MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task (Core 1) from begin()/end(). A nested LifecycleOps binds the pure, host-tested Ops spec to FreeRTOS/PsychicMqttClient.
• end() no longer blind-vTaskDeletes. It requests a cooperative stop; the MQTT task (Core 0) sees a new volatile _stop_requested, tears down its own clients on Core 0 where the mbedTLS contexts live, sets _stop_acked last, and self-terminates. end() waits (bounded) for the ack, then frees the queue/buffers. This removes the "kill the task mid-mbedTLS, then free client buffers on a corrupted heap" teardown path.
• Bounded stop timeout -> reviewed fallback: on timeout the task is force-killed and torn down on Core 1 (the old behavior), but a dirty latch is set so canFlashAfterStop() is false and OTA flashing is withheld.
• begin() has a double-call guard and syncs the Coordinator to Running.
• OTA teardown barrier: simple_repeater's deferred-OTA fire site aborts and resumes the bridge unless the preceding end() reported a clean stop.

Deferred (kept out of this reviewable unit; carried to a Phase 5b / Phase 6): replacing the volatile NTP/reconfigure handshakes with a command channel, and publishing a plain-data status snapshot to repoint the Section 1/Section 2 consumers in MQTT_OWNERSHIP.md (the AlertReporter/buildStatsJson instance-pointer reads that can still touch a torn-down bridge). Those are not required to fix the OTA panic and would enlarge the merge-sensitive diff.

Phase 0 dependency -- CHARACTERIZED + FIXED 2026-07-19: the flat MQTT_STOP_TIMEOUT_MS = 8 s placeholder was too small (measured per-wss-slot teardown ~5-6 s sequential -> ~11-12 s at 2 slots, ~16 s at 3, ~27-30 s at 5; at 8 s healthy multi-slot stops tripped the dirty fallback and the OTA barrier withheld flashing). Replaced with a slot-scaled timeout set per stop in end(): 5 s + 8 s x enabled_slots (MQTT_STOP_TIMEOUT_BASE_MS / MQTT_STOP_TIMEOUT_PER_SLOT_MS, applied via Coordinator::setStopTimeoutMs()). Hardware-verified: a 2-slot stop that force-timed-out pre-fix now acks clean in ~11.7 s within the 21 s budget. See "Hardware Characterization Results (Phase 0 & Phase 7)".

The original plan of record follows. Replace direct task deletion with an explicit lifecycle such as:

Stopped -> Starting -> Running -> StopRequested -> Stopping -> Stopped

The exact representation can differ, but it must provide:

• Idempotent start and stop requests.
• A stop request delivered through the established ownership channel.
• Cessation of new connects, publishes, retries, and reconfigurations.
• Ordered client/service shutdown on the MQTT task.
• A completion acknowledgment before the loop task releases queues, buffers, or other shared resources.
• A bounded timeout with clear diagnostics and a deliberately reviewed fallback.
• Safe restart after a completed stop.
• An OTA barrier that consumes the same completion acknowledgment.

Do not combine this phase with queue-loop deduplication, broad bridge cleanup, or unrelated feature changes.

Acceptance criteria:

• All Phase 4 lifecycle tests pass, including delayed/stale callbacks.
• Characterized behavior from Phase 0 is preserved or differences are explicitly documented and approved.
• OTA teardown-barrier tests pass.
• Repeated hardware start/stop cycles show no downward heap or largest-block trend and no task/client-count growth.

Phase 6: Expand request, queue, connection, and publication integration tests

Status: Partial -- branch phase6/integration-tests (draft PR, base phase5). The remaining inline decision points that were host-testable have been extracted into the pure policy seams and covered:

• WiFi STA reconnect backoff moved out of handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,wifiReconnectDue, nextWifiBackoffAttempt} (behavior-preserving; adversarially reviewed for rollover/boundary equivalence) with test_mqtt_connection_policy cases.
• The (packet, raw) publication-outcome pairing named as MQTTPacketQueuePolicy::queuedPacketPublished() and wired at both queue-drain sites, with test_mqtt_packet_queue_policy cases (partial success = completed).
• MQTTPublicationType values frozen in test_mqtt_topic_router; the bridge-side MQTTMessageType alignment was already a compile-time static_assert.

The WebConfig POST/result/reboot/stop state machine (the largest gap) has a pure, host-tested spec -- src/helpers/WebConfigBatch.h + test/test_webconfig_batch/ -- and as of 2026-07-19 it is wired: WebConfigServer.cpp calls it at every decision point, so the spec is load-bearing and cannot drift from production. MAX_BATCH and STOP_WARN_MS alias kMaxBatch/kStopWarnMs for the same reason.

Two asymmetries between spec and caller are deliberate, and are documented in the header so a future reader does not "simplify" them back:

1. finishRebootAt() returns 0 for "no reboot scheduled", but the caller only ASSIGNS _reboot_at when the result is non-zero. _reboot_at is not solely batch-owned -- the manual /api/reboot route arms it from the async_tcp task, possibly while a batch is still draining -- so an unconditional assign would silently cancel a manual reboot. This was caught during the wiring, not by a test; the host suite does not model the two owners of _reboot_at, which is a coverage gap worth closing.
2. classifyPost() is consulted in two phases, because the change count is only known after the set map is parsed, and parsing must not precede the Replay/Busy answer (a replayed POST carrying a bad key must still get its 202).

Verification status of the wiring: native suite and both smoke builds green; V3 hardware shows a clean WebConfig AP start/stop (the rewired stopStep path takes the Finalize branch with no handler-wait warning). Not verified: the POST -> drain -> result -> reboot sequence over real HTTP. LAN mode needs an admin-password login and the setup AP needs a client associated to the device's SoftAP; neither was driven. Given that this server was originally tuned against real iOS captive-portal behavior, HTTP caching, and route ordering, treat an end-to-end portal save as a required check before this ships.

Still open (each a good follow-up PR): the end-to-end HTTP exercise above, and the queue-orchestration behaviors (FIFO ordering, evict/requeue-failure interplay, the two adapters' drop-vs-keep-head divergence), which need a fake-queue harness. The original scope list follows.

After lifecycle ownership is stable, broaden deterministic integration coverage:

• WebConfig POST/result/reboot/stop behavior, including lost responses, duplicate request IDs, concurrent clients, partial command failure, and stop with an active handler.
• MQTT queue behavior for FreeRTOS and circular-buffer adapters: overflow, delayed retry, requeue failure, stale flush, queue ordering, and millis() rollover.
• Packet succeeds/raw fails and raw succeeds/packet fails.
• Connection backoff, stable reset, breaker probe, WiFi recovery, token renewal, and slot reconfiguration callback ordering.
• Topic and payload contracts at the bridge boundary, so enum/cast/adaptation mistakes are covered in addition to the pure helper tests.

Once both queue backends are covered by the same behavioral contract, consider extracting a shared processQueuedPacket() that returns an outcome while each backend retains pop/requeue/dequeue ownership. Do not make queue deduplication a prerequisite for the stability work.

Phase 7: Establish uptime, memory, and fault-injection gates

Status: Not started. The final validation gate; runs after the lifecycle and OTA-barrier work is in place.

Use hardware soak tests to validate the already-tested design, not to discover basic lifecycle errors for the first time.

Run at least one constrained non-PSRAM board and one PSRAM board through:

• Stable broker operation.
• Broker unavailable, rejecting authentication, and flapping.
• WiFi loss/recovery and credential changes.
• TLS handshake failures and repeated reconnect backoff.
• Queue saturation and slow broker behavior.
• Repeated slot reconfiguration and full bridge stop/start.
• WebConfig start/save/stop cycles.
• JWT renewal and NTP failure/recovery.
• OTA success, preflight abort, download failure, teardown timeout, and resume.
• At least one test crossing the 32-bit millis() rollover boundary, accelerated where hardware time injection is available.

Record and threshold:

• Minimum free internal heap.
• Largest free internal block.
• MQTT task stack high-water mark.
• PSRAM free/largest block where available.
• Queue and outbox high-water marks.
• Connect/disconnect/retry/publish counters.
• Watchdog and reset reason.
• Task/client counts across restart cycles.

Recommended gates are a 72-hour fault-injection run followed by a seven-day stable run. Store machine-readable time series and a concise summary artifact. Avoid enabling high-volume diagnostic logging in production builds.

Branch and Release Channels

observer-firmware-dev is the standing development line, not a one-off merge branch. It is where upstream merges land and where the dev/beta firmware channel is built from; it is promoted into webconfig (and onward to the flex mainline) when its contents are ready to ship. Future upstream merges land ON this branch rather than creating a new dated branch each time.

Two firmware channels are published, fully separated so a node cannot cross between them by accident:

	Production	Dev/Beta
Source branch	flex mainline	observer-firmware-dev
Workflow	build-observer-firmwares.yml (push-triggered)	build-observer-firmwares-beta.yml (manual dispatch)
Release tag	observer-mqtt-latest	observer-mqtt-beta-latest
OTA manifest base	observer.gessaman.com/v	observer.gessaman.com/beta/v
Download host	observer-fw.gessaman.com	observer-fw-beta.gessaman.com
Flasher config	config.json (default)	config-beta.json (?config=config-beta)
Embedded version	v1.16.0.N-observer-<hash>	v1.16.0.N-observer-beta-dev-<hash>

The channel is baked into each binary as OTA_MANIFEST_BASE (injected by build.sh, overridable via OTA_MANIFEST_BASE_URL), so a node only ever receives OTA updates from the channel it was flashed from. Both channels deliberately share FIRMWARE_VERSION: the OTA logic treats a differing base version as "always an update", so channels must separate by manifest URL, never by base version.

Upstream Merge Record -- 2026-07-19 (observer-firmware-dev)

First upstream/dev merge since the v1.16.0 base (8c0d5c5b, 2026-06-06): 191 upstream commits, 14 conflicted files, ~18 hunks. The fork was 349 commits ahead. Recorded here because the next merge starts from these resolutions (all captured in rerere).

The finding that matters most: /com_prefs is safe to reorder

Upstream moved rx_boosted_gain and path_hash_mode to the tail of struct NodePrefs, while the fork holds them mid-struct. This looks like a fleet-critical layout divergence (Constraint 1) and was analyzed as one before resolving. It is not:

/com_prefs is serialized field-by-field at explicit byte offsets, not as a struct dump. NodePrefs member order is in-memory only and has no effect on the file. The fork's extracted writeCommonPrefsImage() was verified byte-identical to upstream's inline writer at every offset -- 79 (pad), 121, 122, and 290-294 -- so either struct order produces the same image. No migration was needed, and none should be invented for this in future merges.

Corrected while here: a comment claiming rx_boosted_gain lives at /com_prefs offset 79. Offset 79 is a pad; the field is written at 290. The comment would have misled exactly the analysis above.

Note the asymmetry this creates: /mqtt_prefs (fork-owned) IS layout-critical and versioned; /com_prefs is offset-addressed and tolerant of struct reordering. Do not generalize one file's rules to the other.

Resolutions

• CommonCLI.h -- kept the fork's NodePrefs (a superset) and adopted upstream's setRxBoostedGain(bool) -> bool signature change, which upstream's CommonCLI.cpp now uses to report "unsupported". A real semantic API change, exactly the kind the discipline list warns can hide behind a clean merge.
• CommonCLI.cpp -- kept the fork's legacy /com_prefs migration block and the writeCommonPrefsImage() call.
• UITask.cpp -- genuine three-way merge: upstream's drawTextCentered and powering-off screen, plus the fork's WITH_WEBCONFIG portal/reboot screens.
• ESP32Board.cpp, MeshCore.h, platformio.ini -- keep-both (fork OTA additions alongside upstream powerOff/enterDeepSleep and Packet.cpp).
• MicroNMEALocationProvider.h -- took upstream's claim()/release() and added the _claims member they depend on.
• MyMesh.cpp/.h (repeater + room server) -- kept the fork's superset defaults.
• Removed duplicate declarations that auto-merge produced without conflicting: RadioLibWrapper::_cad_enabled and MyMesh::getCADEnabled(). These compiled only after being caught by the build, not by Git -- a reminder that a conflict-free merge is not a correct merge.

Verification

Native 15/15 (including upstream's new test_mesh_tables), both MQTT smoke builds, ArduinoJson pin check. On V3 hardware (non-PSRAM, 1 wss slot): clean boot with /com_prefs values intact across the flash (the end-to-end proof of the NodePrefs resolution), WiFi -> NTP -> MQTT1 connect -> status published, Free=137612 Max=124916 matching the pre-merge healthy baseline, and a clean cooperative teardown -- 1 enabled slot(s), timeout 13000 ms -> cooperative stop -> stop acknowledged (clean) -> Bridge stopped (clean). Phase 5 and the slot-scaled timeout both survive the merge.

Not verified: the V4/PSRAM path, and the WebConfig HTTP batch machine (see Phase 6).

Cost signal for scheduling the next merge

Upstream's deltas to the hot files were small (CommonCLI.cpp 165 lines, MyMesh.cpp 70, CommonCLI.h 26) against 107 and 61 fork commits on those same files. The fork is the churn source, not upstream -- so merge cost scales with how much fork work accumulates between merges, not with upstream velocity. Six weeks of drift cost roughly a half-day. Merge after each phase lands rather than batching.

Continuous Upstream-Merge Discipline

Apply these practices throughout every phase:

• Keep fork-owned logic in additive helper files and keep adapters in upstream-heavy files small.
• Avoid drive-by formatting, renames, and unrelated cleanup in MQTTBridge.cpp, CommonCLI, and the role-specific MyMesh files.
• Use single-purpose commits with messages that describe their full scope.
• Merge upstream frequently enough that conflicts remain attributable.
• Before each upstream merge, record the native-test, representative-build, firmware-size, and persistence-fixture baseline.
• After the merge, run native tests, both MQTT smoke builds, preference fixtures, and the relevant lifecycle/OTA tests before resolving the merge as complete.
• Review semantic behavior at adapters even when Git reports no textual conflict; upstream signature, lifetime, and task-context changes can invalidate fork assumptions silently.
• Reuse resolutions only after revalidating them against the new upstream code.
• A conflict-free merge is not a correct merge: the 2026-07-19 merge produced two duplicate declarations that Git resolved silently and only the compiler caught. Always build both smoke targets before treating a merge as done.
• Extract repeater/room-server WebConfig ownership into a shared fork helper when that integration next requires material change; do not perform a standalone broad move solely for aesthetics.

Explicitly Deferred Debt

These items are recognized but are not prerequisites for the sequenced stability work:

• Shared packet-drain processing between FreeRTOS and circular-buffer backends.
• Repeater/room-server WebConfig integration duplication.
• Moving the WebConfig HTML generator out of the common ESP32 build path.
• The secret-placeholder edge case for a credential exactly equal to the UI sentinel.
• Secure setup-AP authentication beyond the currently documented open/optional PSK threat model.

Revisit an item when its code is already being changed, when operational data raises its priority, or when tests make the refactor substantially safer.

Completion Definition

This roadmap is complete when:

• PR CI protects native logic and both representative firmware memory paths.
• Every deployed MQTT preference layout has a passing migration fixture.
• Runtime resources survive repeated start/stop cycles symmetrically.
• Cross-core ownership is explicit and diagnostics consume immutable snapshots.
• Cooperative shutdown passes deterministic lifecycle and OTA-barrier tests.
• Hardware fault-injection and stable soaks meet reviewed memory, stack, task, and reliability thresholds.
• Upstream merges use the same automated gate and do not require broad rewrites of fork-owned behavior.