Merge remote-tracking branch 'origin/keymindCascade' into keymindCascade

docs/lora_ota_automation.md

@@ -964,6 +964,25 @@ bypasses signature/install checks. It does **not** fix later block-retry
 stalls on old firmware or relays beyond the controller's direct radio reach;
 the firmware update is the permanent fix.
 
+For a short path, `--legacy-relay-reboot-recovery` offers a lower-airtime
+alternative after that same exact `0/0` timeout. It requires one managed relay
+whose saved normal-channel and active temporary-channel routes are each **zero
+or one hop**, plus `--source-shares-controller`. Before any reboot, the runner
+checks the relay and destination identities, the exact failed manifest, and
+at least eight minutes of remaining transfer and target TempRadio time. It
+asks for a separate confirmation at the failure (or requires both this flag
+and `--yes` in unattended use). The relay's `reboot` CLI intentionally sends
+no reply: the runner returns the shared source/controller to normal radio,
+authenticates the relay's exact key there, re-arms only the remaining portion
+of the target's TempRadio lease, then proves the relay, destination and same
+manifest on the temporary channel before one pull retry. If the normal path
+cannot be proved after reboot, it stops instead of assuming the reboot worked;
+the other temporary leases still expire. A reboot also clears the relay's
+temporary radio setting, so manually rebooting it during a transfer without
+this re-arming sequence will break the temporary path. When both legacy
+recovery options are enabled, the reboot choice is offered first; declining
+it leaves the direct-cache workaround available on a zero-hop route.
+
 Read-only and replay-safe transmissions retry up to three times. Three retries
 or 90 seconds, whichever comes first, opens a 10-second stop-or-continue
 prompt when stdin is an interactive terminal. Continue remains the default on

docs/rak3401_1.17.1.7_legacy24.md

@@ -0,0 +1,272 @@
+# RAK3401: 1.16.07 to 1.17.1.7 without a bootloader update
+
+Qualified on MercerWoodMesh on 27 September 2026: all 15 application updates
+passed over LoRa, from the exact supplied 1.16.07 image, without a bootloader
+update. Use `RAK3401-1.16.07-to-1.17.1.7-legacy24-mOTA.zip` and verify it against
+the accompanying `.zip.sha256` file before extracting. Follow the exact
+before/after hashes in its `CHAIN.csv`; this is not a single-step stock update.
+
+## Exact compatibility boundary
+
+This is an application-only migration for target `2FA509C1`, hardware
+`RAK_3401_repeater_lora_ota_no_external_sensors`. It is not a generic RAK4631
+package. The installed application must have body hash `71F4026CBE4B8B74`
+(`1.16.07`, full image SHA-256
+`4301fc63ebd661c70f9bc40e7eca44d3cb8a358cb3c0075e700d9676a57478cb`).
+
+The bootloader remains `0.9.2-OTAFIX2.4`, with ABI 2 and codec mask `0x4`.
+Its application staging ceiling is `0xD4000`. Every package is a codec-2
+CRLE in-place delta. DEFLATE is used only for transport by compatible
+intermediate applications; the old bootloader still applies CRLE.
+
+The destination is built from source commit
+`2d03e098cf75d58adcf35fe3d4dbbeaf0af114e1` with the explicit
+`MESHCORE_NRF52_LEGACY_FLASH_OTA=1` build selection. This preserves the
+ordinary S140 linker and disables hybrid flash/RAM staging and bootloader
+self-update. It is deliberately different from the stock hybrid 1.17.1.7
+release. The endpoint has:
+
+- Version: `v1.17.1.7-halo-keymind-cascade-legacy24-2d03e098`
+- Image: 521,496 bytes
+- Body hash: `D437D1F17AF5B9F3`
+- SHA-256: `c9e63a94f58f881b1853fc5d9d348cd75a6cdd454c5b572341d652a6141e3157`
+
+Do not replace this endpoint with the stock ZIP. Do not send a bootloader or
+SoftDevice update. The ZIP/UF2 files used to construct and qualify the chain
+are not themselves LoRa update containers.
+
+Future application updates for this node must also be qualified against the
+unchanged OTAFIX2.4 contract. Keep the legacy build selection; do not assume
+that a later stock release is an interchangeable target for this chain.
+
+## What "smallest" means
+
+The search compares an explicit, hash-pinned set of application images and
+every allowed page-aligned in-place workspace for each increasing-version
+transition. The starting bootstrap is fixed: its 89,844-byte package is the
+only admitted first edge. Synthetic bridge version numbers are real compiled
+versions below 1.17.1.7; their binaries are not relabeled after compilation.
+
+The `bytes` objective minimizes ideal serialized MeshCore traffic first,
+then install count. The `packets` objective minimizes ideal originated
+packets first, then serialized traffic. The older `transport` objective
+minimizes install count before traffic and is not equivalent to either.
+
+Accounting includes manifests, block requests, proof requests/responses,
+DATA fragments, and MeshCore framing. It uses the actual pinned encoder for
+compressed blocks. It excludes discovery, retries, admin/control commands,
+other mesh traffic, flood fan-out, and LoRa preamble/coding overhead. A fixed linear relay path
+can be accounted for separately. These are finite-search minima, not a proof
+that no possible custom firmware or protocol could ever use fewer bytes.
+The ideal compressed counts also assume negotiation remains in that mode.
+An older receiver can fall back to raw DATA after stalls; the physical-test
+transmission totals include that extra traffic.
+
+Best lossless byte count is not a guarantee of the fastest real radio link.
+Use the fastest preset that remains reliable over the actual route. SF5 /
+500 kHz is a direct bench measurement, not a recommendation to switch an
+unreachable field node or unconfigured relay chain to that preset.
+
+## Selected route
+
+The completed search covers 52 images and 47,920 flash geometries. The
+packet-first route has 15 packages, totaling 2,340,160 container bytes. Its
+ideal direct-link traffic is **12,621 packets / 1,808,961 serialized MeshCore
+bytes**. These totals include traffic in both directions, not just DATA.
+Fifteen is also the fewest installs among the searched candidates.
+
+The byte-first route uses the same sequence of application images, with
+four different patch workspaces: 12,625 packets / 1,808,787 bytes. Saving
+174 bytes costs four packets. The delivered packages use the packet-first
+choice; do not substitute packages from the comparison route.
+
+| Step | Resulting version | Container bytes | Ideal packets | Ideal MeshCore bytes |
+| ---: | --- | ---: | ---: | ---: |
+| 1 | 1.16.7.9 | 89,844 | 878 | 101,203 |
+| 2 | 1.16.7.10 | 62,838 | 551 | 70,031 |
+| 3 | 1.16.8.7 | 106,125 | 496 | 70,134 |
+| 4 | 1.16.9.105 | 149,542 | 728 | 104,992 |
+| 5 | 1.16.9.110 | 112,540 | 489 | 67,887 |
+| 6 | 1.16.9.111 | 71,070 | 268 | 35,453 |
+| 7 | 1.16.9.116 | 138,545 | 692 | 101,308 |
+| 8 | 1.16.9.122 | 177,536 | 995 | 148,671 |
+| 9 | 1.16.10.1 | 258,398 | 1,477 | 223,042 |
+| 10 | 1.16.11.0 | 245,426 | 1,450 | 219,953 |
+| 11 | 1.16.12.1 | 217,608 | 1,167 | 172,794 |
+| 12 | 1.16.13.1 | 199,395 | 1,024 | 150,005 |
+| 13 | 1.16.14.0 | 199,527 | 989 | 143,160 |
+| 14 | 1.16.15.0 | 139,188 | 618 | 86,721 |
+| 15 | 1.17.1.7 | 172,578 | 799 | 113,607 |
+
+Some bridge versions also exist as different binaries. `CHAIN.csv`'s exact
+body hashes and package checksums, not this version-only table, determine
+which package is valid at each step. The first two transfers use legacy
+1 KiB blocks; subsequent transfers use negotiated 2 KiB blocks with
+transport DEFLATE and the unchanged CRLE bootloader format.
+The second transfer also omits explicit proof requests: that historical
+receiver waits for the current sender's proactive Merkle proofs. The original
+1.16.07 receiver does not have this optimization.
+
+## Measured MercerWoodMesh replay
+
+All 15 selected packages passed in order on the Pi's debug-wired RAK3401,
+starting with the exact 1.16.07 image and OTAFIX2.4. Every application payload
+crossed LoRa from the Heltec T096. There were no debug writes, application
+substitutions, failed installs, or resumed downloads between the 15 steps.
+
+The direct bench used 909.950 MHz, BW 500 kHz, SF5, CR5, continuous receive,
+and a 128-symbol source preamble. Observations for both radios totaled:
+
+- 14,069 transmitted frames, including discovery and retries.
+- 22 minutes 31 seconds of measured download time.
+- 47 minutes 35 seconds summed active test-step time, including configuration,
+  checks, installs, and reboots; deliberate gaps between runs are excluded.
+
+These are bench measurements, not a field-link completion-time promise. The
+control commands were sent over USB; a remote field console adds LoRa control
+traffic. `BENCHMARK.csv` separates each step's ideal cost from its observed
+frame count and timing. Actual serialized radio bytes were not measured.
+
+Independent debug-port readback matched the complete final application,
+unchanged bootloader, and unchanged MBR/SoftDevice. UICR matched the normal
+post-startup baseline throughout this replay. The initial pre-first-boot
+UICR dump differs because normal Nordic/RAK startup configures the reset pins
+and 3.3 V regulator; this is documented in the evidence, not a bootloader
+upgrade.
+
+After a fresh reboot, the watchdog reported `on`. Both radios returned to
+910.525 MHz / BW 62.5 kHz / SF7 / CR5, and a nonce-matched normal-channel trace
+through the exact target key prefix completed in 423 ms. The inaccessible
+field node has not been touched. The lab RAK remains on this repeater image;
+the Pi's `mctomqtt.service` remains stopped because it previously owned that
+serial port. Radios other than the RAK3401 and sending Heltec T096 were left
+untouched.
+
+## Field preparation
+
+All destination commands below can be sent through the authenticated remote
+repeater console; USB/Bluetooth at the destination is not required. Keep a
+separate control connection available while the sender owns its companion's
+serial port. A local
+Full Companion radio attached to the sender runs the bundled `motatool`
+seeder. Resolve ports by device identity, not a remembered `ttyACM` number.
+
+The qualified source is the Pi's Heltec T096, USB serial
+`651F8E496197F882`, running Full Companion
+`v1.17.1.8-halo-keymind-cascade-dev-d4b3bb56`. It supports the extended
+2 KiB transport and host-backed DEFLATE. The destination is RAK3401 USB
+serial `0B81C9C68D8D01B4`. Older companion firmware or an older `motatool`
+is not an interchangeable sender for this bundle.
+
+Before any change, record `ver`, `ota self`, `ota status`,
+`get bootloader.ver`, `get system.watchdog`, `get radio.rxps`,
+`get powersaving`, `get rxdelay`, `get af`, and `ota config`. Match the exact
+starting body, hardware, bootloader, and signing-key policy. An unsigned
+bundle is not suitable for a node requiring signed packages.
+Choose a stable-power maintenance window long enough for all 15 installs;
+do not begin an install while the destination is at risk of losing power.
+
+Disable the destination system watchdog with `set system.watchdog off`.
+Allow its one-time hardware reset and wait at least 90 seconds; reconnect
+and prove that it reports `off`. A quick software reboot is not a substitute.
+
+Use continuous receive on the destination and sender throughout the bridge
+chain (`set radio.rxps off`). Record and disable a second scanning radio
+profile on the sender if it would take the radio away from the OTA channel.
+Record `rxdelay`, `af`, power-saving, reach, and relay settings before changing
+them. Use `hops 0` only when there are no intermediate relays. Obey applicable
+airtime/duty restrictions and restore all saved settings when finished.
+
+The direct bench also uses `set rxdelay 0` and `set af 0` on both radios,
+`ota speed 1` on the sender, and `set powersaving off` where supported.
+The earliest applications do not recognize the last command; continuous
+receive is still selected through `set radio.rxps off`. The source has
+`set radio2 off` throughout the transfer. Wait for configuration replies to
+finish before handing its serial connection to the seeder.
+
+For the qualified direct bench configuration only, the destination uses
+`tempradio 909.950,500,5,5,120`; the tested current Full Companion accepts
+`tempradio 909.950,500,5,5,120,128` for an explicit 128-symbol preamble.
+Historical destination parsers do not accept the sixth argument. The sender
+and destination must be on the same live radio preset. Do not assume the
+temporary preset survives a bridge reboot.
+
+The newer historical receivers performed poorly with a 16-symbol source
+preamble in this test. Use the qualified longer setting. Also avoid frequent
+`ota status` or `ota stats` polling: these versions rehash the complete running
+application while answering, which can interrupt receive processing. The lab
+runner observes the sender's local log and checks the receiver near completion
+or at a 60-second interval. Set a field maintenance lease appropriate to the
+actual preset and route; the 120-minute value above is a bench setting.
+
+## Applying a qualified bundle
+
+Verify the outer archive hash and every entry in `SHA256SUMS.txt`. Verify
+each `.mota` using the included, hash-pinned `motatool`. `CHAIN.csv` is the
+authoritative ordered list of packages, manifest IDs, and before/after hashes.
+On the Pi, use `tools/motatool-linux-aarch64` (make it executable after
+extracting if necessary). The x86-64 binary is for Linux/WSL, not native
+Windows. The provenance images, tools, and evidence are not sent over LoRa;
+only the selected `.mota` in the current step's serving directory is sent.
+
+For every row, including after an interrupted run:
+
+1. Prove that `ota self` matches that row's exact before hash and that the
+   bootloader reports `apply OK`. Do not select a row by version alone.
+2. Reconcile `ota status`. Resume only that row's known partial MID. If it is
+   already ready, do not redownload it. Stop on an unrelated or ambiguous
+   session. Never send a blind `ota cancel`.
+3. Reapply the agreed radio/receive guardrails after reboot. Start the sender
+   with only this row's package in its serving directory:
+   `motatool serve --dir STEP_DIRECTORY --serial COMPANION_PORT --baud 115200 -v`.
+4. Run `ota ls` until the expected version is advertised. The old listing
+   shows a version, not a MID. Use `ota pull EXPECTED_MID flash` and poll
+   `ota status` at a restrained interval.
+5. Require `ready to install` with the exact MID. Stop the seeder gracefully
+   with Ctrl-C. Recheck watchdog `off`, then send `ota install`.
+6. Allow up to five minutes for the application to return. Require the exact
+   after hash, expected version, and `bootloader: apply OK` before continuing.
+   Stop if any check fails; do not try the next package against another base.
+
+A retained previous container may be re-discovered after boot. If the exact
+previous MID is verifying, wait for it to become ready or idle. Only after
+re-proving the current running hash may that attached, completed previous
+MID be cancelled. An idle `ota cancel` does not prove that flash was erased.
+
+After the final hash is proved, restore the destination, sender, and relays
+to their recorded normal settings and confirm normal-channel reachability.
+Re-enable the system watchdog and require it to report `on`. The current
+endpoint supports `normalradio`; coordinate channel restoration so that the
+remote node does not become unreachable during the handoff.
+The endpoint requires a temporary radio channel for OTA operations, so a
+normal-channel `ota ls` is not a reachability test. Use an ordinary mesh trace
+or authenticated remote command after restoring normal radio.
+
+The existing `rak3401_mota_chain.py` pins older release archives. It must not
+be pointed at this new bundle or have its safety checks bypassed.
+
+## Evidence and reproducibility
+
+`build_rak3401_legacy24_bundle.py` checks every base/target hash, hardware
+identity, increasing version, block capability, flash workspace, and staging
+margin. It reconstructs every target with both zero- and erased-flash-filled
+scratch space, and applies every package with the actual OTAFIX2.4 release
+implementation compiled as a flash simulator. It remeasures transport with
+the pinned encoder. Its output remains explicitly offline-only until a
+separate physical qualification record is attached.
+
+`ACCOUNTING-CORRECTION.json` records the proactive-proof accounting correction.
+It changed the ideal totals only; all package checksums and the selected
+hardware-test sequence remained unchanged.
+
+The archive includes the 52 frozen search images, all 47,920 measured
+geometries, encoder binaries, per-step simulator logs, and raw hardware-test
+records. Only the selected `.mota` containers are served over LoRa. Lab
+scripts in `evidence/` are tied to the Pi's USB/debug fixture and are not a
+remote field deployment controller.
+
+Regression checks: 534 LoRa OTA tests ran successfully with one opt-in
+dependency-install test skipped; all 11 linker-selection tests passed.
+
+All PlatformIO builds must be serialized, including builds in separate
+checkout directories that share the same PlatformIO package installation.