MeshCore-OTA/docs/ota_protocol.md

MeshCore OTA — .mota container & LoRa protocol (v1 draft)

Goals: distribute firmware over LoRa as a self-verifying, resumable, BitTorrent-v2-style block transfer that survives reboots, never auto-applies without consent, and is portable enough for other projects (e.g. Meshtastic) to adopt.

---

1. Conventions

• Endianness: all multi-byte integers are little-endian unless stated.
• Hashes (multihash): the hash family is declared once per manifest via hash_algo. v1 uses 0x12 = SHA-256 (the multihash code for sha2-256). Truncations used:
  • sha2-256:4 — first 4 bytes of the SHA-256 digest. Merkle leaves, internal nodes, root, proofs.
  • sha2-256:8 — first 8 bytes. Base-firmware identity (base_hash, EndF).
  • sha2-256:32 — full digest. The image security anchor (image_hash). Digests are stored bare (just the truncated bytes); the family is implied by hash_algo.
• Signatures: Ed25519 (RFC 8032), 64-byte detached signature, 32-byte public key.

Reference constants:

Name	Bytes (hex)	ASCII
Container MAGIC	6D 4F 54 41	mOTA
Container TRAILER	76 6B 34 39 36	vk496
EndF marker	45 6E 64 46	EndF
hash_algo (sha2-256)	12	—
approval = not approved	FF FF FF FF	(erased)
approval = approved	41 50 52 56	APRV
format_ver	01	—

---

2. Firmware image & the EndF trailer

Every OTA-capable firmware build appends a 16-byte EndF trailer to its flashed image so a running node can discover its own size/identity on any MCU (no linker symbols needed).

flashed image = BODY (image bytes) || EndF trailer
EndF trailer (16 bytes):
  off 0  4  "EndF"        (45 6E 64 46)
  off 4  4  body_len      uint32 LE — length of BODY (excludes this 16-byte trailer)
  off 12 8  body_hash     sha2-256:8 of BODY

• Size discovery: scan flash from the partition top downward for the EndF marker; the byte before it is the last BODY byte. (Same technique as NRF52Board::getBootloaderVersion.)
• Self-identity / delta base matching: a node's body_hash is read directly from its own EndF; a delta's base_hash (§5) must equal it. No self-hashing pass required at match time.
• No circularity: EndF hashes only the BODY, never itself.

The "reconstructed image" referenced by the manifest is the full BODY || EndF (what gets flashed).

---

3. The .mota container

This is the distributed form (host-built, wire-transferred).

off            size   field
0              4      MAGIC = 6D 4F 54 41
4              4      MOTA_TOTAL_SIZE  uint32 LE — total container bytes (incl. manifest, leaves[],
                                       payload, trailer). Lets a node pre-reserve staging and compute
                                       write_start = staging_region_end − MOTA_TOTAL_SIZE.
8              M      MANIFEST         (§4; self-delimited, no length field)
8 + M          P      PAYLOAD          (payload_size bytes; delta or full image)
8 + M + P      5      TRAILER = 76 6B 34 39 36

MOTA_TOTAL_SIZE = 4 + 4 + M + P + 5.

Staged (in-flash) form. Written bottom-aligned so TRAILER ends at staging_region_end. Identical bytes, except the device mutates two regions in place (both NOR-safe, no re-erase): the leaves[] slots (filled as blocks arrive) and the 4-byte approval field (on user approval). Everything else is immutable.

---

4. The manifest

Fields are serialized in this exact order. Conditional fields are present per flags.

off  size   field            notes
0    1      format_ver       = 0x01
1    1      flags            bit0 FULL (0=delta/partial, 1=full image)
                             bit1 SIGNED
                             bits2-7 reserved (0)
2    1      hash_algo        0x12 = sha2-256
3    4      target_id        device/arch/role discriminator (§9)
7    4      fw_version       MAJOR<<24 | MINOR<<16 | PATCH<<8 | pre   (comparable uint32)
11   4      image_size       size of the reconstructed image (BODY||EndF)
15   4      payload_size     PAYLOAD bytes in this container
19   1      block_size_log2  e.g. 0x0A = 1024
20   4      merkle_root      sha2-256:4 over PAYLOAD blocks (§6)
24   32     image_hash       sha2-256:32 of the reconstructed image — SECURITY anchor
56   1      codec_id         0=full/raw, 1=detools-sequential, 2=detools-in-place
57   8      base_hash        [present iff !FULL] sha2-256:8 of the BASE image's BODY (matches EndF.body_hash)
.    32     signer_pubkey    [present iff SIGNED] Ed25519 public key
.    64     signature        [present iff SIGNED] Ed25519 over all bytes from off 0 up to here (exclusive)
.    4      approval         ALWAYS present. FF FF FF FF = not approved; 41 50 52 56 ("APRV") = approved
.    4*BC   leaves[]         ALWAYS present. BC = ceil(payload_size / 2^block_size_log2). sha2-256:4 each

Self-delimiting: a parser knows every offset from format_ver/flags + payload_size (→ BC); no explicit length field is stored.

Manifest size (excluding leaves[]): unsigned-full 57+4=61, signed-full 161, unsigned-delta 69, signed-delta 165.

4.1 Signed region

signature covers manifest bytes [0, signature_offset) — i.e. everything before it, including signer_pubkey and (for deltas) base_hash. It does not cover approval or leaves[]:

• leaves[] are verified against the signed merkle_root (§6), so they need no separate signature.
• approval is device-local consent (§7), deliberately outside the signature.

4.2 The approval field

• Distributed and forced on ingest to FF FF FF FF (a peer can never pre-approve).
• The local user's ota apply writes 41 50 52 56 ("APRV") — a single NOR-safe write (only clears bits from the erased word). Any partial/other value reads as not-approved (fail-safe).
• Auto-bound to this image: it lives in this .mota's manifest and is re-erased when a new .mota is staged.
• It is a consent marker, not a security primitive. Authenticity = signature + image_hash.

---

5. Payload, codecs & delta base

PAYLOAD is either the full reconstructed image (FULL) or a delta (!FULL).

codec_id	Meaning	Notes
0	full / raw	PAYLOAD = reconstructed image (`BODY
1	detools sequential	needs random read of base + sequential write of result (e.g. ESP32 A→B).
2	detools in-place	bounded scratch; rewrites the app region in place (nRF52 single-slot).

For deltas, base_hash = the base image's EndF.body_hash (sha2-256:8 of its BODY). A node applies a delta only if base_hash matches its own EndF.body_hash. After applying, the result MUST hash (sha2-256:32) to image_hash before it is flashed — this is the hard security gate.

Compression is internal to the detools patch; the chosen scheme must be supported by the applier (bootloader contract, §12). Patches are produced by detools 0.53.0 (tools/mota → detools.create_patch) and decoded on-device by detools' own embeddable C decoder, vendored verbatim at src/helpers/ota/detools/ (see its README.meshcore.txt). That build enables only the self-contained NONE + CRLE compressions (no malloc / liblzma / heatshrink), so MeshCore deltas use --codec sequential --compression crle. The ESP32 applier (OtaApply.cpp::ota_apply_detools_mota) wires the decoder's callbacks to: read base ← running OTA slot, stream patch ← fetched bytes in RAM, write output → inactive slot, hashing the output and checking it against image_hash before arming.

---

6. Merkle tree (sha2-256:4)

Purpose: verify each PAYLOAD block against the signed merkle_root before the whole payload exists, so corruption/forgery is localized to a block.

• Blocks: PAYLOAD is split into BC = ceil(payload_size / B) blocks, B = 2^block_size_log2 (default 1024). The last block is its real length (no zero padding).
• Leaf: leaves[i] = sha2-256:4( block_i_bytes ).
• Internal node: node = sha2-256:4( left || right ) (4+4 = 8 input bytes).
• Odd level: if a level has an odd number of nodes, the last node is promoted unchanged to the next level (no duplication).
• Root: reduce until one node remains. BC == 1 → root = leaves[0]. BC == 0 is invalid.

6.1 Proofs

A proof for block i is the ordered list of sibling digests from leaf to root, each tagged left/right. Promoted levels contribute no element. Verification (needs BC to know the shape):

h = leaf_i ; idx = i ; n = BC ; p = 0
while n > 1:
    if (n is odd) and (idx == n-1):        # this node was promoted
        pass
    else:
        sib, side = proof[p] ; p += 1
        h = sha2-256:4( sib || h ) if side==left else sha2-256:4( h || sib )
    idx //= 2 ; n = (n + 1) // 2
accept iff h == merkle_root and p == len(proof)

Over LoRa, leaves[] are omitted from the manifest transfer; a serving node computes a block's proof on demand from its stored leaves[], and the fetcher fills its own leaves[i] as each verified block lands.

---

7. Block availability (persistent, derived from leaves[])

There is no separate availability structure. Block i is present ⟺ leaves[i] is non-erased (!= FF FF FF FF). Because leaves[] live in the staged flash region, availability survives reboot. Commit order per block (crash-safe): (1) verify proof, (2) write block payload to its offset, (3) write leaves[i] last. A power loss before step 3 leaves the slot erased → the block is simply re-fetched (idempotent). On boot a node rebuilds a small in-RAM bitmap (ceil(BC/8) bytes) by scanning leaves[].

A node holding the complete payload (or relaying/serving its own firmware) advertises have_all instead of a bitmap.

---

8. LoRa OTA protocol

Carried in MeshCore packets with PAYLOAD_TYPE_OTA = 0x0C (subject to change if core devs prefer reusing RAW_CUSTOM 0x0F + subtype). Every OTA packet payload:

[0]    ota_msg_type
[1..]  body

• Routing: OTA_ADV/OTA_QUERY flood; OTA_HAVE/OTA_MANIFEST/OTA_REQ/OTA_DATA direct.
• Hop cap: OTA refuses to retransmit when getPathHashCount() >= ota_hop_limit (default 3, configurable). No change to core routing.
• Priority: enqueued at the lowest TX priority (~250) and only when the duty-cycle/airtime budget has spare headroom, so OTA never competes with mesh traffic.
• manifest_id = the manifest's merkle_root (4 bytes) — a compact content id.

ota_msg_type	val	dir	body
OTA_ADV	0x01	flood	target_id(4) fw_version(4) image_size(4) block_size_log2(1) merkle_root(4) image_hash8(8) flags(1) [base_hash(8) if delta] have_all(1)
OTA_QUERY	0x02	flood	target_id(4) min_version(4) caps(1) (caps bit0 want_delta, bit1 want_full)
OTA_HAVE	0x03	direct	manifest_id(4) bitmap_off(2) bitmap[]
OTA_GET_MANIFEST	0x04	direct	manifest_id(4)
OTA_MANIFEST	0x05	direct	manifest_id(4) frag_idx(1) frag_total(1) bytes[] (omits leaves[])
OTA_REQ	0x06	direct	manifest_id(4) want_off(2) want_bitmap[]
OTA_DATA	0x07	direct	manifest_id(4) block_idx(2) frag_idx(1) frag_total(1) [proof in frag0] bytes[]

Sizing against the 184-byte MAX_PACKET_PAYLOAD: OTA_DATA fixed overhead ≈ 9 B → ~175 B/fragment → 6 fragments per 1 KB block; a proof for ≤512 blocks ≤ 9×4 = 36 B (carried in frag0); an availability bitmap for 500 blocks ≈ 63 B (one packet).

Reliability is eventual: the fetcher re-requests un-acked blocks after a timeout, possibly from a different peer. No hard ACKs, no ordering.

8.1 Relay seeding (companion frames)

A node need not store a foreign-target .mota to serve it: a relay advertises a manifest on behalf of an external source and pulls blocks on demand. Companion-app frames: CMD_OTA_PROVIDE_MANIFEST (app→node, starts advertising), event PUSH_OTA_BLOCK_REQ(manifest_id, block_idx) (node→app), reply CMD_OTA_PROVIDE_BLOCK(manifest_id, block_idx, bytes).

---

9. Identity, trust & versioning

• target_id (4 B): compile-time sha2-256:4(pio_env_name + radio_class + ldscript/partition + platform), injected by build.sh, read via MainBoard::getOtaTargetId(). A node only fetches/serves matching target_id. (The PlatformIO env name uniquely captures hardware AND role/partition.)
• fw_version: packed comparable uint32 (MAJOR<<24|MINOR<<16|PATCH<<8|pre).
• Signing & allowlist: a node keeps a runtime-managed allowlist of trusted Ed25519 signer pubkeys (none embedded in firmware). A .mota is eligible for auto-apply only if signed by an allowlisted key, the signature verifies, and image_hash matches; otherwise it is manual-apply only with explicit confirmation. Transfer needs no trust — blocks are content-addressed against the signed root, so any (untrusted) neighbor may relay them.

9.1 Supersession & retention

• Finish-current: a newer version announced mid-download does not abort the in-progress transfer.
• Stale GC: a staged .mota carries a persistent staged_at epoch; it is discarded after ota_stale_ttl (default 30 days) unless pinned (ota keep) or applying — reclaiming flash from superseded-complete and stalled-partial images alike. ota discard frees the slot immediately.

---

10. Apply & bootloader contract (summary)

• ESP32: in-firmware via Update/esp_ota_* into the inactive A/B slot, then set boot + reboot (power-safe, rollback-capable). No bootloader changes.
• nRF52: running firmware never flashes the app. ota apply verifies fully, writes the approval field ("APRV"), then reboots into DFU. The modified bootloader (Adafruit_nRF52_Bootloader_OTAFIX) locates the staged .mota by scanning for MAGIC, re-checks TRAILER + signature + image_hash + approval == "APRV", applies the codec (delta in-place over the app region), then clears state and boots. The signature proves author authenticity; approval proves local owner consent — both required.

---

11. Versioning of this spec

format_ver = 1. Future changes bump format_ver; the multihash hash_algo allows changing the digest family without a format bump. Unknown format_ver/codec_id/ota_msg_type values are ignored (forward-compatible).