Precompute ESP32 release partition lookup table

docs/esp32_partition_catalog.md

@@ -0,0 +1,91 @@
+# ESP32 release partition lookup table
+
+The canonical LUT is `firmware/esp32_partition_catalog.json` in this fork.
+`scripts/precompute_esp32_partitions.py` regenerates it from GitHub release
+assets. MeshCore Open bundles an identical copy for offline phone updates.
+There is no need to run a command on the phone or download old firmware there.
+
+## Coverage
+
+The initial snapshot enumerates every published release (including prereleases,
+excluding drafts) in these repositories, without a recent-release cutoff:
+
+| Repository | Releases | Inspected ESP32 factory images |
+| --- | ---: | ---: |
+| meshcore-dev/MeshCore | 94 | 1,998 |
+| IoTThinks/EasySkyMesh (PowerSaving) | 14 | 336 |
+| mikecarper/MeshCore (including keymindCascade releases) | 69 | 7,808 |
+
+All 10,142 standalone `-merged.bin` / `-cleanInstall.bin` images were parsed,
+with nine distinct partition tables and no unresolved selected assets.
+PowerSaving release tags can contain several firmware patch versions; the LUT
+uses each asset's version, not merely its containing release tag.
+
+Eleven releases have no standalone ESP32 factory image. These include nRF52
+update chains, RP2040/nRF52-only releases, an empty release, checksum-only
+metadata, and the ZIP-only Station G3 KISS modem package. They remain in the
+release inventory with an explicit note, but supply no inferred ESP32 layout.
+Unpublished builds, custom partition tables, and partition tables inside ZIP
+archives are not covered. This is an inventory of published factory layouts,
+not a claim that every Git revision has a known layout.
+
+## Matching and safety
+
+1. Ask the device for `get storage.layout` internally. Complete, valid OTA slot
+   measurements take precedence over the LUT. A truncated reply is not proof
+   that a slot is absent.
+2. Otherwise match board, role, and complete firmware version. Prefer the exact
+   build hash when reported and present. Preserve profile candidates; do not
+   conflate PowerSaving, upstream, and fork version suffixes or V4/R8 boards.
+3. Compare the chosen application length against both OTA slots (use the
+   smaller slot so the next update also fits). If multiple profiles disagree,
+   only make a fit/expansion recommendation when all candidates agree.
+4. An oversized image needs a supported exact-board two-stage migration bundle.
+   A single-app layout cannot accept the bridge via normal dual-slot OTA and
+   needs cable installation. Unknown layouts remain explicitly unknown.
+
+Version-based results are **estimates**. OTA usually replaces only the app, so
+a newer version can run on an older partition table. Never use this LUT alone
+to authorize a partition-table write. The migration bridge must validate the
+actual flash geometry and identity recovery on the device before the final
+application upload. See [two-stage migration](esp32_wifi_partition_migration.md).
+
+The phone blocks known oversized regular uploads and opens its two-step
+migration section. Companion updates use the same capacity guard, but the
+Companion screen does not yet perform partition migration; it offers a smaller
+exact-board image or cable migration when necessary.
+
+## Schema 1
+
+- `repositories`, `generatedAt`: snapshot provenance.
+- `releases`: repository, tag, publication date, asset counts and coverage note.
+- `builds`: release index, original asset name/ID/size/GitHub digest, board,
+  role/profile, version, full build version and layout key.
+- `layouts`: table SHA-256 mapped to partition entries, minimum OTA slot size
+  and `dualOta`. Entry fields are `[type, subtype, offset, size, label, flags]`.
+- `unresolved`: selected assets that could not be read or parsed, with reasons.
+
+The generator reads at most the first 36 KiB of each image, including partition
+tables at merged-file offset 0x8000 or 0x7000. It validates partition bounds,
+overlaps, duplicate app subtypes and the table MD5 when supplied. GitHub asset
+digests are recorded as provenance; a prefix read does **not** verify the full
+asset SHA-256. This LUT is not a substitute for firmware download verification.
+
+## Refresh and tests
+
+Use an authenticated `gh` CLI and Python 3. The generator has bounded downloads,
+retries, and a disposable prefix cache keyed by asset identity/update metadata.
+No secrets or firmware binaries are committed.
+
+```sh
+python3 scripts/precompute_esp32_partitions.py \
+  --output firmware/esp32_partition_catalog.json \
+  --cache-dir /tmp/meshcore-partition-catalog-cache \
+  --copy-to ../meshcore-open-android-5.1.1/assets/firmware/esp32_partition_catalog.json
+python3 test/test_partition_catalog.py
+```
+
+`--cached-inventory` deliberately reuses the saved release listing for a
+resumable, reproducible snapshot. Omit it to discover newly published assets.
+Review coverage counts and `unresolved` after each refresh before shipping the
+LUT. No scheduled refresh job or release upload is installed by this script.

scripts/precompute_esp32_partitions.py

@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+"""Build the canonical MeshCore ESP32 partition LUT from published firmware.
+
+Requires an authenticated gh CLI for release enumeration. Binary prefixes are
+public, bounded HTTP range reads. A table checksum authenticates consistency,
+not the complete asset: GitHub digests are provenance, not locally verified SHA.
+"""
+
+import argparse
+import concurrent.futures
+import hashlib
+import json
+import re
+import struct
+import subprocess
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+
+REPOSITORIES = (
+    "meshcore-dev/MeshCore",
+    "IoTThinks/EasySkyMesh",
+    "mikecarper/MeshCore",
+)
+PREFIX_BYTES = 0x9000
+
+
+def is_merged_asset(name):
+    return name.lower().endswith(("-merged.bin", "-cleaninstall.bin"))
+
+
+def parse_table(prefix):
+    """Merged images may start at flash 0 or 0x1000. Reject corrupt tables."""
+    for position in (0x8000, 0x7000):
+        data = prefix[position:position + 0x1000]
+        if not data.startswith(b"\xaa\x50"):
+            continue
+        entries = []
+        end = 0
+        for index in range(0, len(data) - 31, 32):
+            row = data[index:index + 32]
+            if row[:2] == b"\xeb\xeb":
+                if row[2:16] != b"\xff" * 14:
+                    raise ValueError("Malformed table MD5 record")
+                if hashlib.md5(data[:index]).digest() != row[16:32]:
+                    raise ValueError("Partition table MD5 mismatch")
+                end = index + 32
+                break
+            if row == b"\xff" * 32:
+                end = index
+                break
+            magic, kind, subtype, offset, size, label, flags = struct.unpack(
+                "<HBBII16sI", row
+            )
+            if magic != 0x50AA or not size or offset < 0x9000:
+                raise ValueError("Invalid partition entry")
+            if offset + size > 0x1000000:
+                raise ValueError("Partition exceeds supported 16 MB flash")
+            name = label.split(b"\0", 1)[0].decode("ascii")
+            entries.append([kind, subtype, offset, size, name, flags])
+        if not end or not entries:
+            raise ValueError("Unterminated partition table")
+        ordered = sorted(entries, key=lambda entry: entry[2])
+        if any(a[2] + a[3] > b[2] for a, b in zip(ordered, ordered[1:])):
+            raise ValueError("Overlapping partitions")
+        if len({(e[0], e[1]) for e in entries if e[0] == 0}) != sum(
+            e[0] == 0 for e in entries
+        ):
+            raise ValueError("Duplicate app subtype")
+        slots = [e[3] for e in entries if e[0] == 0 and 0x10 <= e[1] < 0x20]
+        dual = len(slots) >= 2 and any(e[0:2] == [1, 0] for e in entries)
+        return {
+            "partitions": entries,
+            "slotBytes": min(slots) if dual else None,
+            "dualOta": dual,
+            "tableSha256": hashlib.sha256(data[:end]).hexdigest(),
+        }
+    raise ValueError("No partition table at known merged-image offsets")
+
+
+def asset_identity(name):
+    # Preserve profile suffixes and complete versions; PowerSaving release tags
+    # sometimes contain several distinct patch versions.
+    name = re.sub(r"-cleanInstall\.bin$", "-merged.bin", name, flags=re.I)
+    old_power = re.match(
+        r"^repeater-(.+)-(powersaving\d+)-merged\.bin$", name, re.I
+    )
+    if old_power:
+        return old_power.group(1), "repeater", old_power.group(2)
+    match = re.match(
+        r"^(.+?)_(repeater|repeatr|room_server|room_svr|companion_radio|"
+        r"comp_radio|companion|sensor|kiss_modem|terminal_chat)"
+        r"(.*?)-((?:v?\d|powersaving).+)-merged\.bin$", name, re.I
+    )
+    if not match:
+        raise ValueError("Unrecognized board/role/version filename")
+    board, role, profile, version = match.groups()
+    role = {"repeatr": "repeater", "room_svr": "room_server",
+            "comp_radio": "companion_radio", "companion": "companion_radio"}.get(
+                role.lower(), role.lower())
+    profile = profile.replace("-", "_")
+    version = re.sub(r"-freshInstall$", "", version, flags=re.I)
+    version = re.sub(r"-[0-9a-f]{7,40}$", "", version, flags=re.I)
+    return board, role + profile, version
+
+
+def read_prefix(url):
+    if not url.startswith("https://github.com/"):
+        raise ValueError("Not a GitHub release asset URL")
+    request = urllib.request.Request(
+        url, headers={"Range": f"bytes=0-{PREFIX_BYTES - 1}",
+                      "User-Agent": "meshcore-partition-catalog/1"}
+    )
+    for attempt in range(4):
+        try:
+            with urllib.request.urlopen(request, timeout=45) as response:
+                if not response.url.startswith("https://"):
+                    raise ValueError("Insecure asset redirect")
+                if response.status == 206 and not response.headers.get(
+                    "Content-Range", ""
+                ).startswith("bytes 0-"):
+                    raise ValueError("Range response starts at wrong offset")
+                # Also bounded if a server ignores Range and returns HTTP 200.
+                return response.read(PREFIX_BYTES)
+        except (OSError, urllib.error.URLError):
+            if attempt == 3:
+                raise
+            time.sleep(2 ** attempt)
+
+
+def releases(repository, cache, cached_inventory=False):
+    path = cache / (repository.replace("/", "_") + ".json")
+    if cached_inventory and path.exists():
+        return json.loads(path.read_text())
+    for attempt in range(4):
+        result = subprocess.run(
+            ["gh", "api", f"repos/{repository}/releases?per_page=20",
+             "--paginate", "--jq", ".[] | @json"], capture_output=True, text=True
+        )
+        if result.returncode == 0:
+            items = [json.loads(line) for line in result.stdout.splitlines()]
+            items = [item for item in items if not item["draft"]]
+            path.write_text(json.dumps(items, ensure_ascii=True))
+            return items
+        if attempt == 3:
+            raise RuntimeError(f"Cannot enumerate {repository}: {result.stderr}")
+        time.sleep(2 ** attempt)
+
+
+def inspect_asset(job, cache):
+    release_index, asset = job
+    record = {"release": release_index, "asset": asset["name"],
+              "assetId": asset["id"], "assetSize": asset["size"],
+              "assetDigest": asset.get("digest")}
+    try:
+        board, role, version = asset_identity(asset["name"])
+        fingerprint = hashlib.sha256(json.dumps(
+            [asset["id"], asset["updated_at"], asset["size"], asset.get("digest")]
+        ).encode()).hexdigest()
+        path = cache / (fingerprint + ".prefix")
+        if path.exists():
+            prefix = path.read_bytes()
+        else:
+            prefix = read_prefix(asset["browser_download_url"])
+            path.write_bytes(prefix)
+        layout = parse_table(prefix)
+        record.update(board=board, role=role, version=version,
+                      layout=layout["tableSha256"])
+        embedded = re.search(r"-((?:v?\d|powersaving).*)-(?:merged|cleanInstall)\.bin$",
+                             asset["name"], re.I)
+        record["buildVersion"] = re.sub(
+            r"-freshInstall$", "", embedded.group(1), flags=re.I
+        ) if embedded else version
+        return record, layout, None
+    except Exception as error:
+        record["reason"] = str(error)
+        return None, None, record
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--copy-to", type=Path, action="append", default=[],
+                        help="Also write an identical offline app snapshot")
+    parser.add_argument("--cache-dir", type=Path, required=True)
+    parser.add_argument("--cached-inventory", action="store_true")
+    parser.add_argument("--workers", type=int, default=8)
+    args = parser.parse_args()
+    args.cache_dir.mkdir(parents=True, exist_ok=True)
+    catalog = {"schema": 1, "generatedAt": datetime.now(timezone.utc).isoformat(),
+               "repositories": list(REPOSITORIES), "releases": [], "layouts": {},
+               "builds": [], "unresolved": []}
+    jobs = []
+    for repository in REPOSITORIES:
+        items = releases(repository, args.cache_dir, args.cached_inventory)
+        print(f"{repository}: {len(items)} published releases", flush=True)
+        for release in items:
+            assets = release["assets"]
+            merged = [a for a in assets if is_merged_asset(a["name"])]
+            index = len(catalog["releases"])
+            catalog["releases"].append({
+                "repository": repository, "tag": release["tag_name"],
+                "publishedAt": release["published_at"],
+                "assetCount": len(assets), "mergedCount": len(merged),
+                "note": None if merged else "No standalone ESP32 merged images",
+            })
+            jobs.extend((index, asset) for asset in merged)
+    print(f"Inspecting {len(jobs)} merged images (cached ranges reused)", flush=True)
+    with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
+        for done, result in enumerate(pool.map(
+            lambda job: inspect_asset(job, args.cache_dir), jobs
+        ), 1):
+            record, layout, error = result
+            if error:
+                catalog["unresolved"].append(error)
+            else:
+                catalog["builds"].append(record)
+                catalog["layouts"][layout["tableSha256"]] = layout
+            if done % 100 == 0 or done == len(jobs):
+                print(f"{done}/{len(jobs)} inspected; "
+                      f"{len(catalog['unresolved'])} unresolved", flush=True)
+    content = json.dumps(catalog, ensure_ascii=True, separators=(",", ":")) + "\n"
+    for output in [args.output, *args.copy_to]:
+        output.parent.mkdir(parents=True, exist_ok=True)
+        output.write_text(content)
+    print(f"Wrote {len(catalog['builds'])} builds, {len(catalog['layouts'])} layouts; "
+          f"{len(catalog['unresolved'])} unresolved to {args.output}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

test/test_partition_catalog.py

@@ -0,0 +1,131 @@
+"""Host-only regression tests for the release partition LUT generator."""
+import hashlib
+import json
+from pathlib import Path
+import struct
+import sys
+import tempfile
+import unittest
+from unittest.mock import patch
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
+import precompute_esp32_partitions as catalog
+
+
+def entry(kind, subtype, start, size, label):
+    return struct.pack("<HBBII16sI", 0x50AA, kind, subtype,
+                       start, size, label.encode(), 0)
+
+
+def image(entries, offset=0x8000, checksum=True):
+    table = b"".join(entries)
+    if checksum:
+        table += b"\xeb\xeb" + b"\xff" * 14 + hashlib.md5(table).digest()
+    table = table.ljust(4096, b"\xff")
+    return b"\xff" * offset + table
+
+
+class PartitionCatalogTest(unittest.TestCase):
+    def setUp(self):
+        self.entries = [
+            entry(1, 2, 0x9000, 0x5000, "nvs"),
+            entry(1, 0, 0xE000, 0x2000, "otadata"),
+            entry(0, 0x10, 0x10000, 0x140000, "app0"),
+            entry(0, 0x11, 0x150000, 0x140000, "app1"),
+            entry(1, 0x82, 0x290000, 0x170000, "spiffs"),
+        ]
+
+    def test_dual_slot_at_both_merged_offsets(self):
+        for offset in (0x8000, 0x7000):
+            result = catalog.parse_table(image(self.entries, offset))
+            self.assertTrue(result["dualOta"])
+            self.assertEqual(result["slotBytes"], 0x140000)
+
+    def test_legacy_table_without_checksum(self):
+        self.assertTrue(catalog.parse_table(
+            image(self.entries, checksum=False))["dualOta"])
+
+    def test_reject_corrupt_checksum_overlap_and_missing_table(self):
+        broken = bytearray(image(self.entries))
+        broken[0x8000 + 12] ^= 1
+        for data in (broken, image(self.entries + [self.entries[0]]), b"no table"):
+            with self.assertRaises(ValueError):
+                catalog.parse_table(data)
+
+    def test_single_app_is_not_migratable_via_ota(self):
+        result = catalog.parse_table(image([
+            entry(0, 0, 0x10000, 0x300000, "app0")]))
+        self.assertFalse(result["dualOta"])
+        self.assertIsNone(result["slotBytes"])
+
+    def test_missing_otadata_is_not_dual_ota(self):
+        result = catalog.parse_table(image(self.entries[2:]))
+        self.assertFalse(result["dualOta"])
+
+    def test_minimum_of_unequal_slots(self):
+        self.entries[3] = entry(0, 0x11, 0x150000, 0x130000, "app1")
+        self.assertEqual(catalog.parse_table(image(self.entries))["slotBytes"],
+                         0x130000)
+
+    def test_exact_filename_versions_and_profiles(self):
+        cases = {
+            "heltec_v4_repeater-v1.17.1-d929643-merged.bin":
+                ("heltec_v4", "repeater", "v1.17.1"),
+            "Heltec_v3_repeater-PowerSaving17.1.3-freshInstall-merged.bin":
+                ("Heltec_v3", "repeater", "PowerSaving17.1.3"),
+            "Heltec_v3_repeater-PowerSaving13.1-cleanInstall.bin":
+                ("Heltec_v3", "repeater", "PowerSaving13.1"),
+            "repeater-heltec-wsl3-powersaving09-merged.bin":
+                ("heltec-wsl3", "repeater", "powersaving09"),
+            "Generic_ESPNOW_repeatr-full-logging-ota-v1.17.1.6-halo-keymind-cascade-dev-306feebe-merged.bin":
+                ("Generic_ESPNOW", "repeater_full_logging_ota",
+                 "v1.17.1.6-halo-keymind-cascade-dev"),
+            "Heltec_E290_companion_usb-ota-v1.17.1.1-abcdef12-merged.bin":
+                ("Heltec_E290", "companion_radio_usb_ota", "v1.17.1.1"),
+        }
+        for name, expected in cases.items():
+            self.assertTrue(catalog.is_merged_asset(name))
+            self.assertEqual(catalog.asset_identity(name), expected)
+
+    def test_range_is_bounded_even_if_server_ignores_it(self):
+        class Response:
+            status = 200
+            url = "https://release-assets.githubusercontent.com/asset"
+            headers = {}
+            def __enter__(self): return self
+            def __exit__(self, *args): pass
+            def read(self, count):
+                self.count = count
+                return b"x" * count
+        response = Response()
+        with patch.object(catalog.urllib.request, "urlopen", return_value=response):
+            self.assertEqual(len(catalog.read_prefix("https://github.com/a/b")),
+                             catalog.PREFIX_BYTES)
+        self.assertEqual(response.count, catalog.PREFIX_BYTES)
+
+    def test_wrong_range_is_rejected(self):
+        class Response:
+            status = 206
+            url = "https://github.com/a/b"
+            headers = {"Content-Range": "bytes 4096-8192/99999"}
+            def __enter__(self): return self
+            def __exit__(self, *args): pass
+        with patch.object(catalog.urllib.request, "urlopen", return_value=Response()):
+            with self.assertRaises(ValueError):
+                catalog.read_prefix("https://github.com/a/b")
+
+    def test_release_inventory_excludes_drafts_and_retains_prereleases(self):
+        class Result:
+            returncode = 0
+            stdout = '\n'.join(json.dumps(item) for item in [
+                {"draft": True}, {"draft": False, "prerelease": True},
+                {"draft": False, "prerelease": False}])
+        with tempfile.TemporaryDirectory() as directory:
+            with patch.object(catalog.subprocess, "run", return_value=Result()) as run:
+                items = catalog.releases("owner/repo", Path(directory))
+                self.assertEqual(len(items), 2)
+                self.assertIn("--paginate", run.call_args.args[0])
+
+
+if __name__ == "__main__":
+    unittest.main()