diff --git a/docs/mbedtls-tls-footprint.md b/docs/mbedtls-tls-footprint.md new file mode 100644 index 00000000..0985c4f6 --- /dev/null +++ b/docs/mbedtls-tls-footprint.md @@ -0,0 +1,153 @@ +# Shrinking the per-connection TLS footprint on non-PSRAM observers + +## Why + +On a non-PSRAM Heltec V3 running two WSS/JWT broker slots, the largest allocatable block in +internal DRAM walks down in ~16 KiB steps at every TLS reconnect while total free heap stays +flat. Measured on hardware over 50 reconnect cycles: 62,452 → 16,372 bytes, permanently. + +The step size is not a coincidence. `framework-arduinoespressif32 3.20017` (Arduino 2.0.17, +IDF 4.4) builds mbedTLS with the **symmetric** buffer configuration: + +``` +CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN 16384 # sizes BOTH the in and out record buffers +CONFIG_MBEDTLS_SSL_KEEP_PEER_CERTIFICATE 1 +# CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN — not defined +# CONFIG_MBEDTLS_DYNAMIC_BUFFER — not defined +# CONFIG_MBEDTLS_SSL_VARIABLE_BUFFER_LENGTH — not defined +``` + +Read from `packages/framework-arduinoespressif32/tools/sdk/esp32s3/qio_qspi/include/sdkconfig.h`. +Note the separate `framework-arduinoespressif32-libs` package in `~/.platformio` belongs to +the esp32c6 env (pioarduino, IDF 5.3) and is **not** what this env links against — though it +happens to carry identical mbedTLS values. + +So each broker slot costs **2 × 16 KiB = 32 KiB** of contiguous internal DRAM in record +buffers alone, and two slots cost 64 KiB on a board with roughly 80 KiB free. Every reconnect +frees and re-allocates those buffers, and anything that lands in the vacated hole in between +prevents them from going back, which is the ratchet. + +Confirmed by two independent observations: losing a whole TLS session returned exactly 16,384 +bytes of contiguity on one device and exactly 32,768 on another, and per-connection teardown +frees ~41.6–44.7 KB total. + +## What the firmware could already do, and its limit + +`softDisconnect()` (branch `perf/mqtt-renewal-no-stop`, commit `6c51e468`) stops the JWT +renewal bounce from destroying and recreating the esp-mqtt task, keeping its 6 KiB stack out +of the hole. Measured: the staircase arrests after 2 steps at 36,852 through cycle 16, where +the baseline took 4 steps and settled at 16,372 by cycle 11 — about 20 KB better. + +That is as far as the application layer reaches. MQTT 3.1.1 has no re-authentication packet, +so presenting a fresh JWT *requires* a transport reconnect; mbedTLS's internal allocation +order during the handshake is not controllable from the application. The remaining cost is +the record buffers themselves. + +## The changes + +All three are compile-time in mbedTLS, and the Arduino framework ships precompiled `.a` +archives (`tools/sdk/esp32s3/lib/libmbedtls.a`), so a project-level `-D` cannot change them. +A custom framework build is required. + +| Setting | From | To | Saving per connection | +|---|---|---|---| +| `CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN` | unset | `y` | enables the two below | +| `CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN` | 16384 (implied) | 16384 | none — keep it | +| `CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN` | 16384 (implied) | 4096 | **~12 KiB** | +| `CONFIG_MBEDTLS_SSL_KEEP_PEER_CERTIFICATE` | `1` | `n` | ~4 KiB | + +Roughly 16 KiB per connection, 32 KiB across two slots. + +### Why inbound stays at 16 KiB + +A peer may legitimately send a 16 KiB TLS record. Lowering the inbound limit only works if +both brokers negotiate the max-fragment-length extension or provably never send larger, and +getting it wrong produces invalid-record and handshake failures rather than a clean error. +Outbound is ours to choose: this firmware's MQTT and WebSocket frames are far below 4 KiB +(`MAX_TRANS_UNIT`-bounded packets plus small JSON), so 4 KiB is comfortable. + +### Risk on the peer certificate + +Dropping `KEEP_PEER_CERTIFICATE` means `mbedtls_ssl_get_peer_cert()` returns NULL after the +handshake. Chain validation still happens — only retention of the parsed leaf changes. This +firmware verifies against a CA (`GTS_ROOT_R4` / the bundle) and never inspects the peer +certificate or a fingerprint, so it should be safe. It does change `mbedtls_ssl_session` +layout, which is exactly why the whole framework must be rebuilt together rather than +swapping a single archive in. + +## Build procedure + +### Do not use esp32-arduino-lib-builder for this + +`release/v4.4` is the branch matching Arduino 2.0.x, but its `update-components.sh` clones +every dependency at **master**, so it no longer resolves. Five successive failures, each a +different repo: the arduino branch name it passes to `-A` does not exist; `jq` is absent +from the IDF image and its absence makes `build.sh`'s target loop a **silent no-op that +still exits 0**; `esp_littlefs` and `esp32-camera` master require IDF ≥5.0/≥5.1; +`esp32-camera` later needs an `esp_jpeg` version the 4.4 registry cannot satisfy; and +tinyusb's source layout no longer matches `arduino_tinyusb/CMakeLists.txt`. Pinning each +one in turn just surfaces the next. + +### Rebuild only the mbedTLS archives + +More rigorous anyway, because it reuses the shipped `sdkconfig` verbatim — so the archives +differ from stock *only* by the intended change, with no arduino-version or +`DYNAMIC_BUFFER` drift. + +This is ABI-safe for the content-length change specifically: `ssl.h` declares `in_buf` and +`out_buf` as `unsigned char *`, allocated in `ssl_setup()`, and no public struct embeds a +CONTENT_LEN-sized array. The other precompiled archives (esp-tls, esp_http_client, +esp-mqtt) therefore remain compatible. **It is not safe for +`CONFIG_MBEDTLS_SSL_KEEP_PEER_CERTIFICATE`**, which changes `mbedtls_ssl_session` layout — +that one needs everything rebuilt together, so it is excluded here. + +1. Minimal IDF project whose only component requirement is `mbedtls`. +2. `sdkconfig.defaults` = the shipped + `packages/framework-arduinoespressif32/tools/sdk/esp32s3/sdkconfig`, with + `CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384` replaced by the three asymmetric lines. + Verify the diff is exactly 5 lines before building. +3. `docker run --rm -v $PWD:/project -w /project espressif/idf:v4.4.7 idf.py -DIDF_TARGET=esp32s3 build` +4. Confirm `build/config/sdkconfig.h` carries `OUT_CONTENT_LEN 4096`. +5. Stage the four archives under the framework's names — note the rename: + + | built | framework name | stock size | rebuilt | + |---|---|---:|---:| + | `esp-idf/mbedtls/libmbedtls.a` | `libmbedtls.a` | 113,914 | 113,338 | + | `esp-idf/mbedtls/mbedtls/library/libmbedtls.a` | **`libmbedtls_2.a`** | 1,252,150 | 1,245,718 | + | `.../libmbedcrypto.a` | `libmbedcrypto.a` | 4,302,698 | 4,259,458 | + | `.../libmbedx509.a` | `libmbedx509.a` | 676,590 | 673,318 | + + All within ~1% of stock, which is a good check that only the config differs. + +### Wire it in with -L, never platform_packages + +``` +PLATFORMIO_BUILD_FLAGS="-L/path/to/staged/archives" pio run -e Heltec_v3_repeater_observer_mqtt +``` + +Verify with `grep -oE "[^ ]*libmbed[a-z0-9_]*\.a" .pio/build//firmware.map | sort -u` — +every path must be the staged directory. + +**Do not** point `platform_packages` at a `file://` copy of the framework. PlatformIO +installs it *over* the shared `~/.platformio/packages/framework-arduinoespressif32`, +silently changing mbedTLS for every other ESP32 env and project on the machine. It does this +even when the copy's `package.json` version differs — verified twice here, and both times the +fix was `rm -rf` the package and `pio pkg install` to re-download stock. A prepended library +search path keeps the change scoped to one env, because the linker takes each archive member +from the first archive that satisfies an undefined symbol. + +## How to verify it worked + +1. `strings`/`grep` the new `sdkconfig.h` for the four settings. +2. Build and check the RAM figure; static usage should be unchanged (these are heap buffers). +3. On hardware, `get mqtt.stats` at boot with two slots connected: the largest free block + should start roughly 24–32 KiB higher than the current 62–67 KiB. +4. Soak across reconnect cycles and compare the floor against the two recorded runs: + baseline settled 16,372 (cycle 11); `softDisconnect` holds 36,852 (cycle 16). + +## Prior art in this investigation + +`.scratch/mqtt-non-psram-heap-staircase-analysis-2026-08-05.md` (untracked — `.scratch/` is +globally gitignored) holds the full allocation inventory. `~/mqtt-soak/STATE.md` holds the +soak evidence, including two retracted hypotheses worth not repeating: the perf commits were +not the cause, and waev does not cap connections per IP. diff --git a/platformio.local.ini.hold b/platformio.local.ini.hold new file mode 100644 index 00000000..e0d89578 --- /dev/null +++ b/platformio.local.ini.hold @@ -0,0 +1,15 @@ +; Local-only override (gitignored) pointing the Heltec V3 observer env at a custom +; framework whose mbedTLS archives were rebuilt with an asymmetric TLS record buffer: +; CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y +; CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=16384 (unchanged — a peer may send a 16 KiB record) +; CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096 (was 16384) +; +; Expected: ~12 KiB less internal DRAM per TLS connection, ~24 KiB across two broker slots. +; Built from the shipped sdkconfig verbatim plus those three lines, so the archives differ +; only by this change. See docs/mbedtls-tls-footprint.md. +; +; Absolute path, hence local-only: not committable. + +[env:Heltec_v3_repeater_observer_mqtt] +platform_packages = + framework-arduinoespressif32 @ file:///Users/adam/framework-arduinoespressif32-tlsfix