Documents why the largest allocatable block on a non-PSRAM observer walks down in
16 KiB steps at every TLS reconnect, and what to do about it.
The framework builds mbedTLS with the symmetric buffer configuration
(CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN 16384), so each broker slot holds two 16 KiB
record buffers in internal DRAM and two slots cost 64 KiB on a board with about
80 KiB free. Confirmed against the sdkconfig the Heltec env actually links, not
the IDF 5.3 libs package that belongs to the esp32c6 env.
Rebuilding only the mbedTLS archives with an asymmetric outbound buffer of 4 KiB
is ABI-safe, because in_buf/out_buf are heap pointers sized in ssl_setup() and no
public struct embeds a CONTENT_LEN-sized array. Dropping KEEP_PEER_CERTIFICATE is
not, since it changes mbedtls_ssl_session layout, so it is excluded.
Also records two traps worth not rediscovering: esp32-arduino-lib-builder's
release/v4.4 clones its dependencies at master and no longer resolves, and
pointing platform_packages at a file:// framework copy overwrites the shared
package for every other project on the machine.
Nine commits reducing the MQTT bridge's internal-DRAM footprint, plus four
fixes that rode with them (invalid path encodings, stale-JWT scan after a
clock correction, setup-retry interval measured from the failure, retried
setup consuming the reconnect allowance).
Touches no prefs surface -- nothing in NodePrefs, MQTTPrefs, or
ConfigSerializer -- so /prefs.json layout is unaffected and there is no
fleet config risk.
Soak evidence: every soak branch already contained this work in full. Device 1
has run it 69.6 h with 325,852 publishes, 0 errors and 0 reboots. The caveat
worth carrying: that long-duration evidence is all on the reduced-TLS
framework (OUT_CONTENT_LEN 4096). Device 3 is now soaking it on the stock
framework, which is where the allocation-ordering interaction with the full
16 KiB record buffers actually gets exercised.
Both channels move together: the beta workflow's FIRMWARE_VERSION must equal
production's, since a node compares its embedded base against its own channel's
manifest and a mismatch would read as permanently behind.
Resets the per-base build counter to N=1 on both channels. Nodes still on
v1.16.0.N take the diff_base branch in ESP32Board's OTA comparison ('different
base version is always an update'), so the counter going backwards does not
strand them; the build-number arithmetic only runs when the base matches.
The flasher repo's config.json version key still needs the matching bump.
Picks up upstream MeshCore 1.17.0 and the dev commits that followed it.
Notable upstream content:
- 1.17.0 version/build-date bump in the example MyMesh headers.
- anon_req hardening: reply_path_len is now uint8_t with an isValidPathLen()
bounds check and a 0xFF sentinel; reply_path_hash_size is gone.
- LR2021 support (Meshnology W12, Seeed MeshTracker X1) incl. side detectors,
multi-SF and the new `extra.sf` get/set CLI.
- CustomLFS 0.2.2 -> 0.2.3 (GD25Q64C support).
- kiss_modem envs for several nRF52 variants; ThinkNode M6 GPS/flash fixes.
Conflicts resolved:
- platformio.ini: took upstream's CustomLFS 0.2.3, kept our lib_ignore for
the vendored PsychicMqttClient.
- CommonCLI.h: kept both upstream's USE_LR2021 configSideDetectors() hook and
our fault-alert callbacks.
NodePrefs gains upstream's extra_sf[4], which is not registered with
ConfigSerializer, so /prefs.json layout is unchanged.
Verified: 275/275 native host tests pass; Heltec_v3_repeater_observer_mqtt,
Heltec_v3_repeater and heltec_v4_repeater_observer_mqtt all build.
The flasher's /webconfig page frames webui/index.html — the real page this
firmware serves — and runs it against a browser simulator, so visitors drive the
actual product rather than looking at screenshots of it. That only stays true if
the page travels with the docs, so it joins the .md files this workflow already
syncs, and the flasher's build-webconfig-demo.py re-injects the simulator.
Triggers on webui/index.html alongside the existing markdown paths. Still gated
on observer-firmware, so nothing publishes until the portal CLI reaches the
release channel at v1.17.0; the flasher carries a hand-copied page until then.
LilyGo_TLora_V2_1_1_6 observer builds failed to link:
undefined reference to `WebConfigServer::MAX_BATCH'
in handleStatus and handleCliPost
An in-class initialiser is only a declaration under C++11, which is what the
xtensa-esp32 toolchain builds with. Every previous use of MAX_BATCH was a
comparison, which reads the value and needs no symbol. Reporting it as
status.max_cmds, and naming it in the "too many commands" error, passes it to
ArduinoJson — which takes `const T&` — and binding a reference odr-uses it.
It linked on most targets because the compiler folded the reference away, and
failed on the ones where it did not. A cast at the two call sites would have
silenced it just as narrowly; defining the symbols is what stops the next use
from depending on the same luck. MAX_BODY and STOP_WARN_MS get the same
treatment for the same reason, before they are the next to be passed by
reference.
A worktree checked out inside the repo, caught by a `git add -A` and recorded
as a gitlink. Same problem df070833 fixed for .build-wt-028a5dca, and the same
fix: untrack it and widen the ignore rule, which only covered .build-wt-*.
It is not harmless. The gitlink pointed at a different commit than the worktree
had checked out, so `git status` was permanently dirty — which is how it nearly
got re-committed here, and it makes CI checkouts warn about a submodule path
with no .gitmodules entry.
The worktree directory is gone; branch fix/station-g3-ota-manifest-base and its
commit are untouched. That branch still carries one commit not on prod —
5000391c, which adds OTA_MANIFEST_BASE to the station_g3 variant. It belongs on
prod, whose build.sh does not yet inject that flag, and must NOT come to dev,
where build.sh does and an .ini declaration cannot be overridden (SCons reorders
-U/-D) — it would pin dev and beta builds to the production manifest.
Adding build_date to the WebConfigServer constructor broke every
*_room_server_observer_mqtt target: simple_room_server constructs the portal as
well, and only simple_repeater was updated. Nothing caught it because every
build run to that point had been a repeater target.
Two findings from review, both real, both mine.
The CLI could read secrets the portal has never exposed. CommonCLI splits its
surface by CALLER, not by command: a serial caller (sender_timestamp 0, physical
access) reads secrets in plaintext, a remote one gets "******** (serial only)".
Its own comments say so — "Serial only (WiFi creds grant LAN access); remote
sees set/unset". execCommand passes 0, which is what makes `erase`, `stats-*`
and `set freq` reachable at all, and with it the terminal inherited the serial
console's plaintext answers for an HTTP request: `get prv.key` returned this
node's identity, `get wifi.pwd` the operator's network.
Worse in setup mode, which authenticates by proximity to an open AP — and `start
webconfig ap` can be run on an already-configured node, so the secrets are real
by then, not blank.
I had reasoned that the AP was the trust boundary either way because the wizard
can already rewrite these. That conflated two capabilities: replacing a WiFi
password does not reveal the current one, and replacing an identity does not
reveal the existing private key. /api/config has always masked these on read
(wcIsSecretKey); the CLI simply broke that rule. Now only the READ is masked —
the command surface stays whole — in CommonCLI's own words, keeping the
set/unset signal that is the useful part.
Onboarding could also skip the mandatory password. handleConfigPost refuses to
arm a reboot during initial setup without one; the CLI only warned in the
browser, which a pasted script or a direct POST ignores, so a node could reboot
onto the LAN still holding the factory credential. Same rule now applies at
POST. It is satisfied by a `password` command anywhere in the session rather
than only in the same request, so the natural two-step console flow still works
— the form batch always sends both together and never needed that memory.
wcIsSecretReadCommand lives in WebConfigKeys.h beside the rest of the secret
classification, pinned by three host tests: what must be masked, what must not,
and that only reads are touched. 17 keys + 24 batch tests pass; the audit checks
a masked read round-trips as masked.
The full embedded string is v1.16.0.5-observer-beta-dev-a1b2c3d. The -observer
tag is on every observer build and the commit is for machines, so neither tells
a person anything at a glance. Both the page header and the console banner now
show base + published build number + channel, paired with the build date the
way `ver` pairs them:
v1.16.0.5-dev (6 Jun 2026)
v1.16.0.5 (6 Jun 2026)
`ver` still prints the whole string, commit included, for when that is what you
need. The channel suffix follows the release filenames rather than the embedded
tag — build.sh writes FILENAME_CHANNEL_TAG "-dev" for the same builds it tags
"-observer-beta-dev" internally, so "-dev" is the name these already carry.
Carrying the build date meant /api/status had to report it; WebConfigServer now
takes FIRMWARE_BUILD_DATE alongside FIRMWARE_VERSION, from the same defines
`ver` reads.
A local build has neither build number nor channel to show, so the fact worth
knowing about it moves to the second line: "local build, OTA not configured".
build.sh deliberately leaves OTA_MANIFEST_BASE undefined there, and a bare
version number gives no hint that the node cannot update itself.
The console showed "v1.16.0" — the version was there but buried in the header
line beside role and board, and on the build under test it genuinely had no more
to show: `pio run` never goes through build.sh, so no build number, no commit,
no OTA config.
The banner now prints FIRMWARE_VERSION whole and on its own line. Nothing was
truncating it; build.sh composes base[.build][-observer][-channel]-hash, so a CI
build already carries the published build number as a 4th component and the
commit as the trailing token — the two things that actually identify a build.
It also names the channel, which the version string encodes but does not spell
out (OTA_CHANNEL_TAG=beta-dev -> "-observer-beta-dev-"):
v1.16.0.5-observer-beta-dev-a1b2c3d (dev channel)
v1.16.0.5-observer-beta-a1b2c3d (beta channel)
v1.16.0.5-observer-a1b2c3d (release channel)
v1.16.0 (local build — not from CI, OTA not
configured)
That last one earns its wording: build.sh deliberately leaves OTA_MANIFEST_BASE
undefined on local builds so such a node cannot update itself, and nothing about
a bare version number says so.
The mock reports a build.sh-shaped version now (--fw-version switches channel),
and `ver` answers from the same string /api/status does, as both do on-device.
Five findings from review, all confirmed against the source.
Failure classification (P2). Testing replies for an "Err" prefix passed five
other shapes off as success: "Unknown command", "unknown config: x", "??: x",
"Can't find GPS", "(ERR: clock cannot go backwards)" and "File system erase:
Err". They rendered green, and worse, left _batch_all_ok true — so a queued
reboot went ahead after commands that had failed, defeating the gate entirely.
Rather than lengthen one guess, the two questions are now asked separately,
each erring safe:
- colour asks "does this look like a failure", against every shape CommonCLI
actually emits, enumerated in WebConfigBatch.h and pinned by a host test
that uses the literal strings. Getting this wrong is cosmetic.
- the reboot gate asks something narrower and answerable: "did every setting
I asked for take". Only `set`/`password` gate it, and only on the "OK"
prefix every setter keeps. Diagnostics no longer gate a reboot at all, so a
harmless `memory` cannot strand one and no guess is made about "> value".
Reboot deferral (P2). CommonCLI dispatches on a six-byte prefix, so `reboot
now` and `rebooted` reach Board::reboot() too. Matching exactly meant those
variants skipped both the confirmation and the deferral and took the node down
mid-drain — the precise failure deferral exists to prevent. Both sides now
anchor the way the firmware dispatches, and the UI's risk matcher with them.
Three commands the portal cannot honestly serve are refused at POST with a
reason, and dropped from autocomplete, instead of running and lying:
- `start ota` builds a second AsyncWebServer on port 80 with no bind check
and answers "Started" regardless; the portal already holds that port, so it
could only leak the allocation and inhibit sleep.
- `clock sync` takes its time from the caller's timestamp, which a web
request has none of, so CommonCLI always rejected it. `time <epoch>` works
and remains offered.
- bare `log` and `get acl` write their real output to Serial and hand back a
stub the terminal showed as success; `log` also streams a whole file from
the loop task, stalling the mesh and radio while it does.
The mock now emits the same failure shapes it used to fake as successes, so
these are reproducible off-hardware. 24 batch + 14 keys tests pass; audit
reports 119/119 answered, 0 missing, 4/4 refused with a reason.
Nothing has ever read it. The slot count comes from two other places: the
persisted array size (MAX_MQTT_SLOTS / RUNTIME_MQTT_SLOTS in MQTTPresets.h) and
the runtime cap MQTTBridge::getMaxActiveSlots(), which answers 5 with PSRAM and
2 without. A flag reading MAX_MQTT_BROKERS=3 sitting in every observer env
implies a third, lower limit that does not exist — it cost me a wrong answer
about how many MQTT slots a Heltec V4 exposes.
36 lines across 15 variants, all of them =3. Removing an unread -D cannot change
code: heltec_v4_repeater_observer_mqtt builds to a byte-identical size before and
after (1647744). The image checksum does differ, but so does it between two
clean builds of untouched source — ESP-IDF stamps the app descriptor with the
build time — so size is the meaningful comparison here.
All 606 envs still parse; heltec_v4, heltec_v3, station_g2, rak3112 and
xiao_s3_wio observer targets all build.
Two things hardware turned up.
The whole terminal came back red. The endpoint decided a command had succeeded
by testing its reply for an "OK" prefix — the convention the config batch relies
on, and a safe one there because every allowlisted setter uses it. The CLI
reaches the whole surface, where success has no single shape: setters answer
"OK...", getters answer "> value", `erase` answers "File system erase: OK". Only
failure is uniform ("Err", "ERR:", "Error:"), so that is what the CLI now tests
for.
Colour was the visible half. The other half was worse: _batch_all_ok went false
the moment a sequence contained a `get`, so a script ending in `reboot` was told
some commands had failed and the reboot was withheld.
Replies are green now and red means the node said no, which is what red should
have meant all along. The "> " a getter prefixes its value with is dropped for
display — on the serial console it sets the value apart, here it collides with
the prompt glyph that means "you typed this". The mock emits that marker too;
had it done so from the start, this would have shown up before the flash.
Second: discover.neighbors and discover.scopes did not autocomplete, because
MyMesh::handleCommand intercepts a few commands before delegating to CommonCLI
and the table was built by reading CommonCLI alone. setperm, `get acl` and
`shutdown` were missing for the same reason.
The audit could not have caught that: it drove every command the table offered
and checked the mock answered, which only finds gaps in one direction. It now
also reads the command literals the firmware dispatches on — across CommonCLI,
CommonCLI_Observer and MyMesh — and fails on any the table does not offer. That
check found `shutdown` immediately.
122 commands, all answered, none missing. 22 batch + 14 keys tests pass.
The terminal has been driving the mock since it was built. This is the firmware
side, so it works on hardware.
Same 202 + reqid + poll contract as a config save, for the same reason:
CommonCLI touches prefs, the radio and the filesystem, none of which may be
reached from the async_tcp task. Commands go into the deferred slot and tick()
drains them on the loop task. Unlike a save this is not allowlisted — reaching
what the serial console reaches is the point, and execCommand() already passes
sender_timestamp 0, so the terminal gets exactly the serial console's
privilege. Authentication is the boundary, as it is there.
The CLI shares the config batch's slot rather than owning a second MAX_BATCH
array: both drain on the loop task, both are single-slot, and a duplicate would
cost ~8 KB of permanently resident RAM. Sharing also makes a save and a CLI run
mutually exclusive, which they must be. Each reader checks the kind, so neither
can serve the other's results.
Three things the mock could not have taught us:
- Board::reboot() does not return, so a drained `reboot` would take the node
down before the client read a single result. It is answered rather than
executed, and the batch arms the existing deferred-reboot path once the
results have been read — withheld if any command failed, exactly as a save
withholds one. clkreboot/poweroff/ota update do real work on the way down
and cannot be faked, so they still drop the connection; the UI warns first.
- `password <new>` echoes the new password in its reply. The config path
already scrubbed that by key; a CLI entry has no key, so it is matched on
the command. CLI commands are also kept out of the serial log entirely —
the browser session and the serial console are different audiences.
- MAX_BATCH is 24, not the 64 the page assumed. It is reported as
status.max_cmds instead of hardcoded, so the cap cannot drift.
Results stream and page (kCliResultPage = 8), and "done" means the client has
been handed every result, not merely that execution finished — otherwise a
client that stops polling at "done" loses the last page. Commands are never
echoed back: they may carry a secret, and the client matches by index.
New decisions live in WebConfigBatch.h with the rest, covered by three host
tests. Builds clean for heltec_v4_repeater_observer_mqtt; 22 batch + 14 keys
tests pass; the CLI audit reports 119/119 against the updated mock.
A bare 404 carries no body, so r.json() rejected and the parse failure escaped
with no HTTP status attached. Every caller then had to treat "this route does
not exist" as an ambiguous network failure — for the CLI that meant ~14 seconds
of polling before reporting a lost connection, which is the wrong diagnosis and
the wrong wait.
api() now substitutes an empty object when an *error* response has no readable
JSON, so the status survives onto the error. Successful responses must still
parse, or a captive portal's HTML would sail through as valid config.
The CLI names the case outright: firmware without /api/cli says so in 100ms
instead of retrying a route that will never exist.
`get radio.fem.rxgain` returned "unknown config key" from the mock, which reads
as the terminal offering a command that does not exist. It does exist: CommonCLI
implements get and set for it, gated at runtime by Board::canControlLoRaFemLna()
rather than compiled out, so the command is present in every build and the board
answers for itself — "Error: unsupported" where there is no front-end module.
Auditing the whole table found 31 of 70 config keys unanswered, all the ones no
portal form drives: alert.*, bridge.*, owner.info, path.hash.mode, dutycycle and
the rest. Plus 14 verbs (gps, powersaving, sensor, region, clock sync) with no
handler at all. They now live in a "cli" section of the mock config, typed
through the existing lookup tables and stripped from /api/config, which does not
carry them.
Two real bugs behind that:
- the `set` path gated on whether a key was *readable*, so write-only and
computed keys (prv.key, dutycycle, radio.fem.rxgain) were rejected as
unknown. apply_set now owns that decision alone.
- apply_set accepted anything it did not recognise and replied OK. That
leniency is what let the gap hide: a CLI `set` on an unknown key looked
like it worked. It is strict now — verified against every key in
WC_ALLOWED_SET_KEYS so the form batch is unaffected.
Also mqtt.neighbors / mqtt.neighbors.interval, which the MQTT tab binds but the
mock's config never carried, so that toggle could not round-trip.
webconfig_cli_audit.py keeps the two honest: it drives every command the
autocomplete table offers through /api/cli and fails on anything unanswered.
119 commands, all answered.
"meshcore@<node>" borrowed the user@host convention without the referents:
there is no user concept here, and the node name is already in the page header
directly above, larger. It was decoration duplicating what was on screen — and
it crowded the header enough that `help` had to be hidden below 430px.
Removing it fits all three buttons on a 375px phone, so that media query goes
with it.
Operators who already have a prepared config want to paste it, not tap through
four wizard steps. A quiet `>_` chip in the setup-mode header drops straight
into the terminal; everyone else still sees only the wizard.
That makes the CLI reachable in setup mode, which the previous commit had
deliberately blocked. Setup mode authenticates by proximity to the AP rather
than by password — but the wizard already sets the admin password and rewrites
the node's radio config from there, so the trust boundary is the AP either way,
and refusing the console would only push these operators back to serial.
Onboarding by paste does skip the one thing the wizard makes mandatory: the
admin password, which /api/config enforces before it will arm a reboot. Nothing
in CommonCLI enforces it, so the terminal says so on entry, and confirming a
reboot without a `password` command having run warns again. That is a client-
side reminder, not a gate; wiring the real gate belongs with /api/cli on-device.
The console is a one-way door out of the wizard otherwise, so its header grows
a "← setup" button that goes back.
The generator gzipped webui/index.html verbatim, so the page's comments — and
this file is commented heavily by house style — were paying flash rent. A
line-based pass now drops comments, indentation and blank lines before
compressing. The source stays as readable as it was.
Conservative on purpose: only a comment that starts its own line is removed, so
a `//` inside a URL or a `/*` inside a regex can never be mistaken for one.
Line breaks survive, which leaves JS statement boundaries (and the space a
newline contributes between HTML inline elements) exactly as written.
This ships to thousands of devices, so it is not taken on trust:
- check_stripped() fails the build if the page's structure changed or the
output shrank implausibly
- the pass lives in its own module, shared with the mock backend's new
--minify flag, so the bytes exercised in a browser are the bytes that get
embedded rather than a second implementation that could drift
- webconfig_minify.py joins the generator in the freshness hash, so editing
the stripper forces a regenerate
Today's page: 22,678 -> 17,671 bytes gzipped.
Design prototype, driven entirely by the mock backend — nothing here runs
on-device yet.
The portal's form batch is deliberately allowlisted (WebConfigKeys.h), which
leaves everything the serial console can do unreachable from a browser. This
adds a fifth tab holding a real terminal: monospace white-on-black in either
colour scheme, autocomplete over the full ~270-command surface, in-session
history, and a confirmation step for pasted command sequences.
Autocomplete goes past the flasher's <datalist>: rows carry descriptions, Tab
extends to the longest shared prefix before committing to a match, and once
`set <key> ` is complete it switches to completing the VALUE — enums from the
command table, broker presets from /api/presets, packet-type names per CSV
segment. The table is generated from a key list rather than written out per
slot, so mqttN.* tracks active_slots instead of being duplicated six times.
Pasting several lines never mangles the prompt: the lines are parsed (comments,
blank lines and pasted `>` prompts stripped), listed back numbered, and run only
after an explicit confirm. Commands that restart, erase, reflash or move the
node off its network get the same confirmation singly. History is memory-only —
`set wifi.pwd` and `password` pass through it.
/api/cli mirrors the config-save contract (202 + reqid, poll for results) for
the same reason: commands run on the node's main loop, not in the request. The
one difference is that results stream, so a long sequence fills the window as
it executes rather than landing all at once.
The tab is hidden in setup mode, where the portal authenticates by proximity
and no admin password exists yet.
agessaman