A switch can land on an older build: e.g. prod cannot read dev's /mqtt.json,
and prod's M7 image has no Ethernet. Each image carries an ota-compat tag
(state generation + transports). A switch is flashed without rebooting, the
new image's tag is read (bounded by its length), and boot is pointed back at
the running image unless the target keeps this node's state and transports.
CI now requires the tag in every published binary.
Prod and beta are often built from one commit (e.g. after a dev->prod merge),
so a hash match does not mean the images are the same: the native base differs.
After the flash the target channel is native, so this cannot loop.
Every observer build now carries both channel URLs, so checking for a URL's
presence or absence can no longer tell prod from beta (and the old checks would
reject every build). Read the ota-base-* tags from every .bin in out/ instead
of a single ELF, with URLs shared by build.sh and both workflows.
Build counters are per channel, so a cross-channel build number is not
comparable: a switch to a channel with a lower counter reported up to date
and never happened. Off the native channel, update unless the hash matches.
A slot waiting on a token/IATA/credential stays enabled but never attempts a
connection, so it held mqtt_slots_up below mqtt_slots_total (and down= at 1)
permanently -- a standing false alert on a node that is merely unconfigured.
slots_total and down= now count only ready slots, matching the CLI's "wait".
mqtt_connect_failures summed only enabled slots, so disabling a failing slot
made it drop. It now sums every slot, monotonic within a boot.
cf: moves after the error tail in mqttN.diag so a clamped 160-byte reply
loses the count rather than the refusal/TLS detail. A failed start now clears
the attempt flag instead of restoring a possibly stale one. Outage comments
corrected: the clock starts at a slot's first DISCONNECTED, not first connect.
reconnect() on a started client returns ESP_FAIL while an attempt is already
in progress. On a Heltec V3, a live reconfigure hit exactly that and the
attempt then connected, yet cf read 1. Count start_failures only when
connect() on a stopped client fails, and restore the attempt flag to its
prior value on any failed call so the in-progress attempt is still judged by
its own outcome.
connect()/reconnect() returning an error started no attempt, so no event
arrived and the failure went uncounted -- the low-heap case the counter exists
for. Counted now in a bridge-task-only start_failures, summed into cf: and
mqtt_connect_failures.
A renewal or corrected-clock bounce whose softDisconnect() times out gets its
old session's DISCONNECTED after the new attempt raised the flag. That session
is still marked connected until the handler runs, so the gate now also
requires !connected. An ignored late CONNECTED clears the flag too.
cf: no longer depends on dc being non-zero. Comments corrected for the
PSRAM-fallback stack cost and heap_largest's sampling point; the worst-case
test now models 6 slots.
The Starting-state gate raced the event task: client_state becomes Starting
only after connect()/reconnect() returns, so a fast failure could deliver
DISCONNECTED first and go uncounted, while a deliberate stop of an attempt in
flight was counted as a failure. A per-slot flag is now raised before the call
and cleared by onConnect, a failed start, or stopSlotClient().
Also stops zero-valued connect_failures/slots_breaker from creating an empty
stats object.
Two gaps remained after the first connection-health commit.
A slot retrying into a heap that cannot satisfy its TLS allocation moved no
counter at all. disconnect_count rose, but it also rises for dropped live
sessions, so it could not separate "actively failing to connect" from "was
connected, dropped once". Adds a per-slot connect_failures, incremented in
onDisconnect when the event arrives while the client is Starting -- meaning
it ended an attempt that never reached onConnect. Late events from a stopped
or quarantined client, and soft-disconnects of a live session, are in other
states and are not miscounted.
A slot parked by the circuit breaker also looked identical to one still
retrying, despite being materially worse: the breaker only probes every 30
minutes, so recovery is far slower and an operator may need to intervene.
Surfaces both:
- `mqttN.diag` gains `cf:<n>` beside the existing `dc:<n>`.
- the status payload gains `mqtt_connect_failures` (summed across slots) and
`mqtt_slots_breaker` (count of tripped slots), each emitted only when
non-zero so a healthy board's payload does not grow.
The counter is what distinguishes a retrying slot from a quiet one: while a
slot is down its publish counters cannot move by definition, so this is the
only value that shows the device is still trying.
Both suggested by adversarial review.
The worst-case test added with the connection-health fields did not model
production maxima -- it used a 28-byte model and a 24-byte firmware string,
while MQTTBridge permits _origin[32], _device_id[65], _board_model[64],
_firmware_version[64] and derives a 63-byte client version. isValidName()
rejects [ ] / \ : , ? * but permits '"', so a 31-character node name of
quotes is legal and every quote escapes to two bytes.
Measured at those real maxima:
without connection-health fields 762 bytes
with connection-health fields 851 bytes
buffer 768 bytes
So the 768-byte buffer already had only six bytes of headroom before this
work, and the four new fields overflow it. Overflow is silent:
serializeComplete() returns 0 and clears the buffer, and the bridge
publishes only a positive length, so a device would simply stop publishing
status with no error anywhere -- the same blindness the connection-health
fields exist to remove.
Raises STATUS_JSON_BUFFER_SIZE to 1024 and corrects the test to use the
actual maxima. Status serializes into the shared 2048-byte publish scratch
buffer, so this costs no extra RAM; only the PSRAM-allocation-failed stack
fallback grows by 256 bytes. A static_assert keeps the status ceiling within
the shared buffer.
Found by adversarial review of the preceding commit.
A slot that cannot establish TLS reported nothing. The per-slot counters
track publish successes and failures, and a slot that never connects never
attempts a publish -- so its ok count freezes and its error count stays at
zero for the entire outage. Observed on hardware: a slot sat dead for over
five hours while `sN=` showed 0 errors and the published status payload
carried no indication at all.
Two surfaces gain the missing signal:
- `get mqtt.stats` gains `down=<n>` (omitted when zero, like `filt=`) and
marks each disconnected slot with a trailing `!`. The marker goes after
the counts so existing "s(\d)=(\d+)/(\d+)" parsers keep matching.
- The published status payload gains `mqtt_slots_up`, `mqtt_slots_total`,
`heap_largest`, and `mqtt_outage_secs`. These ride on every healthy slot's
status message, because the slot that is down cannot report itself.
`mqtt_slots_up < mqtt_slots_total` is the condition to alert on.
`mqtt_outage_secs` is detail: it is present only when the outage has a known
start, so its absence does not imply health -- a slot that has never
connected since boot has no outage start time.
`heap_largest` is included because free heap alone does not predict a failed
connection. A slot needs one contiguous 16,384-byte block for its TLS record
buffers; measured immediately before an attempt this value predicted the
outcome in every observed case, while total free heap was 85 KB throughout a
multi-hour outage. A low value with every slot up is normal rather than a
fault -- established sessions already hold their buffers.
Publish-error semantics are unchanged: connect failures are not folded into
them.
Adds host coverage for the new fields, for the outage key's presence rule,
and for worst-case payload size. That last one matters because
serializeComplete() returns 0 and clears the buffer on overflow and the
bridge publishes only a positive length, so exceeding the 768-byte status
buffer would silently stop status publication -- the same blindness this
change exists to remove.
The reconstruction-less fallback path in buildPacketJSON (queued packet
with no captured raw frame that cannot be re-serialized) published
fabricated SNR 12.5 / RSSI -65 that consumers could not distinguish
from real radio readings. It now passes NAN / MQTT_RSSI_UNKNOWN, and
buildPacketMessage omits the SNR/RSSI keys entirely when no measurement
exists, mirroring the existing score omission.
Ports the fix from ACETyr's PR #13 (0a037040) on mqtt-bridge-implementation.
Co-Authored-By: ACETyr <35745290+ACETyr@users.noreply.github.com>
agessaman