abbreviate_wind_direction matched full words (NORTHWEST, NORTH, ...), but NOAA's forecast windDirection is always an abbreviation like WNW, SSE or N. Every value fell through to the fallback, which kept the first two characters behind a generic wind emoji, so replies read '💨WN2' or '💨SS1' and even plain N and NE lost their arrow. The 16 compass abbreviations now return the nearest 8-point arrow plus the abbreviation; full words and other values behave as before. Found while pinning wx's formatting on real NOAA responses for the maintainability refactor.
Review found the same reply-limiter drop in two more places: the earthquake service's link post right after its alert, and the announcements command's 'sent' confirmation right after the announcement. Both now skip the reply limiter. multitest called wait_for_tx itself just before send_response, which waits again; with wait_for_tx claiming the slot that pre-wait made the reply wait a full extra interval, so it goes. A test that only documented RateLimiter's behavior, and guarded nothing, is replaced by an earthquake test that drives _check_earthquakes.
A user who raised bot_tx_rate_limit_seconds to 5 noticed wx's weather-then-alerts reply did not seem to honor it. The gap between those two messages was in fact longer than the limit (every send already waits on the bot TX limiter in _check_rate_limits, and wx sleeps limit + 1 s on top), but looking at it turned up two real bugs.
The second part of wx's and gwx's two-part reply, and every part after the first of 'alert all', went through the reply rate limiter (rate_limit_seconds, 10 s by default) a few seconds after the first part had used it, so it was refused: with default settings the alert message was never sent. These now pass skip_user_rate_limit=True, as the other multi-part senders already did. The repeater command's two-part branch gets the same change for consistency.
BotTxRateLimiter.wait_for_tx only checked the time since the last recorded transmission, and record_tx runs after the radio confirms, so concurrent senders (two replies, a reply and a scheduled message) all passed the wait before any of them recorded and transmitted back to back. wait_for_tx now claims the slot under the lock when it returns. A coverage test that monkeypatched can_tx to drive the old loop is rewritten against the new one.
A second review of the restart fix found more of the same class. The map uploader's stop() removes its file log handler and start() never added it back, so after one restart it logged nothing to file; start() now restores it. DARC MoWaS set _running before binding its port, so a port in use left it dead but healthy, and werkzeug's make_server calls sys.exit when the port is taken, which got past every except Exception and stopped the bot; start() now binds first and turns that exit into an OSError.
Boot-time starts go through _start_service_at_boot, which records a restart failure when start() raises, so that service also waits out the backoff, and no longer reports a disabled service as a configuration problem. The health-loop step is _restart_unhealthy_services. New tests cover both, the bridges' session cleanup and the map uploader restart; all 13 lifecycle tests fail on dev.
Review found that marking a not-running service inactive for good also retired services that only returned early for a transient reason: packet capture and the map uploader give up after waiting 30 s for the radio, and the bridges return when meshcore is momentarily unset, which is exactly when the health loop restarts them. Such a service now records a restart failure, so the next attempt waits out service_restart_backoff_seconds; a transient failure recovers, a missing configuration costs one attempt per backoff instead of one every 5 s. The restart decision is _service_restart_due, which the tests now exercise directly.
Also from the review: packet capture and the map uploader never cleared the should_exit flag stop() sets, so after a restart their background loops exited at once while the service reported healthy; start() now clears it. The Discord and Telegram bridges close the HTTP session they open when they return early because meshcore is not available.
configparser's getfloat accepts 'nan', and NaN fails every comparison, so it passed both the negative-value check and the min() cap and reached send_msg_with_retry as the ACK wait. It now falls back to the default like a negative value. 'inf' was already capped at 60 s; both cases are in the parametrized test, and the nan case fails on dev.
A fifth review found two gaps. A scheduled trial that returned False was re-armed at once, but False also follows exhausted no_event_received retries, so the radio could be tried again and again without another answered probe. And a reply's success settlement ran in a worker thread that a cancelled caller could cancel while it was still queued, stranding the trial in flight.
_end_offline_trial ends a trial that went out without succeeding (False or an exception, from a reply or a scheduled send) under the state lock alone; only paths where nothing reached the send still re-arm. The success settlement is shielded, so it completes even if the caller is cancelled. The cancellation test queues the settlement behind a busy one-worker executor and fails without the shield.
The health loop restarted every enabled service whose is_healthy() was False. A service that returns from start() without running because it has nothing to do (Discord or Telegram with no channels, the map uploader with no key) is never healthy, and its restart raised nothing, so it counted as a success and cleared the backoff: the service was restarted every 5 seconds, logging a warning each time.
Services whose start() returns without running are now recorded as inactive, logged once, and skipped by the health loop. A restart that leaves a service running but unhealthy records a failure, so the next attempt waits out service_restart_backoff_seconds.
Restarts also double-subscribed: services subscribed to meshcore events in start() and never unsubscribed (the comments said meshcore cleans up; it does not). BaseServicePlugin gains _subscribe and _unsubscribe_all; the Discord, Telegram, map uploader, packet capture and prefix-collision services subscribe through it, unsubscribe in stop(), and drop the old instance's subscription before re-subscribing after a transport reconnect. Tests use meshcore's real EventDispatcher and fail on dev.
Only scheduled messages and interval adverts could claim the trial armed by an answered health probe, so a bot with neither stayed offline however healthy the radio became. send_dm and send_channel_message now claim an armed trial when offline and run the send through _send_as_offline_trial, under the trial's identity; its result settles the trial (True clears the outage, False or an exception fails it until the next answered probe), so a dead radio still gets at most one attempt per probe. Settlement runs in a worker thread because it can write bot_metadata. Tests drive the real CommandManager guard, and the alert-thread counting in the recovery tests no longer stops asyncio.to_thread's workers.
A third review found four more gaps. A trial send that hit the 60 s wait kept running and could transmit later, during a newer trial; _record_send_success cleared any outage without checking whose success it was, so an old send finishing late could clear a newer one; the wrappers claimed the trial before logging and time lookup, outside the code that settles it; and the startup advert's timeout path recorded its failure on the event loop, where tripping the outage writes bot_metadata.
_record_send_success now takes the trial id and clears only when that trial is still in flight, deciding and resetting under the state lock (_reset_offline_state_locked); success of an ordinary send while offline leaves the outage alone. A trial whose wait fails is cancelled. Each wrapper settles the trial if anything after admission raises. The startup advert records its failure through asyncio.to_thread.
A second review found five gaps. A loop that closed between is_running() and submission stranded the trial in flight; fallback sends (no running main loop) settled every trial as inconclusive; a timed-out send that kept running passed the guard during a later trial, because the context flag was a bare True; the viewer clear checked the outage generation, released the lock, then cleared, so it could clear a newer outage; and the health loop ran the metadata sync, with its database retries, on the event loop.
Trials now carry an id: _admit_measured_send returns it, _as_offline_trial sets it in the context, and is_radio_offline and the settle methods only honor the trial currently in flight. The scheduler's two wrappers share _run_measured_send, which submits inside its error handling (closing the coroutine if submission fails), settles a fallback trial by its real outcome, and settles any trial left open in a finally. _clear_radio_offline_state takes an expected generation and checks it under the state lock. The health loop checks the 30 s throttle on the loop and runs the sync itself in a worker thread. The delayed-write test now waits for the newer outage to trip before releasing the stalled writer, so it fails without publish serialization.
Review of the circuit breaker found four problems. The trial was a shared flag that let every sender through, interactive sends never settled it, and it could stay open forever. The scheduler threw away send_channel_message's result, so a send the radio rejected counted as a success and cleared the outage. Metadata writes happened outside the state lock, so a stalled clear could overwrite a newer outage's 'true' and the viewer check would then cancel it. And one failed read-back of the persisted 'true' disabled the viewer clear for the rest of the outage.
Now a probe arms one trial, which the scheduler's next measured send (scheduled message or interval advert) claims atomically through _admit_measured_send; that send runs under a context flag so only it passes the send-path offline guards. Its real outcome settles it: success clears the outage, a timeout ends the trial until the next answered probe, and a False result or nothing sent re-arms it. _send_scheduled_message_async now returns whether the message went out. Metadata is published under its own lock from the current state, confirmed per outage generation, and retried from the health loop until confirmed. The in-process viewer clear calls the full clear. Tests go through the real scheduler on a live loop, race admission across threads, stall a clear write while a new outage trips, and count alerts from the first trip.
The threshold test timed one request, so a single GC pause or a busy CI runner could fail it. On the Python 3.10 runner it failed four times in a row at 0.22 to 0.26 s on a branch that does not touch the stats path, while the same request takes about 22 ms locally on both that branch and dev, with or without coverage. It now times five requests and asserts on the fastest, which an algorithmic regression still slows, and the failure message lists every timing.
Moving the geocode decision ahead of the await meant every advert that arrived while another was still geocoding read the same uncommitted row and geocoded again; six adverts for one contact made nine or more Nominatim requests instead of two, at the 1 req/s budget. track_contact_advertisement now holds a per-contact asyncio lock across the read, geocode and write, which restores the old one-at-a-time behavior for a contact while different contacts still proceed independently. The lock entry is dropped when its last holder leaves.
Also: geocoding_cache reads in the worker-thread helpers use .get() so the loop-thread expiry pass cannot raise KeyError between the check and the read, the write-block duplicate guard calls _is_duplicate_advert_packet, and the guard test loses leftover scaffolding.
Review follow-up. A migration script containing COMMIT, BEGIN, ROLLBACK, END, SAVEPOINT, RELEASE or VACUUM would end or escape the runner's transaction and bring back the non-atomic failure; _execute_statements now refuses those statements. Added tests for a rollback that undoes a column added to a pre-existing table (the legacy-database case), for a trailing comment after the last statement, and for the refusal. The changelog now notes that the write lock is held for the whole migration run.
Review follow-up. A get_time probe is a host-to-companion frame with no RF transmit, so it answers in exactly the failure mode the offline state exists for; clearing on it would flap offline and online every probe interval and send an alert email each time. A probe success now only opens a trial: sends are let through, and the next send the scheduler measures either clears the state or closes it again, without another alert. The alert goes out once per outage.
The reconnect clear is gone (a connect can still fail after the socket is up, and the probe covers recovery after it). The viewer-clear check runs only after this process has read back its own 'true', so a read that races the trip cannot cancel it; it is throttled to every 30 s and runs before the transport check, so it works while disconnected. State changes take a lock, since the scheduler thread and the event loop both write them.
track_contact_advertisement, _background_geocoding, scan_and_catalog_repeaters and populate_missing_geolocation_data are async but called rate-limited Nominatim synchronously (with the rate limiter's time.sleep), so the whole bot stalled for each lookup. The async-path guard test only scanned commands and service plugins, so it never saw this file.
The calls now go through asyncio.to_thread. track_contact_advertisement decides whether to geocode, and does it, before its write block, which stays free of awaits so concurrent adverts cannot interleave between its reads and writes; the duplicate-packet guard runs both before geocoding and again inside the write block. The guard test now scans repeater_manager.py and knows the state/country and full-location helpers; a new test shows advert geocoding runs off the loop thread and fails without the change.
MigrationRunner.run wraps all pending migrations in BEGIN IMMEDIATE and rolls back on failure, but nine migrations called cursor.executescript, which issues an implicit COMMIT before running. A failure in a later migration therefore left every earlier one, and its schema_version row, committed. Reproduced: appending a failing migration to a fresh database left 23 versions applied.
_execute_statements runs a script statement by statement with execute, splitting on ';' only where sqlite3.complete_statement agrees, so semicolons in string literals and comments are left alone. A fresh database migrates to a byte-identical sqlite_master and schema_version.
After radio_offline_threshold consecutive send timeouts the bot enters radio-offline state and every send path refuses to transmit. The only thing that cleared it was _record_send_success, which runs after a send goes out, so once offline the bot stayed offline until restarted. The web viewer's Clear Offline Flag could not help either: the viewer runs in its own process, so it only wrote bot_metadata, which the bot never read back.
Three paths now clear the state: a health probe the radio answers, a successful (re)connect, and the health loop noticing that bot_metadata says the flag was cleared. The new tests for the probe and viewer paths fail without the change.
send_channel_message compared the set_flood_scope reply's type to the string "ERROR", but it is an EventType whose value is "command_error", so the comparison never matched. A rejected regional scope sent the message at whatever scope the radio held, and a rejected restore to global flood left the radio pinned to the region for every later send, both without a log line. Compare against EventType.ERROR.
The existing tests for this path built their error reply with the same "ERROR" string, so they passed against the broken check; they now use EventType.ERROR and fail without the fix.
Review follow-up. With * listed, an uncorrelated message is still answered globally when no scoped packet was heard around it; the radio's default scope does apply to sends the bot leaves unscoped; [Scheduled_Messages] needs the leading #; and the outbound log lines are quoted as they are actually printed, including that the global one is DEBUG.
Review follow-up. The firmware's direct-path estimate includes hop count and about 750 ms of fixed allowance, not airtime alone, so the comments and changelog now say that. Values above 60 seconds are capped, since a value meant as milliseconds would otherwise hold one DM for hours, and the changelog notes that an unacknowledged DM now takes at least 24 seconds to report failure. The late-ACK test has a wider margin so it cannot flake on a loaded machine.
Review follow-up. The scope read could raise (a bare '%' under interpolation) after the 'sent' row and mesh cooldown were reserved, and the rollback never ran, so the log showed a warning that never went on the air. The scope and body are now prepared before anything is reserved, and a failure is recorded as a failed attempt. [Region_Warnings] flood_scope is read raw and validated, because set_flood_scope's failure path sends anyway at whatever scope the radio holds. Dry runs now log the truncated body the live send would use.
send_msg_with_retry was called with timeout=0, so each attempt waited 1.2 times the radio's suggested_timeout. The companion firmware derives that from airtime alone (500 + (6 x airtime + 250) x (hops + 1) ms direct, 500 + 16 x airtime flood), but the round trip also carries fixed costs that do not shrink with airtime: the recipient's ACK delay, repeater retransmit and rx delays, the radio's TX queue and the host link. On a fast preset a zero-hop attempt gave up after about 1.3 seconds, the ACK of a delivered message arrived after the last attempt, and the send reported failure. That is the residual #290 symptom: webhook DMs erroring within a couple of seconds for messages the recipient had.
New [Bot] dm_min_ack_timeout (default 8 seconds) is passed as min_timeout. An ACK still ends the wait at once, so successful sends are no slower, and since the radio lock is per-frame the longer wait on a failing DM does not hold up other sends. 0 restores the old behavior. The new test drives the real meshcore retry loop with an ACK slower than the radio's estimate and fails without the floor.
Adam Gessaman