core.py conflicts were S3's simplifications of is_radio_offline, _record_send_failure and _record_send_success colliding with #313's rewrite of the same methods; dev's versions are kept.
A fifth review found two gaps. A scheduled trial that returned False was re-armed at once, but False also follows exhausted no_event_received retries, so the radio could be tried again and again without another answered probe. And a reply's success settlement ran in a worker thread that a cancelled caller could cancel while it was still queued, stranding the trial in flight.
_end_offline_trial ends a trial that went out without succeeding (False or an exception, from a reply or a scheduled send) under the state lock alone; only paths where nothing reached the send still re-arm. The success settlement is shielded, so it completes even if the caller is cancelled. The cancellation test queues the settlement behind a busy one-worker executor and fails without the shield.
Only scheduled messages and interval adverts could claim the trial armed by an answered health probe, so a bot with neither stayed offline however healthy the radio became. send_dm and send_channel_message now claim an armed trial when offline and run the send through _send_as_offline_trial, under the trial's identity; its result settles the trial (True clears the outage, False or an exception fails it until the next answered probe), so a dead radio still gets at most one attempt per probe. Settlement runs in a worker thread because it can write bot_metadata. Tests drive the real CommandManager guard, and the alert-thread counting in the recovery tests no longer stops asyncio.to_thread's workers.
A third review found four more gaps. A trial send that hit the 60 s wait kept running and could transmit later, during a newer trial; _record_send_success cleared any outage without checking whose success it was, so an old send finishing late could clear a newer one; the wrappers claimed the trial before logging and time lookup, outside the code that settles it; and the startup advert's timeout path recorded its failure on the event loop, where tripping the outage writes bot_metadata.
_record_send_success now takes the trial id and clears only when that trial is still in flight, deciding and resetting under the state lock (_reset_offline_state_locked); success of an ordinary send while offline leaves the outage alone. A trial whose wait fails is cancelled. Each wrapper settles the trial if anything after admission raises. The startup advert records its failure through asyncio.to_thread.
A second review found five gaps. A loop that closed between is_running() and submission stranded the trial in flight; fallback sends (no running main loop) settled every trial as inconclusive; a timed-out send that kept running passed the guard during a later trial, because the context flag was a bare True; the viewer clear checked the outage generation, released the lock, then cleared, so it could clear a newer outage; and the health loop ran the metadata sync, with its database retries, on the event loop.
Trials now carry an id: _admit_measured_send returns it, _as_offline_trial sets it in the context, and is_radio_offline and the settle methods only honor the trial currently in flight. The scheduler's two wrappers share _run_measured_send, which submits inside its error handling (closing the coroutine if submission fails), settles a fallback trial by its real outcome, and settles any trial left open in a finally. _clear_radio_offline_state takes an expected generation and checks it under the state lock. The health loop checks the 30 s throttle on the loop and runs the sync itself in a worker thread. The delayed-write test now waits for the newer outage to trip before releasing the stalled writer, so it fails without publish serialization.
Review of the circuit breaker found four problems. The trial was a shared flag that let every sender through, interactive sends never settled it, and it could stay open forever. The scheduler threw away send_channel_message's result, so a send the radio rejected counted as a success and cleared the outage. Metadata writes happened outside the state lock, so a stalled clear could overwrite a newer outage's 'true' and the viewer check would then cancel it. And one failed read-back of the persisted 'true' disabled the viewer clear for the rest of the outage.
Now a probe arms one trial, which the scheduler's next measured send (scheduled message or interval advert) claims atomically through _admit_measured_send; that send runs under a context flag so only it passes the send-path offline guards. Its real outcome settles it: success clears the outage, a timeout ends the trial until the next answered probe, and a False result or nothing sent re-arms it. _send_scheduled_message_async now returns whether the message went out. Metadata is published under its own lock from the current state, confirmed per outage generation, and retried from the health loop until confirmed. The in-process viewer clear calls the full clear. Tests go through the real scheduler on a live loop, race admission across threads, stall a clear write while a new outage trips, and count alerts from the first trip.
Review follow-up. A get_time probe is a host-to-companion frame with no RF transmit, so it answers in exactly the failure mode the offline state exists for; clearing on it would flap offline and online every probe interval and send an alert email each time. A probe success now only opens a trial: sends are let through, and the next send the scheduler measures either clears the state or closes it again, without another alert. The alert goes out once per outage.
The reconnect clear is gone (a connect can still fail after the socket is up, and the probe covers recovery after it). The viewer-clear check runs only after this process has read back its own 'true', so a read that races the trip cannot cancel it; it is throttled to every 30 s and runs before the transport check, so it works while disconnected. State changes take a lock, since the scheduler thread and the event loop both write them.
A codex review of the branch found seven places where earlier refactor commits changed behavior. Each is restored here with a test:
- base_service imported flood_scope relatively; the local-service loader also runs that file as local_services.base_service, where the import cannot resolve. Now absolute.
- Radio-health fields were initialized after commands and services were built, so a plugin constructor reading is_radio_offline raised. They are now set before any plugin.
- Removing the per-command can_execute overrides made 31 commands honor skip_channel_check, which their overrides had dropped. BaseCommand.honors_skip_channel_check (False on exactly those commands) keeps them enforcing their channel list.
- CommandManager.get_plugin_metadata, WxCommand.get_weather_for_zipcode and PluginLoader.validate_plugin were public compatibility entry points; they are back.
- feed_manager stopped exporting DEFAULT_MAX_FEED_RESPONSE_BYTES and DEFAULT_MAX_PARSED_FEED_ITEMS despite the commit saying it re-exported them; it does now.
- wx and gwx logged companion-location lookup failures at WARNING; the shared helper logged at debug. It now takes the level.
- The INFO 'MESSAGE ROUTING' / 'DIRECT MESSAGE' lines for pubkey-correlated DMs were removed with the dead path_info computation; they are logged again by a log-only helper.
After radio_offline_threshold consecutive send timeouts the bot enters radio-offline state and every send path refuses to transmit. The only thing that cleared it was _record_send_success, which runs after a send goes out, so once offline the bot stayed offline until restarted. The web viewer's Clear Offline Flag could not help either: the viewer runs in its own process, so it only wrote bot_metadata, which the bot never read back.
Three paths now clear the state: a health probe the radio answers, a successful (re)connect, and the health loop noticing that bot_metadata says the flag was cleared. The new tests for the probe and viewer paths fail without the change.
The local translation path defaulted to a bare `local/translations/`, resolved
against the process cwd and unrelated to `[Bot] local_dir_path`. Since
`local_dir_path` already selects where an operator's commands, service plugins
and config overlay live, the catalog belongs in that same tree. It now defaults
to `<local_dir_path>/translations`, resolved absolute against the bot root, so
relocating `local_dir_path` moves the catalog with it and the lookup no longer
depends on the working directory. An explicit `local_translation_path` still
wins.
Only the constructor at startup was passing the local path. `get_translator()`
builds and caches a translator per detected language, and `reload_config()`
builds a fresh one, and both were still constructing `Translator` with the
distributed path alone. So local overrides silently vanished from any reply that
used the sender's language, and did not survive a config reload. Both now pass
it, `reload_config()` republishes it alongside `translation_path`, and it is
snapshotted and restored on rollback. The DummyTranslator fallback sets it too,
since `get_translator()` would otherwise raise AttributeError there.
Collapsed `_deep_merge_translations` into the existing `_merge_translations`,
which already merged deeply with the primary winning. Rewrote that one to stop
mutating its input: it shallow-copied `fallback` and then recursed into the
shared sub-dicts, so merging a base language over English corrupted the cached
English catalog at every level below the first. Also replaced the two `print()`
calls on the error paths with logger calls.
Registered `local_translation_path` in the config schema next to
`translation_path`, and reverted a trailing-whitespace reflow that touched 84
lines of config.ini.example for a one-key addition.
Tests: local overlay (single-string override, added keys, local-only catalog,
missing directory), the merge no longer mutating its fallback, the default
following `local_dir_path`, an explicit setting overriding it, the overlay
reaching `bot.translator` end to end, and a reload picking up a moved path.
Added methods to dynamically reserve a free port and wait for the admin server to start accepting connections. This improves test reliability by preventing port binding conflicts and ensuring the server is ready before sending requests. Updated existing tests to utilize these new methods, removing fixed sleep calls for better performance and accuracy.
Reworked implementation of the multilingual-response feature (issue #218)
as a reusable, framework-level capability instead of logic bolted onto
HelloCommand.
- modules/lang_detector.py: keyword-first detection (reliable for the short
greetings typical on mesh) with optional langdetect for longer text;
results are constrained to languages that actually have a translation file.
- core.py: cache Translator instances by language (get_translator) so
per-message switching never re-reads files, and expose available_languages()
derived from translations/*.json. Store translation_path; keep cache/path
consistent on config reload.
- base_command.py: task-local translator override via a ContextVar (safe under
asyncio concurrency, no global mutation), plus detect_response_language() and
a respond_in_sender_language() context manager any command can opt into.
- hello_command.py: build the reply inside respond_in_sender_language(); the
only awaited call (send_response) stays outside the override window.
- Gated by [Localization] auto_detect_language (default false); langdetect is
an optional [lang] extra, not a hard dependency.
Tests: unit coverage for the detector and the base-command wiring.
- reload_config: clear all sections before re-reading so keys deleted from
config.ini (e.g. rows removed in the settings UI) don't survive in memory
(ConfigParser.read merges instead of replacing)
- reload_config: re-instantiate command plugins so settings cached in
__init__ (including 'enabled') take effect on reload
- /api/plugins save: treat dynamic_sections / repeating_blocks absent from
the payload as 'don't touch' instead of 'delete all managed keys'
- settings view: respect per-plugin settings_enabled_default so opt-in
commands (Announcements, Greeter) display as off when unconfigured
- ruff: drop unused import, organize command_manager imports
- add tests for ini_writer, settings_schema, settings_store and the
/api/plugins endpoints (38 cases) plus reload regression tests
- Added `on_transport_reconnected` method to service plugins to re-establish event subscriptions after a transport reconnect.
- Updated `core.py` to notify running services of transport reconnections.
- Enhanced logging for services to track re-subscription actions.
- Added tests to verify the behavior of service plugins during transport reconnects.
- Added configuration options for reconnect behavior, including `reconnect_max_retries`, `reconnect_delay_seconds`, and `reconnect_max_delay_seconds`.
- Enhanced the bot's ability to handle transport disconnects by scheduling reconnect attempts with exponential backoff.
- Updated documentation to reflect new connection settings and behavior.
- Added tests to verify reconnect logic and ensure proper handling of transport errors.
Added asyncio timeout handling for sending startup and interval-based adverts, improving reliability by recording failures on timeout. Updated relevant tests to ensure coverage for timeout scenarios and increment failure counts appropriately. Introduced a new Radio Reliability panel in the web viewer for monitoring radio status.
- USE-05: Add /api-explorer page listing all ~65 API endpoints in 9
categories (System, Contacts, Mesh, Channels, Feeds, Radio, Admin,
Maintenance, Config, Greeter) with method badges, descriptions, and
curl example modal. Filter bar and collapse per section. Nav item
added to base.html.
- USE-06: Three targeted error-message improvements:
1. 500 handler now returns user-friendly HTML page (error.html) for
browser requests and sanitized JSON for API/JSON requests instead
of a bare string.
2. Feed processed-items query failures promoted from logger.debug to
logger.warning so operators see them in normal log output.
3. Global JS fetch interceptor in base.html redirects to /login?next=
on any 401 response, handling session expiry mid-page.
- Fix pre-existing test bug: test_reload_endpoint_success mock return
value did not match actual code message from reload_config.
- core.py: add _BotAdminServer daemon thread (Flask, 127.0.0.1 only,
bearer token auth); POST /api/admin/reload calls reload_config() and
returns JSON {success, message}; GET /api/admin/health; started from
start() when [Admin] enabled = true and token is set
- scripts/reload_config.sh: curl wrapper for the reload API; reads
port/token from config.ini [Admin] section; exits 1 on rejection
- tests/test_core.py: TestBotAdminServer — 7 tests covering server
creation, missing token guard, reload success/failure/auth, health
Clean up residual cherry-pick conflict markers and keep SMTP guidance in config templates brief while preserving full behavior in code and tests.
Made-with: Cursor
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
Test modules:
- test_enums: enum values and flag combinations
- test_models: MeshMessage dataclass field and type validation
- test_transmission_tracker: full TransmissionTracker coverage
- test_message_handler: path parsing, RF correlation, message routing
- test_repeater_manager: role detection, ACL, device type classification
- test_core: config loading, radio settings, reload paths
Tracking files:
- BUGS.md: known bugs and fix history log
- TESTING.md: test strategy, coverage targets, and how-to guide
- TODO.md: feature and task backlog with completion status
- scripts/update_todos.py: scans source for # TODO/FIXME/HACK markers
and regenerates the Inline TODOs section in TODO.md
agessaman