core.py conflicts were S3's simplifications of is_radio_offline, _record_send_failure and _record_send_success colliding with #313's rewrite of the same methods; dev's versions are kept.
A fifth review found two gaps. A scheduled trial that returned False was re-armed at once, but False also follows exhausted no_event_received retries, so the radio could be tried again and again without another answered probe. And a reply's success settlement ran in a worker thread that a cancelled caller could cancel while it was still queued, stranding the trial in flight.
_end_offline_trial ends a trial that went out without succeeding (False or an exception, from a reply or a scheduled send) under the state lock alone; only paths where nothing reached the send still re-arm. The success settlement is shielded, so it completes even if the caller is cancelled. The cancellation test queues the settlement behind a busy one-worker executor and fails without the shield.
A third review found four more gaps. A trial send that hit the 60 s wait kept running and could transmit later, during a newer trial; _record_send_success cleared any outage without checking whose success it was, so an old send finishing late could clear a newer one; the wrappers claimed the trial before logging and time lookup, outside the code that settles it; and the startup advert's timeout path recorded its failure on the event loop, where tripping the outage writes bot_metadata.
_record_send_success now takes the trial id and clears only when that trial is still in flight, deciding and resetting under the state lock (_reset_offline_state_locked); success of an ordinary send while offline leaves the outage alone. A trial whose wait fails is cancelled. Each wrapper settles the trial if anything after admission raises. The startup advert records its failure through asyncio.to_thread.
A second review found five gaps. A loop that closed between is_running() and submission stranded the trial in flight; fallback sends (no running main loop) settled every trial as inconclusive; a timed-out send that kept running passed the guard during a later trial, because the context flag was a bare True; the viewer clear checked the outage generation, released the lock, then cleared, so it could clear a newer outage; and the health loop ran the metadata sync, with its database retries, on the event loop.
Trials now carry an id: _admit_measured_send returns it, _as_offline_trial sets it in the context, and is_radio_offline and the settle methods only honor the trial currently in flight. The scheduler's two wrappers share _run_measured_send, which submits inside its error handling (closing the coroutine if submission fails), settles a fallback trial by its real outcome, and settles any trial left open in a finally. _clear_radio_offline_state takes an expected generation and checks it under the state lock. The health loop checks the 30 s throttle on the loop and runs the sync itself in a worker thread. The delayed-write test now waits for the newer outage to trip before releasing the stalled writer, so it fails without publish serialization.
Review of the circuit breaker found four problems. The trial was a shared flag that let every sender through, interactive sends never settled it, and it could stay open forever. The scheduler threw away send_channel_message's result, so a send the radio rejected counted as a success and cleared the outage. Metadata writes happened outside the state lock, so a stalled clear could overwrite a newer outage's 'true' and the viewer check would then cancel it. And one failed read-back of the persisted 'true' disabled the viewer clear for the rest of the outage.
Now a probe arms one trial, which the scheduler's next measured send (scheduled message or interval advert) claims atomically through _admit_measured_send; that send runs under a context flag so only it passes the send-path offline guards. Its real outcome settles it: success clears the outage, a timeout ends the trial until the next answered probe, and a False result or nothing sent re-arms it. _send_scheduled_message_async now returns whether the message went out. Metadata is published under its own lock from the current state, confirmed per outage generation, and retried from the health loop until confirmed. The in-process viewer clear calls the full clear. Tests go through the real scheduler on a live loop, race admission across threads, stall a clear write while a new outage trips, and count alerts from the first trip.
send_nightly_email, send_zombie_alert_email and send_radio_offline_alert_email each read the same notif.* SMTP settings, parsed the port the same way, and repeated the same SSL/STARTTLS/plain transport. maintenance.read_smtp_settings returns an SmtpSettings record and send_smtp_message delivers a message with it. Each sender keeps its own enable checks, recipients, validation messages and body.
Full suite: all pass except tests/unit/test_wx_location_prefix_budget.py::test_a_prefix_is_actually_in_play, which makes a live Nominatim call and fails the same way on dev at the moment.
run_scheduler scheduled feed polling, channel, radio and config operations and the feed message queue on the main loop with five copies of run_coroutine_threadsafe plus a done-callback that logged failures. _submit_to_main_loop(coro, message, log_traceback=True) does both; feed polling keeps logging without a traceback, the others with one, and the messages are unchanged. The temporary-event-loop fallbacks are left as they are until their reachability is settled.
Five copies read meshcore.self_info as a dict or an object with slightly different rules. self_info_name(self_info, fields, by_presence=False) returns the first truthy field, or with by_presence the first field an object has at all, which is how BaseCommand has always read objects (so name='' on an object still falls back to config there, but not in CommandManager). Each caller passes its own field order and keeps its own fallback: budgets read name/user_name, mention matching reads name/adv_name. The device-name sync in core is left alone, since it logs the raw value. The identity golden file is unchanged.
Every name removed here has zero references across modules, tests, docs, local plugins, scripts and generate_website.py, and none is part of the documented plugin surface (db_manager, command_manager send methods, channel_manager stay untouched). About 1,160 lines:
- multitest: four formatter helpers that only referenced each other (the flat and nested variants are the ones in use)
- path: _get_api_cache_data, _filter_recent_repeaters
- repeater command: _handle_contact_stats, _get_geocoding_status
- airplanes: _send_split_response; greeter: get_recent_greeted_users; plugin_loader: validate_plugin
- wx: get_weather_for_zipcode, abbreviate_alert_title, get_current_conditions
- message_handler: discover_message_path, and an ADVERT debug line behind a self.debug attribute that is never set
- utils: _get_node_location_and_key_from_db
- repeater_manager: test_meshcore_cli_commands, toggle_auto_add, restore_repeater, get_purging_stats, populate_missing_geolocation_data, get_daily_advertisement_stats, get_nodes_per_day_stats
- command_manager: get_available_commands_list and get_plugin_metadata (HelpCommand and PluginLoader have the live versions)
- base_command: _load_command_prefix, and a no-op .replace('_', '_')
- scheduler: the first of two _run_data_retention definitions, shadowed by the second since it was added
- core: the write-only last_clock_sync_time, and _setup_routing_capture, which only logged a line
Full suite, ruff and mypy pass, and the characterization golden files are unchanged.
Builds on #303. Channel text is capped at 155 bytes of "<name>: <text>" instead of 124. The firmware encrypts a 4-byte timestamp and a 1-byte text type ahead of the text in 16-byte AES blocks, so 155 fills the 160-byte block exactly (163-byte payload), and the last 5 bytes up to the firmware's 160 would spill into another block (179-byte payload).
124 was set on the premise that repeaters drop 165-byte frames. They don't: allowPacketForward has no size check and MAX_PACKET_PAYLOAD is 184, and 163-byte GRP_TXT payloads are routinely seen at 3+ hops. The frame-budget tests now pin the limit within the firmware's 160 and to a whole cipher block instead of to a 131-byte relay ceiling.
Also corrects the payload model's plaintext header from 4 bytes to 5, and moves a test import back above the helper that had split the import block.
Region scopes had to be hand-edited in config.ini, and the radio's own default
region could not be set from the bot at all. Both now live on the Radio page.
Node Settings gains a Default Region Scope field beside the path hash size,
which is the firmware's own setting — NodePrefs.default_scope_name and
default_scope_key, the region the radio falls back to for any send the bot does
not scope itself. Firmware that has no such command is reported as not having
answered rather than shown as an empty field, and a stored key that is not the
stored name's hash is flagged, because the radio routes by the key and the name
beside it is only a label. A build-flag default stores the bare name while
hashing the '#' form, so the comparison normalizes first.
Clearing it does not go through the meshcore library, because none of its reset
paths can: set_default_flood_scope(None) raises on len(None), "" earns
ILLEGAL_ARG from the firmware, and "*" only works because the frame is padded by
character count rather than by the name it wrote, landing one byte short of the
length the firmware reads as "a scope follows". The firmware's own contract for
clearing is a bare CMD_SET_DEFAULT_FLOOD_SCOPE, so that is what goes out. The
same padding bug is why a device scope name must be ASCII: a multi-byte
character displaces the transport key out of its field.
A separate Region Scopes card covers the bot's side — [Channels] flood_scopes as
an explicit choice between "reply whatever the scope" and an allowlist, and
outgoing_flood_scope_override. It writes config.ini, queues a hot reload, then
polls that reload and reports what the bot did with it, including saying plainly
when nothing picked it up. Per-channel flood_scope.<channel> entries are listed
read-only, since they are the reason a channel can ignore the default.
The two settings interact in a way worth stating on the page: once the bot sends
a scoped message it restores with set_flood_scope("*"), which leaves the radio in
forced-unscoped mode, so the device default stops applying until the bot scopes
another send.
Also from building it:
- flood_scopes left in [Bot] is surfaced rather than hidden. CommandManager
still honours it when [Channels] is empty, so a page that ignored it would
show "replies to every scope" while the bot enforced an allowlist — and
saving "off" hands the allowlist straight back to [Bot], which the banner now
says instead of claiming the old key stops mattering.
- outgoing_flood_scope_override = none is read as global flood on the send path,
as it already was everywhere else. send_channel_message tested the raw value
against a fixed tuple, so the lowercase spelling became the region "#none".
- Scope normalization moved to modules/flood_scope.py so the viewer, a separate
process, can validate a typed name without importing the bot's command
machinery. CommandManager delegates to it.
- Dark mode named only .border, so a .border-start divider drew Bootstrap's
light #dee2e6 onto a dark card. All four directional utilities now match.
Two fixes from the previous round were incomplete:
- The scheduled-message chunk budget used the schedule's explicit scope,
but send_channel_message resolves an unset scope from
flood_scope.<channel> and then outgoing_flood_scope_override. A
schedule with an implicit regional scope was therefore sized for a
global send and every chunk could overshoot once the sender added the
regional header. The budget now resolves the effective scope, and
assumes regional if that resolution fails, since guessing regional only
ever makes chunks smaller.
- Resetting _last_path_distance_km per request fixed sequential reuse but
not concurrency: the decode awaits a database lookup, and the
dispatcher runs handlers as independent tasks, so two path commands can
interleave and render each other's distance. The distance now rides on
the request's own message, with the instance attribute kept only as a
fallback for direct calls.
Two new findings:
- rf_data_is_correlated() treated pubkey and partial-prefix matches as
packet-unique, but a sender prefix identifies a sender, not one
transmission. With several cached packets from the same sender, the
first (usually oldest) was returned and allowed to supply a route.
Those strategies now take the newest match and are authoritative only
when the match is unambiguous; otherwise the entry is marked fallback,
so it still provides SNR/RSSI but never a route.
- The flood_scopes allowlist accepted a scope resolved from an
uncorrelated fallback packet. The HMAC proves the cached packet is in
an allowed scope, not that this message is, so a recent allowed-scope
packet could admit an unrelated message. Scope authorisation now
requires packet-bound correlation and logs plainly when it does not
have it.
That last one is a deliberate fail-closed change to an authorisation
path that predates tonight. Channel messages normally carry raw_hex and
correlate exactly, so the fallback is the exception rather than the rule,
but a deployment using flood_scopes will now stay quiet in cases where it
previously replied on an assumed scope.
Correctness:
- {path_distance} was always blank in production. The resolution code that
builds repeater_info dropped latitude/longitude in every branch, so the
calculator could never find a coordinate. My tests passed hand-built
dicts straight to the calculator and never exercised the builder, which
is why they stayed green. Coordinates are now carried through all four
construction sites, and the new tests drive _lookup_repeater_names via
its lookup_func hook so the real builder runs.
- The #80 route guard was defeated two ways in the channel handler. When
the RF data was an uncorrelated fallback, control fell through to the
raw-hex and routing_info fallbacks below, which took the route from the
unrelated packet anyway; the guard had actually made that path
reachable. message.routing_info was also assigned unconditionally, and
the path command reads it. "Not attributable" is now a terminal branch
and the routing_info hand-off checks provenance.
- Same fix was incomplete for DMs: routing_info was captured and turned
into path_info before the provenance check ran, so the later check only
declined to overwrite an already-wrong value. Guarded at the source.
- Rendering could transmit for real. Capture only intercepts
send_response, but advert calls send_advert() directly and
send_response_chunked never checked capture_sink. Chunked sends are now
captured, and rendering is opt-in via BaseCommand.render_safe (default
False) instead of a denylist that cannot be complete. This also closes
the DM-only leak: schedule is not marked safe, so {cmd:schedule} can no
longer broadcast configuration to a channel.
- Multi-part rendered output was rejoined into one oversized send.
Scheduled messages are now split to the RF body budget and sent through
send_channel_messages_chunked, on character boundaries so multi-byte
text is not corrupted.
- _last_path_distance_km is instance state that was only set on success,
so an invalid path request could show the previous request's distance.
Reset at the start of every execute().
- Stale-contact retries were only counted on non-OK results, so timeouts
and exceptions left a contact eligible forever and could recreate the
storm. All failed attempts count now.
Web viewer:
- A failed config reload was reported as a successful save. The API and
UI now distinguish "saved and active" from "saved, restart needed".
- Preview count was unbounded; clamped to 1-20.
- update_ini_values is a read-modify-replace with no locking, so two
concurrent viewer saves could lose one. Serialised behind a lock.
Two guards on command placeholders in scheduled messages, neither
configurable, because both exist to protect a shared medium:
A 15-minute floor. A schedule containing {cmd:...} that fires more often
is rejected at startup with an error rather than quietly running slower.
The interval is sampled and measured by the tightest gap between
firings, so "0,1 * * * *" is correctly treated as every 60 seconds and
not as hourly. Schedules without a command placeholder are unaffected.
The command's own cooldown_seconds now applies to a render. Scheduling
is not a way around the rate a command was configured to run at. The
execution is recorded before the command runs, matching execute_commands,
so a slow or failing render cannot be retried straight past the cooldown.
Also fills documentation gaps from the preceding commits:
- --install-extras was only in the script's own --help; documented in
service-installation.md (including alongside --update-venv) and
upgrade.md.
- weather-service.md documented weather_alarm's once-a-day scheduling
with no route to more than one forecast a day, which is exactly what
people go there looking for. It now points at {cmd:wx ...} and notes
that sunrise/sunset still belong to weather_alarm.
A scheduled message can now embed the reply of any bot command:
[Scheduled_Messages]
0 6,12,18 * * * = Public:{cmd:wx Seattle}
The schedule side was never the missing piece — [Scheduled_Messages]
keys have been 5-field cron with @presets for a while, so multiple
times, intervals and @hourly were all already expressible. What was
missing was any way to get a service's output into a scheduled message:
placeholders were limited to mesh info (contact counts). That gap is
why services were growing their own schedule parsers.
CommandManager.render_command_output runs a command for its text alone.
MeshMessage.capture_sink makes that safe: when set, send_response
collects the reply and transmits nothing, checked before the
_last_response bookkeeping so a background render cannot overwrite the
response captured for a real user's command.
Degrades quietly in every failure mode — unknown, disabled, admin-only,
non-renderable, timed out or silent commands expand to nothing, so raw
{cmd:...} text is never put on the air, and a message left empty is not
sent at all. Command output is not re-scanned, so a reply containing
{cmd:...} cannot recurse. Commands that transmit directly rather than
returning text (announcements) are refused outright, since rendering
them would broadcast for real.
Bounded by [Bot] scheduled_command_timeout_seconds (default 30).
- Updated mesh graph path splitting and aggregation to run in SQLite, improving performance by avoiding Python materialization.
- Defaulted graph persistence to batched writes for new installations, reducing WAL churn and SD-card writes.
- Enhanced data retention to execute shortly after startup, independent of the nightly maintenance schedule.
- Added a table-specific index for `mesh_connections` to support window and retention queries, ensuring efficient data access.
- Enhanced the coroutine execution logic in the MessageScheduler to ensure proper closure of coroutines when the main event loop is not running, preventing potential resource leaks and runtime warnings.
- Updated tests to utilize a dedicated method for retrieving scheduled message jobs, improving clarity and maintainability of the test suite.
Changed the default setting for `auto_manage_contacts` from `false` to `device` across configuration files and updated related documentation. This ensures that the device handles auto-addition of contacts while the bot manages capacity. Adjusted comments and documentation to reflect this change for clarity and consistency.
Adds device-level companion settings to /radio, headlined by the response
path hash size (mode 0-2 = 1-3 bytes per hop) wired to the existing
firmware config endpoints. New sections: Identity & Adverts (name, advert
lat/lon, location policy, zero-hop/flood advert buttons), Mesh Behavior
(extra ACKs, telemetry permissions), and write-only Advanced Tuning
(RX delay base, airtime factor, sent x1000 per the wire format).
Backend: GET /api/radio/params now returns the full SELF_INFO node fields;
POST accepts the new fields with validation; new POST /api/radio/advert;
scheduler ops handle the writes (other-params group as one read-modify-write
frame so partial updates never clobber device state).
Config-managed settings are not writable from the panel: new-contact mode
is owned by [Bot] auto_manage_contacts and shown read-only; the node name
is locked while bot_name + auto_update_device_name manage it. loop.detect
is removed from the firmware endpoints entirely - it is repeater/room-server
CLI config, and companion custom vars map to sensor settings.
Also: dark-mode styling for disabled form fields (base.html), TASK-01 guard
test updated for the deliberate reintroduction, 23 endpoint/template tests.
Scheduled cron jobs now pass the cron key into the send path, apply a
deterministic [0, N) second delay from scheduled_message_max_stagger_seconds
(default 1.5), and call send_channel_message with skip_user_rate_limit=True
so simultaneous jobs are not dropped by rate_limit_seconds. Per-channel and
bot_tx limits still apply. Document the new Bot key in config.ini.example;
tests assert kwargs, skip flag, and stagger behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Enhanced the FeedManager to prevent coroutine pileup by checking the lock status before acquiring it, allowing for more efficient message processing.
- Introduced a persistent dictionary to track the last send time for each feed, ensuring send intervals are respected without blocking.
- Updated the MessageScheduler to fire-and-forget the message processing, improving responsiveness and logging errors without blocking the main thread.
- Updated the scheduled message format in `config.ini.example` to support 5-field cron expressions and preset aliases, replacing the deprecated HHMM format.
- Improved the `MessageScheduler` class to parse and validate new schedule formats, logging warnings for deprecated usage.
- Adjusted the `ScheduleCommand` to display scheduled messages with their respective cron or preset labels.
- Added unit tests to ensure correct parsing and handling of various schedule formats, including legacy HHMM and cron expressions.
Merge latest dev into the PR branch and keep both DateTrigger and EventType imports so scheduler changes remain compatible and conflict-free.
Made-with: Cursor
Prevent exportView runtime failure by restoring the download anchor setup and move radio param imports to module scope so Ruff passes on the PR changes.
Made-with: Cursor
- Enhanced configuration options for `auto_manage_contacts` to support 'device' mode, allowing firmware to handle companion auto-addition and favourite hygiene.
- Updated `MessageHandler` to track new companions based on the `auto_manage_contacts` setting, with distinct behaviors for 'false', 'device', and 'bot' modes.
- Introduced scheduled jobs in `MessageScheduler` for device mode to manage firmware preferences and favourite keys with specified delays.
- Modified `RepeaterManager` to skip companion auto-purge in device mode, ensuring firmware manages contact slots effectively.
- Added tests to validate new behaviors and configurations, ensuring robust handling of contact management across different modes.
Added a new PendingMessageEntry TypedDict to improve the structure of pending messages. Updated type hints throughout the MessageHandler class for better clarity and consistency, including the initialization method and message processing functions. This refactor enhances code readability and maintainability without altering existing functionality. Additionally, included the new message_handler module in the pyproject.toml for better module organization.
Added asyncio timeout handling for sending startup and interval-based adverts, improving reliability by recording failures on timeout. Updated relevant tests to ensure coverage for timeout scenarios and increment failure counts appropriately. Introduced a new Radio Reliability panel in the web viewer for monitoring radio status.
Log loop-teardown RuntimeError at warning for scheduled sends, feed
queue wait, and repeater cleanup_database; keep interval advert on
broad Exception since send_advert failures use RuntimeError.
- Make MeshCoreBot.stop idempotent; remove duplicate stop from start().
- Avoid web viewer restart during shutdown; gate main-loop restarts.
- Fix packet capture MQTT callbacks to log each broker’s host.
- Only shutdown APScheduler when running; silence double-shutdown noise.
- Add regression tests in tests/test_shutdown_reliability.py.
Make outbound send suppression consistent by honoring both radio-offline and zombie states across command-manager and scheduler paths. Preserve strict SSRF defaults while adding explicit private-feed URL opt-ins, persist allow_local_smtp from notifications config writes, reconcile zombie alert setting precedence, and replace deprecated UTC timestamp calls with timezone-aware UTC usage.
Added new configuration options for radio health monitoring and alerting in config.ini.example, including radio_offline_threshold, radio_offline_alert_enabled, and radio_offline_alert_email. Introduced an [Admin] section with settings for a local admin API, including enabled, port, and token fields. Updated related files to reflect these changes, improving the organization and functionality of the configuration structure.
Moved radio-related configuration options from the [Bot] section to a new [Connection] section in config.ini.example for better organization. Updated the core logic to reference the new configuration paths, ensuring that radio health monitoring and alert settings are correctly handled. This change enhances clarity and maintainability of the configuration structure.
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
- base.html: persistent danger banner appears on every web page when
is_radio_zombie is true; shows datetime zombie was detected; includes
"Restart Bot Processing" button (POST /api/admin/zombie-recover) that
clears _radio_zombie_detected and _radio_fail_count on the live bot
object and removes the persisted DB flag; banner turns green on success
- config.html: new "Zombie Radio Alert" card with enable/disable toggle
and alert-email field; "Save" writes to bot_metadata (immediate,
survives restarts); "Save to config.ini" also persists values to
config.ini and keeps the in-memory config in sync; card shows
current config.ini values as baseline defaults
- app.py: inject_template_vars context processor now provides
radio_zombie and radio_zombie_since to all templates; added
GET/POST /api/config/zombie-alert endpoints (GET returns both
bot_metadata and config.ini values; POST supports write_to_config
flag); added POST /api/admin/zombie-recover endpoint; stored
config_path on self for write-back use
- scheduler.py: send_zombie_alert_email now prefers bot_metadata
(zombie.alert_enabled, zombie.alert_email) over config.ini so web
UI changes take effect without a restart; uses isinstance(..., str)
guard so mock/None values safely fall through to config.ini defaults
When the radio firmware is unresponsive but not yet flagged as zombie,
send_advert() would hang indefinitely on the main event loop, blocking
all packet processing (no data in/out) for 60+ seconds until the
scheduler thread's future.result() timed out — but that only timed
out the waiting thread, not the coroutine itself.
Fix: wrap every send_advert() call that runs on the event loop with
asyncio.wait_for(timeout=30.0). On timeout in the interval-advert
path, _radio_fail_count is incremented so repeated timeouts feed into
the existing zombie-detection threshold.
agessaman