Phase 8's mechanical part. The 19 asyncio.get_event_loop() calls that run directly in an async def (found by walking each file's AST; calls inside nested sync functions or lambdas don't count) become asyncio.get_running_loop(). Inside a running coroutine both return the same loop; get_running_loop says so and never creates one. The five calls in sync functions (maintenance.run_data_retention, radio_link._handle_serial_probe_error, scheduler._run_in_current_loop and run_scheduler, weather_service._run_daily_forecast) can run on threads without a loop, where the two differ, so they stay. No test patches get_event_loop.
S4. BaseCommand and the 40 bundled overrides that took no argument now take `message: MeshMessage | None = None`, matching joke and rain, which already use the message. None of the 40 bodies read it, so their help text is unchanged. help and command_manager keep their TypeError retry, since local plugins may still implement get_help_text(self). For bundled commands the retry no longer runs: `help <command>` calls get_help_text(message) once instead of failing and calling it again. A new test pins the signature for every bundled command class. The offline smoke run against the real config produces the same help text for all 46 commands.
S6. The sender's latest position (airplanes, path, test) and the repeater-or-room-server lookup by name (prefix, solarforecast) were each pasted into the commands, alongside location.py's own copies for get_companion_lat_lon and lookup_repeater_lat_lon. They now run through location.latest_contact_position_rows and location.repeater_by_name_rows. The copies differed only in how they drop a zero coordinate, so that is a zero_rule parameter: "both" (location.py's helpers: drop only 0,0) or "either" (the commands: drop a fix with either coordinate zero), and each caller passes the rule it had. Each command keeps its private wrapper, its pubkey access, try/except, log text and value conversion; airplanes' _get_bot_location now calls get_bot_lat_lon, which is the same code.
I did not move the commands onto resolve_location: its labels and geocoding rules differ from what each command does today. Old-vs-new over 20,000 randomized cases against a real SQLite table (zero coordinates, ties, near-duplicate names, missing pubkeys, database errors, no db_manager) matched on results and log calls for all eight functions; four mutations were each caught.
#323 ported this branch's weather slices to dev and #333 built the wx/gwx convergence on them, so dev's wx, gwx, weather_common and location are a superset of the branch's: I took dev's side there, with the characterization goldens #333 regenerated. The rest:
- core.py: kept the template file and carried dev's knots option into modules/templates/default_config.ini.
- utils.py: kept the shared _geocode_city_steps and applied dev's zero-coordinate fix (is not None instead of truthiness) to its seven checks.
- weather_service.py: kept run_periodic/_cancel_tasks and took dev's per-point NWS coverage cache.
- solarforecast: restored self.geolocator, which #323 kept for local plugins and the merge would have dropped.
A codex review of this port found solarforecast relying on the enabled_attr hook, which dev's BaseCommand does not have, so [Solarforecast_Command] enabled = false stopped working; its can_execute override is back as on dev. It also found names an earlier refactor commit had dropped as unused, which local plugins could still read: WxCommand.forecast_duration, num_wx_alerts, use_metric, zulu_time and geolocator, geolocator on gwx and solarforecast, the WXSIM_PARSER_AVAILABLE and WX_INTERNATIONAL_AVAILABLE flags, and the _mqtt_weather_error_key hook (now a WeatherCommandMixin method that _mqtt_weather_line calls through self, so an override takes effect again). tests/unit/test_weather_compat_surface.py pins all of them.
This brings the weather slices of my long-running refactor branch to dev ahead of converging wx and gwx, so the convergence PRs build on one copy of the code instead of rewriting it twice.
- `modules/weather_common.py` (WeatherCommandMixin): code wx and gwx shared by copy (MQTT and WXSIM source lookup, multi-day sending, high/low formatting, the forecast-option parser, the Open-Meteo model loader).
- `modules/nws_alerts.py`: one NWS alert entry parser for wx and the weather service, keeping each caller's own Special Weather Statement rules.
- `modules/http_retry.py`: one retrying session builder for the weather HTTP clients.
- wx: get_noaa_weather, the hourly, tomorrow and multi-day formatters, _add_period_details and the extract_* readers broken up or made table-driven; NOAA fetches go through `_noaa_fetch`.
- gwx: one fetch path; `_open_meteo_current` and `_open_meteo_daily_tail` split out of get_open_meteo_weather.
- solarforecast: `_format_forecast` built from one timestamp parser and one day builder (418 → 155 lines).
- aurora, rain, wx, gwx: companion and bot location lookups through `modules.location` (which gains optional logging arguments).
The changes outside these files that the refactor branch made (the base command's enabled_attr hook, the service base's run_periodic and _cancel_tasks) stay out: aurora and rain keep their can_execute overrides and the weather service its own loops, exactly as on dev.
Tests: characterization goldens over real captured NOAA, NWS, Open-Meteo and forecast.solar responses (tests/characterization, fixtures in tests/fixtures), 50 tests and several hundred cases, all of which also pass against dev's unmodified weather code. Each refactor step was additionally checked on the branch with an old-vs-new differential run (tens of thousands of generated inputs, including log calls and exceptions) and reviewed by codex.
5aee1d59 looked up the first forecast day's watt-hours only after counting production hours, so for a malformed response (watt_hours_day as a list, plus a None power) a different TypeError escaped than before. A codex review found it; the lookup is first again, and that input now raises the original error.
The timestamp parse was written out six times, the distinct-production-hours count four times, and the per-day summary four times (today, tomorrow, Day+2, Day+3). They are now `_solar_timestamp_to_naive` (module level), a `production_hours` and `utilization` pair, and `day_part`, which covers all four days since the first-day %util and the peak can only fall on today or tomorrow. The two first-day hour counts keep their different rules (today: distinct hours at 0.1 W; tomorrow: data points at 1% of the panel), and the dead `now.strftime(...)` statement goes. 418 → 155 lines. The new goldens are unchanged, and old and new agree on 20,000 generated forecasts (timestamp formats, timezones, NaN and zero panels, None powers, threshold-boundary values) including every translate and logger call in order; two threshold mutations were caught.
A codex review found that moving dashboard code out of app.py dropped `modules.web_viewer.app.DashboardStatsService`. The public-API golden only recorded names a module defines, so I added a second one, `public_imports.json`, recording the public names each module imports from another project module, generated from dev's tree as the baseline. It found 20 more names earlier refactor commits had stopped importing (the Nominatim and MQTT-weather helpers in wx, gwx, aqi and solarforecast; `PayloadVersion`, the region-warning verdicts and `decode_path_len_byte` in message_handler; two feed helpers and two feed limits in the web viewer). All are imported again as re-exports, so `modules.<module>.<name>` and patches of it resolve as they did on dev.
A codex review of the branch found seven places where earlier refactor commits changed behavior. Each is restored here with a test:
- base_service imported flood_scope relatively; the local-service loader also runs that file as local_services.base_service, where the import cannot resolve. Now absolute.
- Radio-health fields were initialized after commands and services were built, so a plugin constructor reading is_radio_offline raised. They are now set before any plugin.
- Removing the per-command can_execute overrides made 31 commands honor skip_channel_check, which their overrides had dropped. BaseCommand.honors_skip_channel_check (False on exactly those commands) keeps them enforcing their channel list.
- CommandManager.get_plugin_metadata, WxCommand.get_weather_for_zipcode and PluginLoader.validate_plugin were public compatibility entry points; they are back.
- feed_manager stopped exporting DEFAULT_MAX_FEED_RESPONSE_BYTES and DEFAULT_MAX_PARSED_FEED_ITEMS despite the commit saying it re-exported them; it does now.
- wx and gwx logged companion-location lookup failures at WARNING; the shared helper logged at debug. It now takes the level.
- The INFO 'MESSAGE ROUTING' / 'DIRECT MESSAGE' lines for pubkey-correlated DMs were removed with the dead path_info computation; they are logged again by a log-only helper.
32 commands overrode can_execute only to return False while their own <name>_enabled flag was off and otherwise defer to the base. BaseCommand now has an enabled_attr class attribute naming that flag, which can_execute checks first. Each command keeps reading its flag exactly as before (same section, default and legacy aliases), and code and tests that set help_enabled, webviewer_enabled and the like still work, since the flag is read at call time.
Only overrides that were exactly 'if not self.<flag>: return False; return super().can_execute(...)' were converted, checked by AST. Commands with extra conditions (wx's delegate, joke's dark-joke DM rule, admin-only and DM-only checks, dadjoke's reversed order) keep their overrides. Several of the removed overrides called super().can_execute(message) without passing skip_channel_check; nothing calls them with it set, so honoring it now changes nothing.
Found by scanning every __init__ assignment for reads across modules, tests and generate_website.py. Removed: wx's forecast_duration, num_wx_alerts, use_metric and zulu_time; the geolocator that wx, gwx, aqi and solarforecast each built and never used (all geocoding goes through the rate-limited helpers); and the web viewer's _db_connection/_db_lock/_db_last_used/_db_timeout pooling fields left over from before connections moved to DBManager. Two test fixtures patched get_nominatim_geocoder only to stub that unused construction.
Left alone although unread: BaseCommand._command_prefix (plugin-facing base class), MeshGraph._batch_task (holds a task reference), and config reads that feed known bugs (trace timeout_per_hop) or documented keys.
Ten leftovers from earlier unused-variable cleanups: dict lookups, .get() calls and one config read evaluated for nothing. None can raise where it sits: the subscripts read keys a SELECT * row and a fixed four-tuple always carry, and row[0] precedes reads of row[1] and row[2].
Correctness:
- {path_distance} was always blank in production. The resolution code that
builds repeater_info dropped latitude/longitude in every branch, so the
calculator could never find a coordinate. My tests passed hand-built
dicts straight to the calculator and never exercised the builder, which
is why they stayed green. Coordinates are now carried through all four
construction sites, and the new tests drive _lookup_repeater_names via
its lookup_func hook so the real builder runs.
- The #80 route guard was defeated two ways in the channel handler. When
the RF data was an uncorrelated fallback, control fell through to the
raw-hex and routing_info fallbacks below, which took the route from the
unrelated packet anyway; the guard had actually made that path
reachable. message.routing_info was also assigned unconditionally, and
the path command reads it. "Not attributable" is now a terminal branch
and the routing_info hand-off checks provenance.
- Same fix was incomplete for DMs: routing_info was captured and turned
into path_info before the provenance check ran, so the later check only
declined to overwrite an already-wrong value. Guarded at the source.
- Rendering could transmit for real. Capture only intercepts
send_response, but advert calls send_advert() directly and
send_response_chunked never checked capture_sink. Chunked sends are now
captured, and rendering is opt-in via BaseCommand.render_safe (default
False) instead of a denylist that cannot be complete. This also closes
the DM-only leak: schedule is not marked safe, so {cmd:schedule} can no
longer broadcast configuration to a channel.
- Multi-part rendered output was rejoined into one oversized send.
Scheduled messages are now split to the RF body budget and sent through
send_channel_messages_chunked, on character boundaries so multi-byte
text is not corrupted.
- _last_path_distance_km is instance state that was only set on success,
so an invalid path request could show the previous request's distance.
Reset at the start of every execute().
- Stale-contact retries were only counted on non-OK results, so timeouts
and exceptions left a contact eligible forever and could recreate the
storm. All failed attempts count now.
Web viewer:
- A failed config reload was reported as a successful save. The API and
UI now distinguish "saved and active" from "saved, restart needed".
- Preview count was unbounded; clamped to 1-20.
- update_ini_values is a read-modify-replace with no locking, so two
concurrent viewer saves could lose one. Serialised behind a lock.
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
BUG-001: web viewer login/session auth (in web viewer commit)
BUG-002: db_manager ALTER TABLE for missing channel_operations and
feed_message_queue columns on startup
BUG-015: scheduler thread blocked on future.result(); replaced all
blocking waits with add_done_callback (fire-and-forget)
BUG-016: reboot_radio sends meshcore.commands.reboot() before disconnect
BUG-017: radio disconnect uses asyncio.wait_for(timeout=10)
BUG-022: custom asyncio loop exception handler suppresses IndexError
from meshcore parser at DEBUG level
BUG-024: last_db_backup_run updated after each run; 2-min startup
window; last-run seeded from DB on restart
BUG-025: send_channel_message retries up to 2 times (2s delay) on
no_event_received via _is_no_event_received() helper
BUG-026: split_text_into_chunks() and get_max_message_length() added
to CommandManager; keyword dispatch uses send_response_chunked()
BUG-028: byte_data = b"" initialised before try block in
decode_meshcore_packet to prevent UnboundLocalError in except handler
TraceCommand: path nodes reversed and return path truncated; fixed
format_elapsed_display: UTC normalisation before elapsed computation (#75)
RepeaterManager: auto_manage_contacts guard before any purge logic (#50)
Command aliases: [Aliases] config section injects shorthands at startup
JSON logging: _JsonFormatter; json_logging = true in [Logging]
Structured JSON logging compatible with Loki, Elasticsearch, Splunk
Discord bridge, Telegram bridge, and all service plugins updated
MeshGraph edge promotion logic corrected
Shutdown: scheduler and meshcore disconnect joined cleanly; log spam fixed
All modules: ruff and mypy cleanup applied (type annotations, imports)
- Introduced a new utility function `get_config_timezone` to centralize timezone retrieval and validation from the bot's configuration.
- Updated various commands and the scheduler to utilize the new function, ensuring consistent timezone handling and fallback mechanisms.
- Removed direct dependencies on `pytz` in favor of a more flexible approach that supports both `pytz` and `zoneinfo` for timezone management.
- Enhanced logging for invalid timezone configurations to improve troubleshooting.
- Updated the `send_response` method calls in various command classes to include a `skip_user_rate_limit` parameter for message continuations, ensuring that the per-user rate limit applies only to the first message.
- This change improves user experience by allowing seamless message continuations without unnecessary rate limiting.
- Updated utility functions to improve country name validation, ensuring accurate identification of country names and US states.
- Enhanced geocoding logic in multiple command classes to handle default state and country configurations more effectively.
- Modified error messages to reflect the correct region based on user input, improving user experience.
- Added short usage descriptions for satellite pass commands in multiple languages, enhancing internationalization support.
- Updated the `generate_html` function to include detailed command usage information, including syntax, examples, and parameters for better user guidance.
- Added CSS styles for improved presentation of command usage and parameters in the generated website documentation.
- Enhanced command classes with structured documentation fields, allowing for consistent and informative command descriptions across the platform.
- Implemented a configuration option for enabling or disabling commands across multiple command classes.
- Each command now checks its enabled state before execution, improving control over command availability.
- Updated the configuration loading mechanism to retrieve the enabled state from the config file for commands like Advert, AQI, Catfact, and others.
- Removed per-command user cooldown tracking from individual command classes.
- Implemented a unified cooldown management system in BaseCommand for both global and per-user cooldowns.
- Updated command execution methods to call the new centralized cooldown handling, improving code clarity and reducing redundancy.
- Enhanced error handling and logging during cooldown checks to ensure robust command execution.
- Introduced caching mechanisms for internet connectivity status in CommandManager to optimize performance and reduce redundant checks.
- Updated command execution logic to skip commands requiring internet access when connectivity is unavailable, improving user experience and error handling.
- Added synchronous and asynchronous utility functions for checking internet connectivity in utils.py.
- Marked relevant commands (e.g., AlertCommand, AqiCommand, DadJokeCommand) as requiring internet access to ensure proper execution conditions.
agessaman