Commit Graph
1157 Commits
Author SHA1 Message Date
agessaman 2dbd425dcb refactor(solarforecast): build _format_forecast from one timestamp parser and one day builder
The timestamp parse was written out six times, the distinct-production-hours count four times, and the per-day summary four times (today, tomorrow, Day+2, Day+3). They are now `_solar_timestamp_to_naive` (module level), a `production_hours` and `utilization` pair, and `day_part`, which covers all four days since the first-day %util and the peak can only fall on today or tomorrow. The two first-day hour counts keep their different rules (today: distinct hours at 0.1 W; tomorrow: data points at 1% of the panel), and the dead `now.strftime(...)` statement goes. 418 → 155 lines. The new goldens are unchanged, and old and new agree on 20,000 generated forecasts (timestamp formats, timezones, NaN and zero panels, None powers, threshold-boundary values) including every translate and logger call in order; two threshold mutations were caught.
2026-10-01 19:53:53 -07:00
agessaman 0d11717acd test(characterization): pin solarforecast's reply formatting over real forecast.solar responses
Three locations captured from api.forecast.solar on 2026-10-01, replayed with a frozen local clock from the evening before the first forecast day to late on the second, plus a 4-day variant (the free tier returns two days) for the Day+2/Day+3 line, five panel sizes, location labels, and nine distortions (Z and negative-offset timestamps, no watts, no days, a missing today, a None power value, an unparseable timestamp, all-zero watts). Exceptions that escape _format_forecast are recorded too. The goldens pin two existing quirks: timestamps with a negative UTC offset are never parsed, so they count no production hours, and a None power value raises out of the formatter.
2026-10-01 19:47:15 -07:00
agessaman 346ee05cb0 refactor(web_viewer): share the Socket.IO subscribe flag and history replay
Five subscribe handlers repeated the same lock-and-flag block, now `_mark_subscribed`; the command, packet and message handlers repeated the same packet_stream query and emit loop, now `_replay_packet_stream` with a type-to-event mapping. Each handler keeps its own debug line and its own 'Error replaying ... history' warning. The command and message replays now also select `type`, which they filter on anyway. Both helpers live in app.py, so `request`, `emit` and `sqlite3` still resolve through `modules.web_viewer.app`.
2026-10-01 19:44:57 -07:00
agessaman f4a981350e refactor(wx): find tomorrow's NOAA periods in _find_tomorrow_periods
The three-pass lookup (named Tomorrow, named for tomorrow's weekday, or the periods after today's) moves out of format_tomorrow_forecast, each pass returning early; the write-only current_day_periods counter goes. Old and new agree on 30,000 generated forecasts across every weekday and hour with a frozen clock, including the number of datetime.now() calls and the error logs; a mutation of the two-period cap was caught.
2026-10-01 19:41:42 -07:00
agessaman 8f7a707bba refactor: restore 21 imported names earlier slices stopped exposing, and guard them
A codex review found that moving dashboard code out of app.py dropped `modules.web_viewer.app.DashboardStatsService`. The public-API golden only recorded names a module defines, so I added a second one, `public_imports.json`, recording the public names each module imports from another project module, generated from dev's tree as the baseline. It found 20 more names earlier refactor commits had stopped importing (the Nominatim and MQTT-weather helpers in wx, gwx, aqi and solarforecast; `PayloadVersion`, the region-warning verdicts and `decode_path_len_byte` in message_handler; two feed helpers and two feed limits in the web viewer). All are imported again as re-exports, so `modules.<module>.<name>` and patches of it resolve as they did on dev.
2026-10-01 19:36:29 -07:00
agessaman bd235b9358 refactor(gwx): split get_open_meteo_weather into one fetch, current conditions and the daily tail
The tomorrow/multiday branch repeated the request, status check and JSON parse; all three types now share one. The current-conditions block becomes `_open_meteo_current` and the today/tomorrow/precipitation tail `_open_meteo_daily_tail`, both moved verbatim apart from dedenting, and both still raise into get_open_meteo_weather's handler. The unit symbol and `daily` are read before the current conditions instead of in the middle (both are plain lookups). The new Open-Meteo goldens are unchanged.
2026-10-01 19:20:47 -07:00
agessaman 6d780bd1f8 refactor(wx): keep format_hourly_forecast's original <= budget comparison
6267db74 flipped it to `> max_length: break`, which a codex review caught: a NaN max_length then admitted lines instead of none, and a None or string max_length logged the error with '>' instead of '<='. The original comparison is back.
2026-10-01 19:20:47 -07:00
agessaman cd01a22023 test(characterization): pin gwx's Open-Meteo formatting over real responses
Five locations captured from api.open-meteo.com on 2026-10-01 (three metric, two imperial), replayed with a frozen clock across day/night, length budgets up to 600, the default/tomorrow/multiday types and fifteen distortions of the current and daily fields, recording the reply text, the request parameters and the info/warning/error log calls. The goldens include a real failure: Open-Meteo's default model returns nulls for the last forecast days, so a 15- or 16-day gwx reply for London or Reykjavik is the multiday error.
2026-10-01 19:17:18 -07:00
agessaman f04ac9d93d refactor(web_viewer): move dashboard snapshots into DashboardSnapshotMixin
`modules/web_viewer/dashboard.py` holds building the dashboard stats service from config, its database connection, the 7-day multi-byte contact count it is given, and the background refresher: five methods moved verbatim, still on `BotDataViewer`.
2026-10-01 19:14:57 -07:00
agessaman 8dc442ad13 refactor(web_viewer): move the live streams into LiveStreamMixin
`modules/web_viewer/live_stream.py` holds the SocketIO broadcasts to subscribed clients (commands, packets, mesh edges and nodes, messages, log lines), the log tail thread and the database poll: nine methods moved verbatim, still on `BotDataViewer`. `_strip_ansi_codes` and `_ANSI_ESCAPE_RE` moved with them and are re-exported from app.
2026-10-01 19:12:00 -07:00
agessaman 9e35a678a5 refactor(wx): group multi-day periods through _multiday_day_name and one update rule
The two weekday scans in format_multiday_forecast become `_multiday_day_name`, and the three-branch day/night bookkeeping collapses to one condition: a period replaces the stored one unless it is a night period and the stored one is a day period. `is_day` is now `'night' not in name` ("tonight" already contains "night"). Old and new agree on 40,000 generated forecasts, including the error log calls; a mutation of the update rule was caught.
2026-10-01 19:07:35 -07:00
agessaman 6267db74b4 refactor(wx): split format_hourly_forecast into the future-period filter, hour label and line builder
The two copies of the startTime parse become `_parse_noaa_start_time` (its three branches parsed the same way), the 12-hour label is one expression, and each hour's line comes from `_hourly_line`. Exceptions still reach the method's own handler, so error replies are unchanged. Old and new agree on 30,000 generated period lists (offsets, Z times, unparseable and non-string times, missing fields, null precipitation) including the error log calls; two deliberate mutations were caught.
2026-10-01 19:03:37 -07:00
agessaman a634a91246 refactor(wx): read the extract_* helpers through one table-driven _first_match
Eight extract_* methods (humidity, precipitation chance and probability, UV index, dew point, visibility, wind gusts, pressure) each looped over their own regex list with an optional range check. Their patterns are now module-level tuples and one helper does the loop; the methods keep their names and signatures. Old and new agree on 152k texts (the NOAA fixtures plus generated keyword/number mixes, including out-of-range values and non-ASCII digits).
2026-10-01 19:00:02 -07:00
agessaman e71f91adda refactor(core): read the probe interval through _radio_probe_interval_seconds in the main loop
The main loop computed the same clamped [Connection]/[Bot] radio_probe_interval_seconds expression inline that the health probe already reads through the helper.
2026-10-01 18:55:54 -07:00
agessaman fe912299d5 refactor(core): move service supervision into ServiceSupervisorMixin
`modules/service_supervisor.py` holds starting services at boot, the backoff-gated restart of unhealthy services and `get_system_health`: six methods moved verbatim, still reachable on `MeshCoreBot`. Stopping services stays in `_stop_inner`, where it is one step of the shutdown order.
2026-10-01 18:53:21 -07:00
agessaman 3fc3bbf8fb refactor(core): move the radio link and health probe into RadioLinkMixin
`modules/radio_link.py` now holds connecting and reconnecting the transport, the host->radio command serializer and `radio_session`, zombie state and the periodic health probe: 20 methods moved verbatim, so every name is still on `MeshCoreBot`. `_serialize_command_frames` and `_radio_session_held` moved with them and are re-exported from `modules.core`; the serializer's `bot` annotation is now the mixin. `core.py` keeps `import meshcore` because tests patch `modules.core.meshcore.MeshCore`. core.py: 2,164 → 1,599 lines.
2026-10-01 18:50:20 -07:00
agessaman 5735483f20 Merge dev through #320 into refactor/maintainability
Brings in #309, #311, #314, #317, #319 and #320. Conflicts: command_manager's imports (kept both `inspect` and `math`) and wx's alert send, where I kept the S1 removal of the dead `weather_data[3]` statement and took #319's `skip_user_rate_limit=True`. The NOAA wx goldens are regenerated because #320 changes the wind text (`💨N0` → `⬆️N0`); nothing else in them moved.
2026-10-01 18:45:39 -07:00
Adam Gessaman 6dfd02711a Merge pull request #320 from agessaman/fix/wx-wind-direction
fix(wx): show NOAA's 16-point wind directions instead of truncating them
2026-10-01 18:40:58 -07:00
agessaman 09cf1f3e64 Merge remote-tracking branch 'origin/dev' into fix/wx-wind-direction 2026-10-01 18:40:49 -07:00
Adam Gessaman dc11733aa0 Merge pull request #319 from agessaman/fix/bot-tx-spacing
fix(send): stop dropping follow-up reply parts, and space concurrent transmissions
2026-10-01 18:40:48 -07:00
agessaman b7783ba443 Merge remote-tracking branch 'origin/dev' into fix/bot-tx-spacing
# Conflicts:
#	CHANGELOG.md
2026-10-01 18:40:39 -07:00
Adam Gessaman c430c06dca Merge pull request #317 from agessaman/fix/dm-ack-timeout-nan
fix(dm): treat a NaN dm_min_ack_timeout as unset
2026-10-01 18:40:38 -07:00
agessaman 9ffdd49b7f Merge remote-tracking branch 'origin/dev' into fix/dm-ack-timeout-nan 2026-10-01 18:40:29 -07:00
Adam Gessaman c42e515d0c Merge pull request #314 from agessaman/fix/atomic-migrations
fix(db): apply migrations atomically
2026-10-01 18:40:27 -07:00
agessaman 6971a9faf8 Merge remote-tracking branch 'origin/dev' into fix/atomic-migrations
# Conflicts:
#	CHANGELOG.md
2026-10-01 18:40:14 -07:00
Adam Gessaman 1c5e82ab79 Merge pull request #311 from agessaman/docs/region-scopes
docs: add a region scopes page
2026-10-01 18:40:05 -07:00
Adam Gessaman e8f3cbe401 Merge pull request #309 from agessaman/fix/region-warning-channel-scope
fix(region-warnings): send channel warnings at the bot's configured scope
2026-10-01 18:40:01 -07:00
agessaman 6460bded4a refactor(wx): only shorten Tomorrow's forecast text once the period passes its guard
Review of the get_noaa_weather breakup: computing the abbreviated text before the temperature/forecast check evaluated len(shortForecast) for periods the old code skipped, so a null shortForecast raised inside the catch-all and replaced the whole reply with an error. The abbreviation now runs inside the guard again. A test with null Tomorrow fields passes on the code before the breakup, fails on 5ff0f84 and passes now.
2026-10-01 16:07:05 -07:00
agessaman f4b7df2671 refactor(wx): turn _add_period_details into a table-driven loop
Six copies of 'if value: piece = ...; append if it fits' become one loop over (value, template) in the same order (humidity, dew point, visibility, precipitation, gusts, pressure). Observation values still win and the forecast-text extractors still run only when the observation lacks a value; the precipitation probability still comes from the text alone. Six discarded 'current_weather_length + ...' expressions go. Goldens unchanged; mutating the order, a template, the fallback or the length check fails them.
2026-10-01 16:03:11 -07:00
agessaman 74772207a3 test(characterization): pin _add_period_details over details text, observations and budgets
Real NOAA forecast text rarely carries humidity, gusts or pressure, and never a precipitation probability in a phrasing the extractor matches, so the length check and several formats went unexercised. A matrix of five synthetic forecast texts, three observation dicts (none, full, falsy values), five current lengths and two budgets pins the output; goldens generated on the code before the next commit.
2026-10-01 16:03:11 -07:00
agessaman 5ff0f847fc refactor(wx): pull the repeated forecast-period formatting out of get_noaa_weather
The Today, Tonight and Tomorrow blocks each rebuilt the same ' | Name: <emoji><forecast> <high/low or temp>' string and the same fit-checked wind suffix. They now call _noaa_period_str (None when the period lacks a temperature or forecast) and _noaa_period_wind with each block's own threshold: max_length - 10 for Today and Tonight, Tomorrow's fixed 115/120. Tomorrow's text shortening is _abbreviate_noaa_forecast, verbatim. Each block's control flow stays in place, including Tonight's details step sitting outside its temperature check. The NOAA goldens are unchanged, and mutating each helper fails them. get_noaa_weather drops from 394 to 281 lines.
2026-10-01 15:58:44 -07:00
agessaman 3587bc0bd3 test(characterization): reach wx's long-forecast abbreviation with multi-clause forecasts
The real NOAA samples never had a long short-forecast after a night period, so the Tomorrow block's abbreviation went unexercised. A variant rewrites later periods' shortForecast to multi-clause phrases ('A then B', 'X Becoming Y') and runs at 130, 160 and 200 characters; goldens generated on the code before the get_noaa_weather breakup.
2026-10-01 15:58:30 -07:00
agessaman 55e5f0eda3 fix(wx): show NOAA's 16-point wind directions instead of truncating them
abbreviate_wind_direction matched full words (NORTHWEST, NORTH, ...), but NOAA's forecast windDirection is always an abbreviation like WNW, SSE or N. Every value fell through to the fallback, which kept the first two characters behind a generic wind emoji, so replies read '💨WN2' or '💨SS1' and even plain N and NE lost their arrow. The 16 compass abbreviations now return the nearest 8-point arrow plus the abbreviation; full words and other values behave as before. Found while pinning wx's formatting on real NOAA responses for the maintainability refactor.
2026-10-01 15:53:04 -07:00
agessaman eb80c2b3b4 test(characterization): pin wx's NOAA forecast formatting on real responses
Replays api.weather.gov points, forecast, station and observation responses captured on 2026-10-01 for Seattle, Denver, Miami and Anchorage (geometry stripped), plus hourly forecasts for two of them, with wx_command's clock frozen at capture time. Each location runs get_noaa_weather with 0-3 leading periods dropped (other times of day), with and without a station observation, at 130 and 200 characters, and pins format_tomorrow_forecast, format_multiday_forecast for 3/5/7 days, _add_period_details and format_hourly_forecast. Coverage of those functions is 72-85%, the rest mostly error paths. This guards the planned breakup of get_noaa_weather and friends.
2026-10-01 15:48:56 -07:00
agessaman 4d16764f1e Merge remote-tracking branch 'origin/dev' into fix/dm-ack-timeout-nan
# Conflicts:
#	CHANGELOG.md
2026-10-01 15:46:26 -07:00
agessaman c00e9de3e3 Merge remote-tracking branch 'origin/dev' into fix/atomic-migrations
# Conflicts:
#	CHANGELOG.md
2026-10-01 15:46:25 -07:00
agessaman 90ef72355a Merge remote-tracking branch 'origin/dev' into fix/region-warning-channel-scope
# Conflicts:
#	CHANGELOG.md
2026-10-01 15:46:24 -07:00
agessaman 4e26175054 Merge remote-tracking branch 'origin/dev' into fix/bot-tx-spacing
# Conflicts:
#	CHANGELOG.md
2026-10-01 15:37:44 -07:00
agessaman 03ba510b53 refactor(web_viewer): move channel helpers into a ChannelAdminMixin
The seven channel helpers (listing, extra decode channels, number lookup, lowest free index, statistics, and the queued add/remove operations) move verbatim to modules/web_viewer/channels.py, apart from a function-local 'import sqlite3' the module import now covers. BotDataViewer inherits the mixin. app.py drops to about 5,570 lines.
2026-10-01 15:34:21 -07:00
agessaman e30cc8a498 refactor(web_viewer): move contact tracking and database admin into mixins
_get_tracking_data, _get_contact_detail and _calculate_distance move to modules/web_viewer/tracking.py (ContactTrackingMixin); the database statistics, table info and description, optimization and cache view move to modules/web_viewer/database_admin.py (DatabaseAdminMixin). All nine bodies are verbatim; BotDataViewer inherits both. tracking.py joins app.py on mypy's suppressed list, since the moved code carries the type errors mypy never checked in app.py. app.py keeps validate_sql_identifier and humanize_span imported so those module attributes still resolve. No test patched anything these methods use. app.py drops to about 5,800 lines, from 9,095 at the start of the refactor.
2026-10-01 15:30:48 -07:00
agessaman b49181342d fix(send): two more dropped follow-ups, and drop a pre-wait that now double-waits
Review found the same reply-limiter drop in two more places: the earthquake service's link post right after its alert, and the announcements command's 'sent' confirmation right after the announcement. Both now skip the reply limiter. multitest called wait_for_tx itself just before send_response, which waits again; with wait_for_tx claiming the slot that pre-wait made the reply wait a full extra interval, so it goes. A test that only documented RateLimiter's behavior, and guarded nothing, is replaced by an earthquake test that drives _check_earthquakes.
2026-10-01 15:26:21 -07:00
agessaman 134667ad22 refactor(services): one _cancel_tasks for the cancel-and-wait blocks in stop()
Eight stop() blocks did 'if task: task.cancel(); with suppress(CancelledError): await task'. BaseServicePlugin._cancel_tasks(*tasks) does the same, in order, skipping None, and like awaiting directly it re-raises an error a task had already finished with. Callers that reset their task attributes to None still do; the weather service still does not. DARC MoWaS (cancels without waiting) and packet capture (skips finished tasks) keep their own loops.
2026-10-01 15:22:22 -07:00
agessaman 87a4f45a85 refactor(services): one run_periodic loop for four poll loops
The earthquake poll and the weather service's alert, rain-nowcast and lightning loops were the same twelve lines: run the work, sleep the configured interval, stop on cancellation, and on any other error log it and wait 60 s. BaseServicePlugin.run_periodic now holds that loop; the four loop methods keep their names (tests look for them among spawned tasks) and their start logs, and pass the interval as a callable so it is still read every round. The error lines render the same text, now always through %s formatting. The World Cup loop (wakes early on fastcast pushes) and the bridges' queue processors have their own shapes and stay as they are.
2026-10-01 15:18:38 -07:00
agessaman b09340fdf3 Merge remote-tracking branch 'origin/dev' into refactor/maintainability 2026-10-01 15:11:53 -07:00
Adam Gessaman 9027c2aef5 Merge pull request #318 from agessaman/fix/service-restart-loop
fix(services): back off services that don't start, and unsubscribe on stop
2026-10-01 15:11:44 -07:00
agessaman f556a52cfa refactor(web_viewer): move mesh graph evidence into a MeshEvidenceMixin
The eight methods that derive multi-byte path edges and neighbor-confirmed edges for the mesh graph, and the NeighborEvidenceKeys tuple they return, move verbatim from app.py into modules/web_viewer/mesh_evidence.py; BotDataViewer inherits the mixin and app.py re-exports NeighborEvidenceKeys. The only textual change is an unquoted return annotation. app.py keeps timedelta and NamedTuple imported so those module attributes still resolve. The one test that froze modules.web_viewer.app.datetime to pin _compute_multibyte_evidence_edges' cutoff now freezes it where the method lives. app.py is now about 7,120 lines.
2026-10-01 15:11:01 -07:00
agessaman 45ba381b31 refactor(web_viewer): move the multi-byte rollout statistics into a MultibyteRolloutMixin
The nine methods behind the multi-byte rollout page (hop-chunk collection and caching, path encoding badges, single-byte relay metrics, legacy degree and the rollout payload) move verbatim from app.py into modules/web_viewer/multibyte_rollout.py; BotDataViewer inherits the mixin, so callers and tests are unchanged. The mixin declares the viewer attributes it reads for mypy. app.py's imports are untouched. app.py drops to about 7,540 lines.
2026-10-01 15:06:56 -07:00
agessaman 1d749d3847 fix(send): stop dropping follow-up reply parts, and space concurrent transmissions
A user who raised bot_tx_rate_limit_seconds to 5 noticed wx's weather-then-alerts reply did not seem to honor it. The gap between those two messages was in fact longer than the limit (every send already waits on the bot TX limiter in _check_rate_limits, and wx sleeps limit + 1 s on top), but looking at it turned up two real bugs.

The second part of wx's and gwx's two-part reply, and every part after the first of 'alert all', went through the reply rate limiter (rate_limit_seconds, 10 s by default) a few seconds after the first part had used it, so it was refused: with default settings the alert message was never sent. These now pass skip_user_rate_limit=True, as the other multi-part senders already did. The repeater command's two-part branch gets the same change for consistency.

BotTxRateLimiter.wait_for_tx only checked the time since the last recorded transmission, and record_tx runs after the radio confirms, so concurrent senders (two replies, a reply and a scheduled message) all passed the wait before any of them recorded and transmitted back to back. wait_for_tx now claims the slot under the lock when it returns. A coverage test that monkeypatched can_tx to drive the old loop is rewritten against the new one.
2026-10-01 15:01:29 -07:00
agessaman 5c5bd0969c refactor(web_viewer): move feed subscription helpers into a FeedSubscriptionsMixin
The twelve feed methods (subscription CRUD, activity, errors, statistics, preview and item formatting) and the two module helpers they use move from app.py into modules/web_viewer/feeds.py; BotDataViewer inherits the mixin, so every method stays on the viewer for routes and tests. Method bodies are unchanged apart from dropping function-local 'import sqlite3' lines the module import now covers, and _validate_feed_interval's annotation (object to Any, for mypy, which now checks this module). app.py keeps every name it imported for them so modules.web_viewer.app.<name> still resolves. The two tests that patched create_safe_requests_session and safe_requests_request to steer _preview_feed_items now patch them where the method lives; the SafeUrlPolicy.validate patches target the shared class and are unchanged. app.py drops to about 8,200 lines.
2026-10-01 14:42:57 -07:00
agessaman 4bdb1ef186 fix(services): keep restarted services logging, and stop a taken port from killing the bot
A second review of the restart fix found more of the same class. The map uploader's stop() removes its file log handler and start() never added it back, so after one restart it logged nothing to file; start() now restores it. DARC MoWaS set _running before binding its port, so a port in use left it dead but healthy, and werkzeug's make_server calls sys.exit when the port is taken, which got past every except Exception and stopped the bot; start() now binds first and turns that exit into an OSError.

Boot-time starts go through _start_service_at_boot, which records a restart failure when start() raises, so that service also waits out the backoff, and no longer reports a disabled service as a configuration problem. The health-loop step is _restart_unhealthy_services. New tests cover both, the bridges' session cleanup and the map uploader restart; all 13 lifecycle tests fail on dev.
2026-10-01 14:33:56 -07:00