Three locations captured from api.forecast.solar on 2026-10-01, replayed with a frozen local clock from the evening before the first forecast day to late on the second, plus a 4-day variant (the free tier returns two days) for the Day+2/Day+3 line, five panel sizes, location labels, and nine distortions (Z and negative-offset timestamps, no watts, no days, a missing today, a None power value, an unparseable timestamp, all-zero watts). Exceptions that escape _format_forecast are recorded too. The goldens pin two existing quirks: timestamps with a negative UTC offset are never parsed, so they count no production hours, and a None power value raises out of the formatter.
A codex review found that moving dashboard code out of app.py dropped `modules.web_viewer.app.DashboardStatsService`. The public-API golden only recorded names a module defines, so I added a second one, `public_imports.json`, recording the public names each module imports from another project module, generated from dev's tree as the baseline. It found 20 more names earlier refactor commits had stopped importing (the Nominatim and MQTT-weather helpers in wx, gwx, aqi and solarforecast; `PayloadVersion`, the region-warning verdicts and `decode_path_len_byte` in message_handler; two feed helpers and two feed limits in the web viewer). All are imported again as re-exports, so `modules.<module>.<name>` and patches of it resolve as they did on dev.
Five locations captured from api.open-meteo.com on 2026-10-01 (three metric, two imperial), replayed with a frozen clock across day/night, length budgets up to 600, the default/tomorrow/multiday types and fifteen distortions of the current and daily fields, recording the reply text, the request parameters and the info/warning/error log calls. The goldens include a real failure: Open-Meteo's default model returns nulls for the last forecast days, so a 15- or 16-day gwx reply for London or Reykjavik is the multiday error.
Brings in #309, #311, #314, #317, #319 and #320. Conflicts: command_manager's imports (kept both `inspect` and `math`) and wx's alert send, where I kept the S1 removal of the dead `weather_data[3]` statement and took #319's `skip_user_rate_limit=True`. The NOAA wx goldens are regenerated because #320 changes the wind text (`💨N0` → `⬆️N0`); nothing else in them moved.
Review of the get_noaa_weather breakup: computing the abbreviated text before the temperature/forecast check evaluated len(shortForecast) for periods the old code skipped, so a null shortForecast raised inside the catch-all and replaced the whole reply with an error. The abbreviation now runs inside the guard again. A test with null Tomorrow fields passes on the code before the breakup, fails on 5ff0f84 and passes now.
Real NOAA forecast text rarely carries humidity, gusts or pressure, and never a precipitation probability in a phrasing the extractor matches, so the length check and several formats went unexercised. A matrix of five synthetic forecast texts, three observation dicts (none, full, falsy values), five current lengths and two budgets pins the output; goldens generated on the code before the next commit.
The real NOAA samples never had a long short-forecast after a night period, so the Tomorrow block's abbreviation went unexercised. A variant rewrites later periods' shortForecast to multi-clause phrases ('A then B', 'X Becoming Y') and runs at 130, 160 and 200 characters; goldens generated on the code before the get_noaa_weather breakup.
abbreviate_wind_direction matched full words (NORTHWEST, NORTH, ...), but NOAA's forecast windDirection is always an abbreviation like WNW, SSE or N. Every value fell through to the fallback, which kept the first two characters behind a generic wind emoji, so replies read '💨WN2' or '💨SS1' and even plain N and NE lost their arrow. The 16 compass abbreviations now return the nearest 8-point arrow plus the abbreviation; full words and other values behave as before. Found while pinning wx's formatting on real NOAA responses for the maintainability refactor.
Replays api.weather.gov points, forecast, station and observation responses captured on 2026-10-01 for Seattle, Denver, Miami and Anchorage (geometry stripped), plus hourly forecasts for two of them, with wx_command's clock frozen at capture time. Each location runs get_noaa_weather with 0-3 leading periods dropped (other times of day), with and without a station observation, at 130 and 200 characters, and pins format_tomorrow_forecast, format_multiday_forecast for 3/5/7 days, _add_period_details and format_hourly_forecast. Coverage of those functions is 72-85%, the rest mostly error paths. This guards the planned breakup of get_noaa_weather and friends.
Review found the same reply-limiter drop in two more places: the earthquake service's link post right after its alert, and the announcements command's 'sent' confirmation right after the announcement. Both now skip the reply limiter. multitest called wait_for_tx itself just before send_response, which waits again; with wait_for_tx claiming the slot that pre-wait made the reply wait a full extra interval, so it goes. A test that only documented RateLimiter's behavior, and guarded nothing, is replaced by an earthquake test that drives _check_earthquakes.
Eight stop() blocks did 'if task: task.cancel(); with suppress(CancelledError): await task'. BaseServicePlugin._cancel_tasks(*tasks) does the same, in order, skipping None, and like awaiting directly it re-raises an error a task had already finished with. Callers that reset their task attributes to None still do; the weather service still does not. DARC MoWaS (cancels without waiting) and packet capture (skips finished tasks) keep their own loops.
The earthquake poll and the weather service's alert, rain-nowcast and lightning loops were the same twelve lines: run the work, sleep the configured interval, stop on cancellation, and on any other error log it and wait 60 s. BaseServicePlugin.run_periodic now holds that loop; the four loop methods keep their names (tests look for them among spawned tasks) and their start logs, and pass the interval as a callable so it is still read every round. The error lines render the same text, now always through %s formatting. The World Cup loop (wakes early on fastcast pushes) and the bridges' queue processors have their own shapes and stay as they are.
The eight methods that derive multi-byte path edges and neighbor-confirmed edges for the mesh graph, and the NeighborEvidenceKeys tuple they return, move verbatim from app.py into modules/web_viewer/mesh_evidence.py; BotDataViewer inherits the mixin and app.py re-exports NeighborEvidenceKeys. The only textual change is an unquoted return annotation. app.py keeps timedelta and NamedTuple imported so those module attributes still resolve. The one test that froze modules.web_viewer.app.datetime to pin _compute_multibyte_evidence_edges' cutoff now freezes it where the method lives. app.py is now about 7,120 lines.
A user who raised bot_tx_rate_limit_seconds to 5 noticed wx's weather-then-alerts reply did not seem to honor it. The gap between those two messages was in fact longer than the limit (every send already waits on the bot TX limiter in _check_rate_limits, and wx sleeps limit + 1 s on top), but looking at it turned up two real bugs.
The second part of wx's and gwx's two-part reply, and every part after the first of 'alert all', went through the reply rate limiter (rate_limit_seconds, 10 s by default) a few seconds after the first part had used it, so it was refused: with default settings the alert message was never sent. These now pass skip_user_rate_limit=True, as the other multi-part senders already did. The repeater command's two-part branch gets the same change for consistency.
BotTxRateLimiter.wait_for_tx only checked the time since the last recorded transmission, and record_tx runs after the radio confirms, so concurrent senders (two replies, a reply and a scheduled message) all passed the wait before any of them recorded and transmitted back to back. wait_for_tx now claims the slot under the lock when it returns. A coverage test that monkeypatched can_tx to drive the old loop is rewritten against the new one.
The twelve feed methods (subscription CRUD, activity, errors, statistics, preview and item formatting) and the two module helpers they use move from app.py into modules/web_viewer/feeds.py; BotDataViewer inherits the mixin, so every method stays on the viewer for routes and tests. Method bodies are unchanged apart from dropping function-local 'import sqlite3' lines the module import now covers, and _validate_feed_interval's annotation (object to Any, for mypy, which now checks this module). app.py keeps every name it imported for them so modules.web_viewer.app.<name> still resolves. The two tests that patched create_safe_requests_session and safe_requests_request to steer _preview_feed_items now patch them where the method lives; the SafeUrlPolicy.validate patches target the shared class and are unchanged. app.py drops to about 8,200 lines.
A second review of the restart fix found more of the same class. The map uploader's stop() removes its file log handler and start() never added it back, so after one restart it logged nothing to file; start() now restores it. DARC MoWaS set _running before binding its port, so a port in use left it dead but healthy, and werkzeug's make_server calls sys.exit when the port is taken, which got past every except Exception and stopped the bot; start() now binds first and turns that exit into an OSError.
Boot-time starts go through _start_service_at_boot, which records a restart failure when start() raises, so that service also waits out the backoff, and no longer reports a disabled service as a configuration problem. The health-loop step is _restart_unhealthy_services. New tests cover both, the bridges' session cleanup and the map uploader restart; all 13 lifecycle tests fail on dev.
Path parsing, hop counting, per-path distances and node location by prefix (ten functions and the hex-token set, about 450 lines) move verbatim from utils.py to mesh_paths.py, and utils re-exports every one of them, so existing imports and module.utils lookups are unchanged. The two functions that need calculate_distance import it from utils at call time, which avoids a circular import and keeps patches of modules.utils.calculate_distance effective. The five tests that patched _get_node_location_from_db to steer calculate_path_distances now patch it where it lives, and a companion test goes through the old modules.utils entry point. utils.py drops to 1,798 lines.
core.py conflicts were S3's simplifications of is_radio_offline, _record_send_failure and _record_send_success colliding with #313's rewrite of the same methods; dev's versions are kept.
Review found that marking a not-running service inactive for good also retired services that only returned early for a transient reason: packet capture and the map uploader give up after waiting 30 s for the radio, and the bridges return when meshcore is momentarily unset, which is exactly when the health loop restarts them. Such a service now records a restart failure, so the next attempt waits out service_restart_backoff_seconds; a transient failure recovers, a missing configuration costs one attempt per backoff instead of one every 5 s. The restart decision is _service_restart_due, which the tests now exercise directly.
Also from the review: packet capture and the map uploader never cleared the should_exit flag stop() sets, so after a restart their background loops exited at once while the service reported healthy; start() now clears it. The Discord and Telegram bridges close the HTTP session they open when they return early because meshcore is not available.
configparser's getfloat accepts 'nan', and NaN fails every comparison, so it passed both the negative-value check and the min() cap and reached send_msg_with_retry as the ACK wait. It now falls back to the default like a negative value. 'inf' was already capped at 60 s; both cases are in the parametrized test, and the nan case fails on dev.
A fifth review found two gaps. A scheduled trial that returned False was re-armed at once, but False also follows exhausted no_event_received retries, so the radio could be tried again and again without another answered probe. And a reply's success settlement ran in a worker thread that a cancelled caller could cancel while it was still queued, stranding the trial in flight.
_end_offline_trial ends a trial that went out without succeeding (False or an exception, from a reply or a scheduled send) under the state lock alone; only paths where nothing reached the send still re-arm. The success settlement is shielded, so it completes even if the caller is cancelled. The cancellation test queues the settlement behind a busy one-worker executor and fails without the shield.
The health loop restarted every enabled service whose is_healthy() was False. A service that returns from start() without running because it has nothing to do (Discord or Telegram with no channels, the map uploader with no key) is never healthy, and its restart raised nothing, so it counted as a success and cleared the backoff: the service was restarted every 5 seconds, logging a warning each time.
Services whose start() returns without running are now recorded as inactive, logged once, and skipped by the health loop. A restart that leaves a service running but unhealthy records a failure, so the next attempt waits out service_restart_backoff_seconds.
Restarts also double-subscribed: services subscribed to meshcore events in start() and never unsubscribed (the comments said meshcore cleans up; it does not). BaseServicePlugin gains _subscribe and _unsubscribe_all; the Discord, Telegram, map uploader, packet capture and prefix-collision services subscribe through it, unsubscribe in stop(), and drop the old instance's subscription before re-subscribing after a transport reconnect. Tests use meshcore's real EventDispatcher and fail on dev.
Only scheduled messages and interval adverts could claim the trial armed by an answered health probe, so a bot with neither stayed offline however healthy the radio became. send_dm and send_channel_message now claim an armed trial when offline and run the send through _send_as_offline_trial, under the trial's identity; its result settles the trial (True clears the outage, False or an exception fails it until the next answered probe), so a dead radio still gets at most one attempt per probe. Settlement runs in a worker thread because it can write bot_metadata. Tests drive the real CommandManager guard, and the alert-thread counting in the recovery tests no longer stops asyncio.to_thread's workers.
geocode_city and geocode_city_sync were two 300-line copies, and geocode_zipcode and geocode_zipcode_sync two more, differing only in await. Each now has one body written as a generator that yields its Nominatim requests; _run_geocode_steps_sync and _run_geocode_steps_async perform them with the matching rate-limited calls and send back the result or throw back the exception, so the body's own error handling (bare excepts included) sees what a direct call would. The async variant keeps its own rate limiter path.
The city copies had drifted on one rule, kept behind async_rules: the async variant always reverse-geocodes an unqualified city's first hit to judge whether it is an obscure place in the default country, with a looser name match, and reuses that lookup for address info; the sync variant only checks when address info is requested. tests/unit/test_geocode_city_variants.py pins both. A side-by-side run of the old and new module over 3,840 city cases and 960 zip cases (scripted results, failures and caches, both variants, with and without address info) matched on return values, Nominatim call sequence and cache calls; flipping any one rule branch produced differences.
Seven places in message_handler and graph_trace_helper ran the same pair of queries, byte for byte: count the repeaters and room servers whose key starts with the prefix and were heard within the recency window, and if exactly one was, fetch its key (starred first, then most recent). contacts_repo.unique_recent_repeater_key now runs it and returns (count, key); each caller keeps its own logging, including the two that report the match count. Same queries, same order, same handling of execute_query's empty result on a database error.
A third review found four more gaps. A trial send that hit the 60 s wait kept running and could transmit later, during a newer trial; _record_send_success cleared any outage without checking whose success it was, so an old send finishing late could clear a newer one; the wrappers claimed the trial before logging and time lookup, outside the code that settles it; and the startup advert's timeout path recorded its failure on the event loop, where tripping the outage writes bot_metadata.
_record_send_success now takes the trial id and clears only when that trial is still in flight, deciding and resetting under the state lock (_reset_offline_state_locked); success of an ordinary send while offline leaves the outage alone. A trial whose wait fails is cancelled. Each wrapper settles the trial if anything after admission raises. The startup advert records its failure through asyncio.to_thread.
A second review found five gaps. A loop that closed between is_running() and submission stranded the trial in flight; fallback sends (no running main loop) settled every trial as inconclusive; a timed-out send that kept running passed the guard during a later trial, because the context flag was a bare True; the viewer clear checked the outage generation, released the lock, then cleared, so it could clear a newer outage; and the health loop ran the metadata sync, with its database retries, on the event loop.
Trials now carry an id: _admit_measured_send returns it, _as_offline_trial sets it in the context, and is_radio_offline and the settle methods only honor the trial currently in flight. The scheduler's two wrappers share _run_measured_send, which submits inside its error handling (closing the coroutine if submission fails), settles a fallback trial by its real outcome, and settles any trial left open in a finally. _clear_radio_offline_state takes an expected generation and checks it under the state lock. The health loop checks the 30 s throttle on the loop and runs the sync itself in a worker thread. The delayed-write test now waits for the newer outage to trip before releasing the stalled writer, so it fails without publish serialization.
Codex caught three deleted public methods earlier and a later scan found twelve more, because nothing tested for them. The golden records every public module function, class and method; the test fails only on a removal, and accepts a name that still resolves through a base class or re-export.
#315 changed populate_missing_geolocation_data, which S1 had deleted as dead. It is public, and local plugins may call public manager and command methods, so it comes back in dev's form, along with 11 other public methods S1 removed: CommandManager.get_available_commands_list, GreeterCommand.get_recent_greeted_users, WxCommand.abbreviate_alert_title and get_current_conditions, MessageHandler.discover_message_path, and RepeaterManager.add_discovered_contact, get_daily_advertisement_stats, get_nodes_per_day_stats, get_purging_stats, restore_repeater, test_meshcore_cli_commands and toggle_auto_add. All are verbatim from dev; every self. attribute they use still exists. A scan of every public name in modules/ on dev against this branch now finds nothing missing.
Review of the circuit breaker found four problems. The trial was a shared flag that let every sender through, interactive sends never settled it, and it could stay open forever. The scheduler threw away send_channel_message's result, so a send the radio rejected counted as a success and cleared the outage. Metadata writes happened outside the state lock, so a stalled clear could overwrite a newer outage's 'true' and the viewer check would then cancel it. And one failed read-back of the persisted 'true' disabled the viewer clear for the rest of the outage.
Now a probe arms one trial, which the scheduler's next measured send (scheduled message or interval advert) claims atomically through _admit_measured_send; that send runs under a context flag so only it passes the send-path offline guards. Its real outcome settles it: success clears the outage, a timeout ends the trial until the next answered probe, and a False result or nothing sent re-arms it. _send_scheduled_message_async now returns whether the message went out. Metadata is published under its own lock from the current state, confirmed per outage generation, and retried from the health loop until confirmed. The in-process viewer clear calls the full clear. Tests go through the real scheduler on a live loop, race admission across threads, stall a clear write while a new outage trips, and count alerts from the first trip.
The threshold test timed one request, so a single GC pause or a busy CI runner could fail it. On the Python 3.10 runner it failed four times in a row at 0.22 to 0.26 s on a branch that does not touch the stats path, while the same request takes about 22 ms locally on both that branch and dev, with or without coverage. It now times five requests and asserts on the fastest, which an algorithmic regression still slows, and the failure message lists every timing.
Two synthetic entries carry CAP fields inside a nested cap:info element, which only wx's DOM localName pass reads. Goldens regenerated on the pre-refactor parsers.
Both parsers run over a trimmed sample of real NWS active-alert feeds (TX, CA, AK, FL on 2026-10-01) and synthetic entries that reach every Special Weather Statement headline and summary branch, the title-only fallbacks, missing titles and each link shape. wx is pinned through get_weather_alerts_noaa (full list, compact text and one feed per entry, plus its warning/error logs); the service through _parse_alert_entry. Key order is recorded because the golden JSON sorts keys.
agessaman