48 Commits
Author SHA1 Message Date
agessaman 332a61eb35 feat(web_viewer): fetch a message row's packet by hash when it's opened on Real-time
The replay on load is the newest 50 messages and the newest 50 packets, and packets reach back far less, so older message and command rows had a hash and no packet. Opening such a row now asks GET /api/packets/by-hash/<hash>?near=<epoch> for that packet's stored receptions and fills in route, path, receptions and Inspect bytes. The near time bounds the scan to the timestamp index, since the hash is only inside the row's JSON (about 6 ms against a full scan's 250 ms on 90k rows).

The fetched receptions fill the row only: they stay out of the spark and the replay's coverage, since they aren't new hearings. The detail says Loading while it waits, that the packet isn't in the bot's log when nothing comes back, or that it couldn't load and reopening retries.
2026-10-24 16:11:16 -07:00
agessaman 8d1c5ea9c0 build(web_viewer): vendor the last CDN libraries and keep them updated with Dependabot
Bootstrap 5.1.3, Font Awesome 6.0.0, Socket.IO 4.7.2 and Chart.js 4.4.0 are now served from static/vendor/ at the versions the CDNs served, so every page works on a LAN with no internet. Only map tiles still need it. The Content-Security-Policy drops jsDelivr and cdnjs.

Every vendored package, the map stack and login fonts included, is pinned in scripts/web-vendor/package.json and its lockfile. sync.mjs copies the listed files into static/, strips the source-map comment, empties each directory it owns first, and regenerates the version table in static/vendor/README.md; it reproduces the previously vendored files byte for byte. Font Awesome ships only its woff2 fonts, which every supported browser takes before the .ttf.

Dependabot watches that directory: security fixes as advisories land, grouped minor and patch releases monthly, majors one per PR. Dependabot can't regenerate the vendored files, so vendor-refresh.yml adds them to its PRs as a second commit and dispatches the tests, since a GITHUB_TOKEN push doesn't trigger them. It runs on pull_request_target, so it uses the base branch's sync.mjs, spells out its commands instead of running the PR's npm scripts, and installs with install scripts disabled. A new CI job re-runs the sync and fails if anything under static/ changes.

Tests now check that no template loads a script, stylesheet or font from another host, that every vendored file a template names exists, and that the CSP allows no CDN host. In Chrome, 12 pages load with no external request, failed load or console error.
2026-10-24 12:02:40 -07:00
agessaman 2a1c2f86d8 feat(web_viewer): rebuild Real-time as one tape of what the radio heard
Real-time was three equal panels (commands, packets, messages) that each scrolled on their own, couldn't be held still, and showed the same transmission up to three times. It's now one tape with one row per transmission: a packet, the channel message decrypted from it and a command it triggered share a row, joined by packet hash, and the bot's reply hangs under the command with how many times a repeater was heard echoing it. Those echoes fold into the reply instead of getting rows of their own, and a packet heard several times is one row with a count and every reception (SNR, RSSI, hops, path) when opened.

A receiver strip at the top says whether the bot is listening or the radio is offline or not responding (from the banner poll base.html already runs, now published as a banner-status event), when it last heard a packet, the packet rate and a 10-minute activity bar. The tape holds still while you read it: Pause (P), scrolling down, a moving mouse pointer or keyboard focus queue new rows behind an "N new" button. Rows can be kept (safe from the 300-row limit and from Clear), filtered to messages or commands, by packet type or channel, and searched (/). Replayed history sits under a "Heard before you opened this page" divider, and adverts and keyed group text are decoded in the browser and labeled as such. Each row's glyph chip carries its packet type's icon in the dashboard chart's color for that type; decoded messages get a stronger tint, and the bot's commands and replies are violet.

Server side, command rows carry trigger_packet_hash (from the message's routing info) and, once the transmission tracker matches a repeat, reply_packet_hash. Replayed rows carry stream_ts and replayed, and live rows stream_ts. Commands stored before this pair with their message only on an exact sender, channel and text match within 20 seconds.

Real-time is admin-only now: its socket already was, so anonymous visitors got three "Admin login required" badges over spinners that never filled. They're redirected to login and the nav link is hidden from them.

The byte breakdown sliced multibyte paths by hop count, so 2- and 3-byte hops spilled into the payload; it uses the path's byte count and shows the raw path-length byte. Route types 0 and 2 were swapped, and the channel key store was often built before the decoder had loaded, so it never decrypted anything. js-sha256 is vendored under static/vendor instead of loading from cdnjs.
2026-10-17 19:40:23 -07:00
agessaman 4d9f758eff perf(web_viewer): load page libraries where used and vendor the map stack
Every page loaded Chart.js (~200KB), Moment and js-sha256 from a CDN whether it used them or not. Chart.js now loads only on the dashboard, Contacts and Multibyte, and js-sha256 only on Realtime, through a page_libs block in base.html. Moment is gone: its one call wrote to a #last-update element that no page renders, and that formatting is now done natively. The remaining base scripts stay at the end of <body>, so they don't block first paint.

The Mesh page loaded Leaflet 1.9.4, MapLibre GL 5.6.0, maplibre-gl-leaflet 0.1.4 and vis-network 9.1.9 from unpkg, and the login page loaded DM Sans and Outfit from Google Fonts, so neither worked on a LAN without internet. They are now vendored under static/vendor/ and static/fonts/ with their licenses and a README recording versions and sources (source-map comments stripped, since the maps aren't vendored). With nothing left on unpkg or Google Fonts, the CSP drops those hosts; the characterization golden is regenerated for that. package-data gains a fourth directory level for Leaflet's images.

Bootstrap, Font Awesome, Socket.IO, Chart.js and js-sha256 still come from CDNs; PRODUCT.md records that as the remaining offline gap.
2026-10-17 18:10:02 -07:00
agessaman f5f3b6e52a fix(web_viewer): review fixes for plugin settings coverage and template previews
Fixes from a Codex review of the previous commit, plus the shared settings the rebase onto dev showed were still missing.

Settings that misstated or changed what runs:
- sports never read [Sports_Command] api_timeout (documented in the example config); the new field did nothing. It now sets the ESPN timeout.
- The path-byte requirement help on test, multitest and path said "bytes per hop"; the gate checks the path's total size (at least 2, or exactly 3).
- feed's allow_private_urls showed false while it inherited true from [Feed_Manager]. Fields can now name an inherit_section whose value they show when unset.
- notify_on_prefix_bytes was an int field for a value the service reads as a list (2,3), so the form refused a valid setting.
- The greeter's form defaults differed from the fallbacks it uses when the keys are missing. Both now come from shared constants.
- The MQTT weather limits accepted values the service clamps (payload 256 to 1 MiB, staleness 5 s to 7 days, length 64 to 4000).

Shared settings: the wx card now carries the [Weather] temperature, wind and precipitation units and the three temperature_*_format templates (with previews through format_temperature_high_low), next to wx's own [Wx_Command] unit overrides, which default to "Same as [Weather]". Two fields with one key in different sections are told apart as key@Section in the save payload; a bare key still resolves when only one field has it. An enum's "" option, like a blank number, removes the key. Packet capture brokers gain username, password, topic prefix, JWT timings and include_decoded; blank inheriting values are left out of block saves, since the service reads them with getint/getboolean when present.

Previews:
- They use the bot's saved config: a blank test format falls back to [Keywords] test, multitest lays out paths per condense_paths, and MQTT applies passthrough_max_length.
- A quoted empty template ("") counts as blank, as the commands strip the quotes first.
- A str.format width or precision over 500 isn't rendered, so `{sender:100000000}` can't make the endpoint build a huge string.
- The page invalidates an in-flight preview as soon as the field changes, and shows the server's note about fallbacks instead of guessing it.

The endpoint inventory golden gains the preview route (401 for anonymous requests when a password is set).
2026-10-04 15:55:17 -07:00
agessaman 132f157f0c test: pin packet decoding over the committed MQTT packet fixture
S0's missing replay guard. The packet_decode move was checked at the time by replaying a private 4,009-packet capture that cannot go in the repository. This golden decodes the ten public packets in tests/fixtures/mqtt_packets.json, each whole, halved, header-only and with an odd-length hex string, at 2-, 4- and 6-character path prefixes, plus calculate_packet_hash. dev's MessageHandler.decode_meshcore_packet (c44e8f47) produces the same output byte for byte.
2026-10-03 22:52:53 -07:00
agessaman 36ab4f983c test: a missing golden file fails instead of being written
assert_golden used to write a golden that didn't exist yet whenever CI was unset, so a deleted or misnamed golden re-recorded whatever the code did and the test passed locally. It now writes only with UPDATE_GOLDEN=1 and otherwise fails with the same message CI already gave. Space bunny flagged this earlier; the public-API guards were hardened the same way in b62806f5.
2026-10-03 22:46:30 -07:00
agessaman ba8e3f0229 fix(refactor): restore 24 module-level names the moves stopped binding, and guard all of them
Codex's final review found that moving code out of modules dropped imports that local plugins could import from them, for example `from modules.service_plugins.discord_bridge_service import EventType`, which would make such a plugin fail to load. Comparing every top-level name bound on dev against the branch (both trees imported, so names under TYPE_CHECKING are excluded) found 24 in 11 modules: EventType, copy and contextlib in both bridges; sha256, Iterable and hmac_mod in message_handler; RotatingFileHandler, colorlog, contextlib, contextvars and functools in core; hashlib and re in utils; contextlib in feed_manager, earthquake_service and weather_service; Any in cmd and help; time in path. Each is re-imported exactly as dev imported it.

The public-import guard only covered `from modules... import` statements, so it missed these. It now records every unconditional top-level import and assignment, regenerated from dev's tree (461 → 1,597 names), and fails if the discord bridge stops exposing EventType.
2026-10-03 22:13:19 -07:00
agessaman 145e4d1e29 test(web_viewer): treat the platform's two .ico MIME types as one in the endpoint inventory
The golden was recorded on macOS, whose MIME table maps .ico to image/x-icon; the Linux CI runners map it to image/vnd.microsoft.icon through /etc/mime.types, so /favicon.ico failed the inventory on every Python version. The app's behavior is unchanged; the test now normalizes that one alias.
2026-10-03 21:43:37 -07:00
agessaman c905024102 test(weather): let the dispatch golden's wx WXSIM fake take location_name
wx passes a named WXSIM source as location_name=, which the fake did not accept, so every named-source wx case recorded the error reply instead of a successful send or multiday split. The regenerated golden passes against both the code before e76383f2 and after it.
2026-10-03 17:48:30 -07:00
agessaman bbeff40573 test(weather): characterize how wx and gwx execute dispatch a request
Records the ordered calls each request makes, with every provider mocked: custom MQTT and WXSIM sources (default and named), the no-location fallback chain, provider failures, the two-part alert reply, and wx's alert-only geocoding. Pins execute before its shared steps move into WeatherCommandMixin.
2026-10-03 17:20:02 -07:00
agessaman 2f788abc9b Merge origin/dev (#342) into refactor/maintainability
#342 removed the pending-message correlation list. message_handler keeps this branch's mixin layout without PendingMessageEntry; the default config comment for message_correlation_timeout moves into modules/templates/default_config.ini (core.py keeps reading the template, which matches dev's inline text otherwise); both sides' new tests at the end of test_message_handler.py stay.

The public-API golden drops exactly the five names #342 removed on purpose.
2026-10-03 12:28:45 -07:00
agessaman a10baf6de2 Merge origin/dev (#322, #323, #333) into refactor/maintainability
#323 ported this branch's weather slices to dev and #333 built the wx/gwx convergence on them, so dev's wx, gwx, weather_common and location are a superset of the branch's: I took dev's side there, with the characterization goldens #333 regenerated. The rest:

- core.py: kept the template file and carried dev's knots option into modules/templates/default_config.ini.
- utils.py: kept the shared _geocode_city_steps and applied dev's zero-coordinate fix (is not None instead of truthiness) to its seven checks.
- weather_service.py: kept run_periodic/_cancel_tasks and took dev's per-point NWS coverage cache.
- solarforecast: restored self.geolocator, which #323 kept for local plugins and the merge would have dropped.
2026-10-02 16:26:53 -07:00
agessaman 17a93104cc fix(gwx): label current conditions by is_day, moon at night, name tomorrow's weather
At 5:58 am wx 98258 (Open-Meteo provider) replied "Tonight: ☀️Clear … | H:68°F L:50°F | Tomorrow: 🌫️ H:67°F L:51°F". Any hour before 06:00 was labeled "Tonight", which next to today's high/low and a "Tomorrow" for the following day read like last night's forecast, and code 0 always showed a sun.

I now request Open-Meteo's is_day and use it for both the label and the emoji, falling back to 06:00–18:00 local when it's absent. Night before noon is "Overnight" (the NWS name for that period, new key in every catalog), night after noon stays "Tonight". Clear and mostly clear at night show 🌙. Tomorrow's segment adds its description when it fits and keeps the emoji-only form otherwise.

Goldens change for the new request field, Overnight on the fixtures captured after midnight, 🌙 on clear night fixtures, and tomorrow's description.
2026-10-02 07:04:52 -07:00
agessaman 1536f2a634 fix(weather): codex review of the whole branch
- A scheduled {cmd:gwx ...} or {cmd:wx ...} with a forecast and an alert slept out the TX pacing between them during capture; with bot_tx_rate_limit_seconds = 30 that passed the 30 s render timeout and the whole reply was lost. Pacing is skipped when the message has a capture_sink.
- gwx tomorrow ignored its byte budget (a Russian Reykjavik forecast was 99 bytes against 89). It now drops the precipitation amount, gusts, wind, the precipitation chance and the description, in that order. The Open-Meteo goldens give tomorrow runs a roomy budget, since their key-name translator makes the text much longer than real; trimming is tested with real catalogs in test_gwx_tomorrow_budget.
- Custom default sources (bare wx/gwx) still returned True after a refused send.
- An option without a location (wx hourly, gwx tomorrow, wx 5d) showed usage, since the fallback location was resolved before the option was parsed; the option now applies to the fallback, custom default sources included. The docs say how far each provider's forecast reaches.
- wx read its unit settings through get_config_value, whose [Weather] fallback logged a notice to move them into [Wx_Command] on every lookup (14 per reply); _unit_setting reads [Wx_Command], then [Weather], without it.
- NOAA forecasts that mention thunderstorms after sun ("Sunny then Slight Chance Showers And Thunderstorms") still got ☀️; thunderstorms now outrank sun.
- The catalog tests passed with keys removed, since a missing key differs from English; they now require each key and every wind point, with en-GB and Polish allowed to use the English letters.
2026-10-02 01:10:39 -07:00
agessaman 0853739f30 fix(weather): review follow-ups: warning units, wx localization, help text
A space bunny review of this branch's commits found:
- gwx's high-wind warning converted the speed to mph for every catalog, so on a km/h bot it read 49 next to a reply saying 80, and Polish (whose template said km/h) regressed in the merge. The threshold stays 30 mph; the warning shows the speed in the configured unit with a {unit} placeholder in every catalog.
- tomorrow_not_available, tomorrow_error, multiday_not_available and multiday_error were still English in seven catalogs; they are translated, and the catalog test now fails on any weather reply left in English.
- wx kept NOAA's English wind letters and English weekday letters while gwx's are translated, so one German bot showed NE and NO for the same wind. wx now translates both through the catalog (commands.wx.day_abbrev added); English output is byte-for-byte unchanged, which the goldens show (their stub translator gives English for these short labels, via tests.helpers.key_name_translate).
- Polish feels_like was ungrammatical (odczuwalna -> odczuwalnie), Russian's used a nonstandard abbreviation, and Russian gwx's MQTT subscriber error lacked wx's hint.
- The help text (commands.wx.description/usage, commands.gwx.usage) described wx as ZIP-only; it now lists cities, coordinates and the options, under 130 bytes in every language.
- The generated default config, the docs' default_state wording, _fit_tomorrow_parts' docstring and the CHANGELOG's stats claim were inaccurate.
2026-10-02 00:48:31 -07:00
agessaman 3d86b70899 fix(wx): fit NOAA's tomorrow forecast into its byte budget
format_tomorrow_forecast took max_length and never used it, so two long periods (Miami's "Chance Showers And Thunderstorms" day and night) came to 132 bytes against a 100-118 byte budget and the send path split the reply in two. It now tries the full reply, then without the night period's wind, without any wind, the first period alone, and that period without wind. Measured over the captured NOAA forecasts with the real English catalog at budgets 100-235, nothing overruns now (15 of 384 did); gwx and wx's default reply already fit. The regenerated goldens differ only in tomorrow keys, whose key-name translations are longer than real text.
2026-10-01 23:17:47 -07:00
agessaman da6e699008 fix(wx): read NOAA's precipitation, gust and below-zero wording
NOAA writes "Chance of precipitation is 60%." and "with gusts as high as 25 mph", and none of the precipitation or gust patterns allowed either, so the 🌦️ and 💨 details never appeared for real forecasts (the captured Miami, Anchorage and Denver forecasts all carry them). High/low extraction took only unsigned numbers and rejected Fahrenheit values under 20, so "Low around -5." or a high of 14 lost the high/low. Both now cover NOAA's range, -65 to 130 °F or -55 to 55 °C. The regenerated goldens differ only by the newly read details, which at tight budgets now take the place of pressure.
2026-10-01 23:08:02 -07:00
agessaman 4895da3f77 Merge remote-tracking branch 'origin/fix/weather-wind-emoji' into feat/wx-gwx-convergence
Goldens conflicted; I regenerated them, and against this branch they differ only in NOAA emojis, gwx wind labels and the Phoenix width budgets the merged branch corrected. gwx's new hourly test expected the old truncating conversion (202.5° as SW); with nearest-point directions it is SSW.
2026-10-01 22:50:34 -07:00
agessaman e9122b2dfd fix(weather): let rain and snow outrank cloud in NOAA emojis; skip non-finite wind degrees
A review of this branch found three things. Moving the partly/mostly cloudy check ahead of rain gave forecasts like "Chance Rain Showers then Mostly Cloudy" (ten of the 79 forecasts it sampled from live NWS points) a sun-behind-cloud emoji; rain and snow now come before every cloud check, so they get 🌦️ or ❄️. A NaN or infinite wind direction made gwx's new nearest-point conversion raise; it is now left out like a null one. The Phoenix daily-width test's budgets were written for a reply one byte shorter (SW is now WSW), so they no longer pinned <= at equality; they are 330/343 now and the regenerated golden shows each segment fitting exactly at its budget.
2026-10-01 22:46:47 -07:00
agessaman 37aabe5641 Merge remote-tracking branch 'origin/feat/gwx-hourly' into feat/wx-gwx-convergence
Conflicts: NOAA hourly wind keeps this branch's full 16-point direction and fix/noaa-configured-units' speed conversion (a test pins both); gwx's hourly forecast returns (reply, {}) like the other forecast types since fix/gwx-warnings-structured, and _open_meteo_current keeps fix/gwx-missing-current's Optional return.
2026-10-01 22:42:38 -07:00
agessaman f4aaa7556f Merge remote-tracking branch 'origin/fix/gwx-missing-current' into feat/wx-gwx-convergence
The distorted-London golden conflicted; I regenerated it, and it differs from this branch only in the no_current and null_temp keys the merged change means to move.
2026-10-01 22:29:46 -07:00
agessaman 0e8ba9bb61 fix(weather): nearest-point wind directions in gwx, and NOAA emoji precedence
gwx's _degrees_to_direction picked the 8-point sector starting below the angle (22.5° → NE, 241° → SW, negative angles → N); it now picks the nearest of the 16 points the catalog already translates, with the arrow wx uses, via _COMPASS_16/_ARROWS_8 moved to weather_common so both commands share them. wx's get_weather_emoji tested 'cloudy' before 'partly cloudy'/'mostly cloudy' and rain before thunderstorms, so those got ☁️ and 🌦️; thunderstorms (and 't-storm') now come right after heavy rain, and the partly/mostly cloudy check before plain cloudy. Golden moves: NOAA emojis for thunderstorm and partly/mostly cloudy periods (Seattle hourly fits one more hour, ⛅ being shorter than ☁️), and gwx wind labels where the nearest point differs.
2026-10-01 22:08:37 -07:00
agessaman 151cf0b78a fix(wx): take today from the location's clock, and date undated periods from their neighbors
A codex review found two holes in using the first NOAA period's date as today: at 1 AM the first period can still be the night that began the evening before, so tomorrow and multi-day started a day early; and one period without startTime switched the forecast to date grouping and then dropped that period (multi-day showed Friday night's low as Friday). Today is now the current instant in the UTC offset NOAA writes on its periods; an undated period takes its neighbor's date (same date for a night after a day, else the next). With the characterization clock at Thursday afternoon the drop-period variants start on Friday again, as before this PR; against the base branch only Miami's seven-day forecast (a seventh date) and its already-started hourly entry differ for NOAA.
2026-10-01 21:59:50 -07:00
agessaman 65eb3492fa feat(gwx): hourly forecasts; custom sources say when an option is unavailable
gwx advertised hourly but parsed it as part of the place name. It now requests Open-Meteo's hourly precipitation probability alongside the hourly fields it already got and formats the hours after Open-Meteo's location-local current time in wx's compact one-reply style, sharing the hour label, description shortening and byte packing through WeatherCommandMixin. Hourly lines drop the wind arrow (both commands) so the arrow's six bytes do not push an hour out of the reply; wx's NOAA hourly now keeps the full 16-point letters instead of the first two. WXSIM sources answer hourly and alerts with source_option_not_available instead of their default conditions; wx no longer skips an MQTT alias for alerts (the alias then got geocoded as a city); gwx answers 'alerts' the same way. WXSIM highs and lows of exactly 0° are no longer treated as missing, and gwx warns about a stale WXSIM forecast as wx does. Only noaa_wx_hourly_seattle moved (WNW/NNW instead of WN/NN).
2026-10-01 21:51:34 -07:00
agessaman a4513d17c9 fix(gwx): leave out missing current fields instead of inventing values
_open_meteo_current defaulted every missing current field to 0, so a response without current conditions read as clear skies, 0° and 0% humidity, and called int() on present-but-null fields, which failed the whole reply. A missing current temperature now returns None (the caller replies with error_fetching and logs a warning); null or absent humidity, wind, gusts, feels-like and weather code are each left out. The no_current and null_temp distortion goldens move accordingly; nothing else does.
2026-10-01 21:42:44 -07:00
agessaman 8874c54a67 fix(weather): date forecasts by the location's own times, not the bot's clock
wx took "today" from the bot's clock: on a server running UTC, or answering for a city in another zone, tomorrow and multi-day started a day late, and hourly compared NOAA's local wall-clock times with the bot's after dropping their offsets, keeping hours that had passed. wx now takes today from the forecast's first period date and compares hourly times as absolute times; multi-day groups NOAA periods by their start date, which also keeps holiday-named periods ("Columbus Day") the weekday-name grouping skipped. Name-based grouping remains the fallback when periods have no startTime. gwx chose Today/Tonight and the multi-day weekday labels from the bot's clock; it now uses Open-Meteo's location-local current time and daily dates. gwx also asked for N days counting today and then skipped today, so 7d showed six days; it now asks for N+1 (capped at Open-Meteo's 16). The NOAA characterization clock gains its time zone, so aware comparisons do not depend on the machine running the tests; the regenerated goldens move only where these fixes apply (drop-period variants whose first period is a later day, Miami's already-started 6 PM hour, gwx Today/Tonight and weekday labels, and gwx's forecast_days request parameter).
2026-10-01 21:24:03 -07:00
agessaman 5601a126dd refactor(weather): port the behavior-neutral weather refactor from refactor/maintainability
This brings the weather slices of my long-running refactor branch to dev ahead of converging wx and gwx, so the convergence PRs build on one copy of the code instead of rewriting it twice.

- `modules/weather_common.py` (WeatherCommandMixin): code wx and gwx shared by copy (MQTT and WXSIM source lookup, multi-day sending, high/low formatting, the forecast-option parser, the Open-Meteo model loader).
- `modules/nws_alerts.py`: one NWS alert entry parser for wx and the weather service, keeping each caller's own Special Weather Statement rules.
- `modules/http_retry.py`: one retrying session builder for the weather HTTP clients.
- wx: get_noaa_weather, the hourly, tomorrow and multi-day formatters, _add_period_details and the extract_* readers broken up or made table-driven; NOAA fetches go through `_noaa_fetch`.
- gwx: one fetch path; `_open_meteo_current` and `_open_meteo_daily_tail` split out of get_open_meteo_weather.
- solarforecast: `_format_forecast` built from one timestamp parser and one day builder (418 → 155 lines).
- aurora, rain, wx, gwx: companion and bot location lookups through `modules.location` (which gains optional logging arguments).

The changes outside these files that the refactor branch made (the base command's enabled_attr hook, the service base's run_periodic and _cancel_tasks) stay out: aurora and rain keep their can_execute overrides and the weather service its own loops, exactly as on dev.

Tests: characterization goldens over real captured NOAA, NWS, Open-Meteo and forecast.solar responses (tests/characterization, fixtures in tests/fixtures), 50 tests and several hundred cases, all of which also pass against dev's unmodified weather code. Each refactor step was additionally checked on the branch with an old-vs-new differential run (tens of thousands of generated inputs, including log calls and exceptions) and reviewed by codex.
2026-10-01 21:11:23 -07:00
agessaman 4f9a9e2414 test(characterization): pin the Open-Meteo and solarforecast edge branches the reviews listed
Codex reviews of the gwx and solarforecast refactors named branches the first characterization missed. These 228 cases cover them: Open-Meteo request timeouts, connection errors, bad JSON and HTTP errors for each forecast type with and without a model override; the default reply's precipitation arrays and width boundaries on both sides of max_length - 5 and - 10; tomorrow's short or missing wind, gust and precipitation arrays, light wind and small gusts; multiday's short weather_code array, short and long descriptions, an untranslated day name, num_days above and below the data and nulls in the middle; and solarforecast's tied peaks, exact threshold powers, several samples per hour, NaN powers and panels, sparse day sets and malformed containers, with debug logs. The Open-Meteo request and formatter functions are now fully covered apart from their exception handlers, which no input reaches.
2026-10-01 20:52:31 -07:00
agessaman b62806f58b test(characterization): fail when a public-API golden is missing instead of regenerating it
Both guards in test_public_api.py rewrote a missing golden and passed, so deleting public_imports.json along with an import would have hidden the removal. They now regenerate only under UPDATE_GOLDEN=1.
2026-10-01 20:21:03 -07:00
agessaman d7958ff951 Merge dev through #321 into refactor/maintainability
#321's null-precipitation guard in the default reply lands in `_open_meteo_daily_tail`, where bd235b93 moved that code; the conflict was only there. The Open-Meteo goldens are regenerated for #321: London's and Reykjavik's 15/16-day replies now list the days with data instead of the multi-day error, and the null-high distortions give the not-available texts instead of the error replies. No other golden entry changed.
2026-10-01 20:15:49 -07:00
agessaman 0d11717acd test(characterization): pin solarforecast's reply formatting over real forecast.solar responses
Three locations captured from api.forecast.solar on 2026-10-01, replayed with a frozen local clock from the evening before the first forecast day to late on the second, plus a 4-day variant (the free tier returns two days) for the Day+2/Day+3 line, five panel sizes, location labels, and nine distortions (Z and negative-offset timestamps, no watts, no days, a missing today, a None power value, an unparseable timestamp, all-zero watts). Exceptions that escape _format_forecast are recorded too. The goldens pin two existing quirks: timestamps with a negative UTC offset are never parsed, so they count no production hours, and a None power value raises out of the formatter.
2026-10-01 19:47:15 -07:00
agessaman 8f7a707bba refactor: restore 21 imported names earlier slices stopped exposing, and guard them
A codex review found that moving dashboard code out of app.py dropped `modules.web_viewer.app.DashboardStatsService`. The public-API golden only recorded names a module defines, so I added a second one, `public_imports.json`, recording the public names each module imports from another project module, generated from dev's tree as the baseline. It found 20 more names earlier refactor commits had stopped importing (the Nominatim and MQTT-weather helpers in wx, gwx, aqi and solarforecast; `PayloadVersion`, the region-warning verdicts and `decode_path_len_byte` in message_handler; two feed helpers and two feed limits in the web viewer). All are imported again as re-exports, so `modules.<module>.<name>` and patches of it resolve as they did on dev.
2026-10-01 19:36:29 -07:00
agessaman cd01a22023 test(characterization): pin gwx's Open-Meteo formatting over real responses
Five locations captured from api.open-meteo.com on 2026-10-01 (three metric, two imperial), replayed with a frozen clock across day/night, length budgets up to 600, the default/tomorrow/multiday types and fifteen distortions of the current and daily fields, recording the reply text, the request parameters and the info/warning/error log calls. The goldens include a real failure: Open-Meteo's default model returns nulls for the last forecast days, so a 15- or 16-day gwx reply for London or Reykjavik is the multiday error.
2026-10-01 19:17:18 -07:00
agessaman 5735483f20 Merge dev through #320 into refactor/maintainability
Brings in #309, #311, #314, #317, #319 and #320. Conflicts: command_manager's imports (kept both `inspect` and `math`) and wx's alert send, where I kept the S1 removal of the dead `weather_data[3]` statement and took #319's `skip_user_rate_limit=True`. The NOAA wx goldens are regenerated because #320 changes the wind text (`💨N0` → `⬆️N0`); nothing else in them moved.
2026-10-01 18:45:39 -07:00
agessaman 6460bded4a refactor(wx): only shorten Tomorrow's forecast text once the period passes its guard
Review of the get_noaa_weather breakup: computing the abbreviated text before the temperature/forecast check evaluated len(shortForecast) for periods the old code skipped, so a null shortForecast raised inside the catch-all and replaced the whole reply with an error. The abbreviation now runs inside the guard again. A test with null Tomorrow fields passes on the code before the breakup, fails on 5ff0f84 and passes now.
2026-10-01 16:07:05 -07:00
agessaman 74772207a3 test(characterization): pin _add_period_details over details text, observations and budgets
Real NOAA forecast text rarely carries humidity, gusts or pressure, and never a precipitation probability in a phrasing the extractor matches, so the length check and several formats went unexercised. A matrix of five synthetic forecast texts, three observation dicts (none, full, falsy values), five current lengths and two budgets pins the output; goldens generated on the code before the next commit.
2026-10-01 16:03:11 -07:00
agessaman 3587bc0bd3 test(characterization): reach wx's long-forecast abbreviation with multi-clause forecasts
The real NOAA samples never had a long short-forecast after a night period, so the Tomorrow block's abbreviation went unexercised. A variant rewrites later periods' shortForecast to multi-clause phrases ('A then B', 'X Becoming Y') and runs at 130, 160 and 200 characters; goldens generated on the code before the get_noaa_weather breakup.
2026-10-01 15:58:30 -07:00
agessaman eb80c2b3b4 test(characterization): pin wx's NOAA forecast formatting on real responses
Replays api.weather.gov points, forecast, station and observation responses captured on 2026-10-01 for Seattle, Denver, Miami and Anchorage (geometry stripped), plus hourly forecasts for two of them, with wx_command's clock frozen at capture time. Each location runs get_noaa_weather with 0-3 leading periods dropped (other times of day), with and without a station observation, at 130 and 200 characters, and pins format_tomorrow_forecast, format_multiday_forecast for 3/5/7 days, _add_period_details and format_hourly_forecast. Coverage of those functions is 72-85%, the rest mostly error paths. This guards the planned breakup of get_noaa_weather and friends.
2026-10-01 15:48:56 -07:00
agessaman 40989a4085 test(characterization): fail when a public name under modules/ disappears
Codex caught three deleted public methods earlier and a later scan found twelve more, because nothing tested for them. The golden records every public module function, class and method; the test fails only on a removal, and accepts a name that still resolves through a base class or re-export.
2026-10-01 09:26:36 -07:00
agessaman c7001e9f3d test(characterization): reach wx's localName CAP pass with nested entries
Two synthetic entries carry CAP fields inside a nested cap:info element, which only wx's DOM localName pass reads. Goldens regenerated on the pre-refactor parsers.
2026-10-01 07:44:19 -07:00
agessaman e3f4b09231 test(characterization): pin wx and weather-service NWS alert parsing
Both parsers run over a trimmed sample of real NWS active-alert feeds (TX, CA, AK, FL on 2026-10-01) and synthetic entries that reach every Special Weather Statement headline and summary branch, the title-only fallbacks, missing titles and each link shape. wx is pinned through get_weather_alerts_noaa (full list, compact text and one feed per entry, plus its warning/error logs); the service through _parse_alert_entry. Key order is recorded because the golden JSON sorts keys.
2026-10-01 07:38:36 -07:00
agessaman 7d91d9f557 refactor(web_viewer): route-level _api_errors decorator for the common error handler
42 routes wrapped their whole body in try / except Exception as e: self.logger.error(f'<prefix>: {e}'); return jsonify({'error': str(e)}), 500. Exactly those (matched by AST: one handler, that log call shape, that return, nothing else) now use @self._api_errors('<prefix>') under their route decorator, which logs the same line and returns the same response; functools.wraps keeps the endpoint names the CSP nonce list and url_for depend on. The endpoint inventory golden is unchanged. Routes with any other error handling are untouched.

Dedenting the bodies changed the indentation inside one multi-line SQL string in api_recent_commands; SQLite ignores it.
2026-10-01 07:10:45 -07:00
agessaman 53361328a2 refactor(weather): share wx and gwx helpers through WeatherCommandMixin
Five methods were the same in WxCommand and GlobalWxCommand: _format_high_low, _get_custom_mqtt_weather_topic, _mqtt_weather_line (only the catalog namespace differed), _count_display_width and _send_multiday_forecast. They now live once in modules/weather_common.WeatherCommandMixin, which both commands list before BaseCommand; each sets translation_ns ('commands.wx' / 'commands.gwx'). The mixin is outside modules/commands, so the plugin loader does not treat it as a command.

A new characterization test pins how a multi-day forecast is packed into messages (packing, rate-limit skips, pauses); it passes on the code before and after.
2026-10-01 06:25:01 -07:00
agessaman 64d6e1a9c8 refactor(web_viewer): read and write metadata-backed settings through two helpers
The notifications, log-rotation and maintenance settings routes each looped over '<prefix>.<field>' keys in bot_metadata, filled unset values with '' and then applied defaults to empty ones, and the matching POST routes (plus zombie-alert's) each looped over an allowed set. _read_meta_settings and _write_meta_settings do both. The zombie-alert GET and the radio probe/offline routes are left as they are: they treat non-string values differently or merge config.ini fallbacks. A new test pins defaults, stored-value precedence, empty-string fallback and allowed-field saves; it passes on the code before and after.
2026-10-01 01:42:44 -07:00
agessaman f829ed9ddd test(characterization): pin packaging coverage and the web viewer route table
The packaging test checks pyproject.toml against git ls-files: every directory under modules/ with Python files must be a regular package, and every tracked runtime data file must match a package-data glob, since include-package-data is off and a new subpackage or data file would otherwise silently miss the wheel. The endpoint inventory records all 104 routes with their methods and endpoint names, plus the status, content type, redirect target, script-src CSP (nonce normalized) and framing headers of anonymous requests with and without a password, and of anonymous mutating requests when locked. Endpoint names feed the CSP nonce allowlist and url_for, so a helper extraction that renames or reroutes anything shows up here.
2026-10-01 01:02:33 -07:00
agessaman eda2f12e1d test(characterization): pin execution accounting and reload rollback
Execution accounting runs execute_commands with an advancing clock: when the cooldown is recorded, how a second record_execution inside execute moves the boundary, which stats rows each outcome writes, and that zero-argument cooldown methods from local plugins still work. The stale _last_response defect (a silent command after any reply counts as responded) is pinned as current behavior so the refactor cannot change it by accident. Reload rollback injects a failure at each stage (command construction, introduced plugin failures, the region-warning monitor, the scheduler) and asserts every attribute the rollback owns is restored.
2026-10-01 01:00:49 -07:00
agessaman ca26f3e9fc test(characterization): pin flood-scope resolution and node-name lookups
First slice of the maintainability refactor: golden-file characterization of behavior that later commits consolidate. The scope matrix runs every resolver (MeshMessage, resolve/effective_channel_send_scope, channel_body_budget, the scheduler's budget resolver, BaseServicePlugin.get_mesh_flood_scope, and the scopes a real send_channel_message hands to set_flood_scope) against 420 combinations of override, per-channel and section scope. The identity matrix runs each node-name lookup against every self_info shape. Both record today's divergences rather than fixing them, so a refactor commit that changes either golden file is a behavior change by definition.
2026-10-01 00:58:53 -07:00