- Added a type ignore comment for the assignment of optionxform in config_schema.py to suppress type checker warnings.
- Updated path inference logic to return None when bot latitude or longitude is not set, ensuring safer handling of missing values.
- Enhanced the decoding of path nodes to check for None before appending repeater details, preventing potential errors.
Three independent wins, measured against a 1.4 GB production database
(714k observed_paths, 38.7k mesh_connections):
- Migration 0014: partial covering index on observed_paths for
bytes_per_hop >= 2. The multibyte evidence query was a full table
scan (2.2 s); with the index it reads only the index (77 ms). The
optional days filter is covered too via last_seen in slot 2.
- Compress responses with flask-compress when the client supports it.
/api/mesh/edges alone is 16.3 MB of JSON uncompressed, 3.9 MB
gzipped. Falls back gracefully (with a warning) if the package is
not installed.
- Fetch stats, edges, and nodes concurrently on the mesh page instead
of three serialized awaits. Node prefixes are now computed client-side
from public_key at the edge prefix length, which removes the
edges-before-nodes ordering dependency. Rendering still waits for
stats so initial framing can use the bot location.
Also call map.invalidateSize() before the initial fitBounds: if the
map was created while its container had no layout (e.g. a background
tab), Leaflet's cached zero width made fitBounds zoom to max.
Nodes occasionally carry wrong GPS, which stretched the initial
fit-to-all-nodes view across continents. /api/mesh/stats now exposes
the bot's configured position ([Bot] bot_latitude/bot_longitude), and
the mesh page BFSes the multi-byte-evidence subgraph from repeaters
within 30 km of the bot (nearest one as fallback), framing the initial
map view to that connected component. Falls back to the old fit when
the bot position is unset or the component is trivially small; other
mesh islands stay on the map, just outside the initial frame.
- Hot connections mode now also colors repeater dots with viridis by
their total observation count across touching edges (log-normalized,
same dark-mode ramp compression as edges); node popup gains a Path
Observations row and the heat legend a clarifying note
- Density pass: smaller dot base/cap (5..13 vs 6..16), edge widths
capped at 5px, and both dots and lines scale down to 55% as the map
zooms out (full size from zoom 12 in), rescaled live on zoomend
- Node degree and observation totals are now precomputed once per
filter pass (computeNodeStats) instead of per-node scans over all
edges, using the same endpoint resolution as the renderers
- Multi-byte evidence is now the default mode for the mesh graph
- Mesh display theme: follows the site theme until toggled on the page,
then remembered independently (localStorage); dark CARTO basemap,
dark Leaflet popups/controls/legends, dark graph canvas
- Hot connections: optional viridis edge coloring by log-scaled
observation count with a gradient legend; the near-black cold end of
the ramp is compressed away in dark mode so cold edges stay visible
- Single-row header (title + stat chips + icon controls) replaces the
title and stats-card rows; footer hidden on this page to maximize the
visualization viewport
- Mobile: page scrolls naturally instead of starving the fixed-height
flex container (which stranded the absolutely-positioned legends)
The old slider defaulted to floor(avg observations) — on heavy-tailed data
the average sits above the 75th percentile, so the first visit hid most of
the graph. Worse, the default (e.g. 98) exceeded the slider's max of 50:
the browser clamped the control while the label kept showing the unclamped
value, so the label lied about the active filter.
- slider position now maps logarithmically onto 1..max(observation_count),
rescaled whenever edge data loads, giving fine control at the low end
and full reach into the tail
- first-visit default is the median observation count of the loaded edges
- the persisted value is the threshold itself, so it stays meaningful when
the scale changes (e.g. switching evidence modes)
- label shows the threshold plus live 'shown/total edges' feedback
Single-byte repeater identity guessing has been persistently unreliable,
and mesh_connections flattens away evidence provenance (confirmed_2byte
never persists). observed_paths retains it: bytes_per_hop marks which
paths carry unambiguous multi-byte hops.
- /api/mesh/edges?evidence=multibyte derives edges purely from unique
multi-byte path observations (consecutive hop pairs, aggregated with a
distinct-path count); 2-byte edges coalesce into a 3-byte edge only on
a unique prefix match, mirroring MeshGraph.add_edge
- default mode tags each edge evidence=multibyte|singlebyte by key length
- Evidence filter on the mesh page (persisted with the other filters);
single-byte edges render dashed in map and graph views, with an
evidence line in tooltips and a legend entry
- client-side haversine fallback for edges without a stored distance
- prefix-compatible edge/node matching so degree sizing and node details
work when edge and node prefixes differ in resolution
- /api/mesh/stats reports multibyte_edges; shown on the Edges stat card
Adds device-level companion settings to /radio, headlined by the response
path hash size (mode 0-2 = 1-3 bytes per hop) wired to the existing
firmware config endpoints. New sections: Identity & Adverts (name, advert
lat/lon, location policy, zero-hop/flood advert buttons), Mesh Behavior
(extra ACKs, telemetry permissions), and write-only Advanced Tuning
(RX delay base, airtime factor, sent x1000 per the wire format).
Backend: GET /api/radio/params now returns the full SELF_INFO node fields;
POST accepts the new fields with validation; new POST /api/radio/advert;
scheduler ops handle the writes (other-params group as one read-modify-write
frame so partial updates never clobber device state).
Config-managed settings are not writable from the panel: new-contact mode
is owned by [Bot] auto_manage_contacts and shown read-only; the node name
is locked while bot_name + auto_update_device_name manage it. loop.detect
is removed from the firmware endpoints entirely - it is repeater/room-server
CLI config, and companion custom vars map to sensor settings.
Also: dark-mode styling for disabled form fields (base.html), TASK-01 guard
test updated for the deliberate reintroduction, 23 endpoint/template tests.
The legacy [Section] enabled alias map (e.g. [Jokes] joke_enabled,
[Stats] stats_enabled) was duplicated in BaseCommand.get_config_value and
modules/settings_schema.py, so runtime behavior and the web settings UI
could drift apart. Both now read config_schema.LEGACY_ENABLED_ALIASES.
- meshcore_bot.py: run config validation on normal startup and print
errors/warnings (capped at 15) — misspelled sections/keys previously
failed silently, the top source of configuration confusion
- ini_writer: append new keys directly under a section's last entry
instead of after its trailing blank line
- add .claude/launch.json for browser-preview of the standalone viewer
Verified end-to-end in the browser: /plugins renders 43 command cards,
opt-in commands show off, dynamic-row add/save writes config.ini with a
timestamped backup and preserved comments.
- reload_config: clear all sections before re-reading so keys deleted from
config.ini (e.g. rows removed in the settings UI) don't survive in memory
(ConfigParser.read merges instead of replacing)
- reload_config: re-instantiate command plugins so settings cached in
__init__ (including 'enabled') take effect on reload
- /api/plugins save: treat dynamic_sections / repeating_blocks absent from
the payload as 'don't touch' instead of 'delete all managed keys'
- settings view: respect per-plugin settings_enabled_default so opt-in
commands (Announcements, Greeter) display as off when unconfigured
- ruff: drop unused import, organize command_manager imports
- add tests for ini_writer, settings_schema, settings_store and the
/api/plugins endpoints (38 cases) plus reload regression tests
- Updated `config.ini.example`, `config.ini.minimal-example`, and `config.ini.quickstart` to clarify connection type precedence and required keys for serial, BLE, and TCP connections.
- Added detailed comments and examples for each connection type, ensuring users understand how to configure their connection settings effectively.
- Introduced a new `[Service_Overrides]` section in `config.ini.example` for alternative service plugin implementations, improving extensibility.
- Enhanced documentation in `configuration.md` to reflect these changes and provide clearer guidance on connection options and templates.
- Modified the core service to start the webhook service before establishing a radio connection, reducing the window for connection refusals.
- Implemented a readiness check in the webhook service to return a 503 status when the bot is not connected, ensuring clear communication to external callers.
- Added unit tests to verify the webhook service's behavior when the bot's connection status changes, including rate limiting and readiness responses.
Add optional payload decoding to the packet capture service. GRP_TXT
channel messages are decrypted (sender/text), ADVERTs are parsed
(name/role/lat-lon), and a nested "decoded" object is attached to each
packet alongside the unchanged raw fields.
- Comprehensive channel key store: bot's configured radio channels plus
decode_hashtag_channels, [Channels_List], decode_channel_keys, and the
built-in default Public key.
- Publishing the decoded object to MQTT is off by default and
configurable per broker via mqttN_include_decoded.
- Configurable packet-log rotation (off/size/time) for historical dumps.
The decoder lives in a standalone, dependency-free module
(modules/meshcore_payload_decode.py) so it can be shared verbatim with
the meshcore-packet-capture project.
Closes#197Closes#35
- Updated logging statements in the `MessageHandler` class to use `debug` level instead of `info` for `RAW_DATA` and `NEW_CONTACT` events, reducing log verbosity while maintaining essential information.
- This change enhances the clarity of logs by reserving `info` level for more significant events.
- Resolves issue #214
- Added a `--strict` option to `validate_config.py` to fail on "Unknown section" messages, aimed at CI validation of example configs.
- Updated GitHub Actions workflow to validate shipped example configs against the canonical section list using the new strict mode.
- Expanded the `CANONICAL_NON_COMMAND_SECTIONS` in `config_validation.py` to include additional sections for improved validation accuracy.
- Introduced a regression test to ensure example configs do not trigger unknown section warnings, maintaining consistency with the canonical list.
- Updated `config.ini.example` to support single, multiple, and decorative command prefixes, improving flexibility in command invocation.
- Refactored `CommandManager` and `BaseCommand` to utilize new command prefix parsing logic, allowing for better handling of multiple prefixes and optional bare commands.
- Enhanced documentation to clarify command prefix configuration options and behavior, ensuring users understand how to set and use prefixes effectively.
- Added unit tests to verify the correct functionality of command prefix handling, including support for permissive and strict prefix modes.
- Updated configuration examples to include new `{hops}` and `{hops_label}` placeholders for path command replies, enhancing response detail.
- Modified `BaseCommand` to support hop count retrieval and formatting, ensuring accurate display of hop information in responses.
- Enhanced unit tests to verify correct formatting of hop-related placeholders in response messages, improving overall command functionality.
- Updated the `[Rain_Command]` section in `config.ini.example` to include support for snow alongside rain, improving the command's functionality.
- Enhanced documentation for the rain command to reflect the new snow alias and clarify response behavior based on the selected keyword.
- Added a new `collect_stats` option in the `[Stats_Command]` section, allowing stats collection to be enabled independently of the user-facing command, with updated documentation to explain its behavior.
- Improved the web viewer documentation to clarify how stats are collected and displayed, ensuring users understand the configuration options.
- Updated the `CommandManager` to respect the `channels` override for the help command, ensuring it checks channel permissions directly when processing help requests.
- Refactored the logic to fall back to global `monitor_channels` only when no specific help command is loaded, improving channel-specific access control.
- Added unit tests to verify the correct behavior of the help command in disallowed channels, ensuring it only responds in permitted contexts.
- Updated the `CommandManager` to respect the `help_enabled` flag for the help command, ensuring it only responds when the command is enabled.
- Refactored help request processing to streamline checks for help keywords and channel restrictions.
- Added unit tests to verify the behavior of the help command when enabled and disabled, ensuring proper suppression of responses when the command is not active.
- Deleted the `[Custom_Syntax]` section from `config.ini.example` and `core.py`, simplifying the configuration by removing outdated custom syntax patterns and examples.
- This change streamlines the configuration files, focusing on essential settings and improving clarity for users.
- Enhanced `CommandManager` to persist normalized message content, preventing prefix stripping from interfering with command matching.
- Updated `BaseCommand` to skip prefix handling if already normalized, ensuring all commands can be matched correctly.
- Modified `HelpCommand` to ensure unused commands are listed even when command statistics are present, improving user accessibility to all commands.
- Added unit tests to verify the correct behavior of command matching and help command functionality under various scenarios.
- Introduced per-channel flood scope configuration in `config.ini.example`, allowing for more granular control over message routing.
- Updated `CommandManager` to normalize channel names and resolve flood scopes based on channel-specific settings, improving message delivery accuracy.
- Enhanced `MeshMessage` to consider channel-specific flood scopes when determining the effective outgoing flood scope.
- Added unit tests to verify the new flood scope resolution logic and ensure correct behavior across various scenarios.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
- Enhanced the `TestCommand` class to include the `rssi` value in the response format, allowing for more detailed output.
- Updated the example configuration to reflect the new placeholder usage for `rssi`.
- Added a unit test to verify that the `rssi` placeholder expands correctly in the response message.
- Modified `WxCommand` to use its own keywords along with any aliases from `BaseCommand` when delegating to `GlobalWxCommand` for the Open-Meteo weather provider.
- Added a new test fixture for `WxCommand` configured to delegate to `GlobalWxCommand`, ensuring that aliases are preserved and correctly matched in keyword checks.
- Enhanced unit tests to verify the delegation behavior and keyword matching for both standard and aliased commands.
- Added a new function `nws_http_means_no_coverage` to determine if NWS HTTP status codes indicate no coverage for a location.
- Updated `WxCommand` and `WeatherService` classes to utilize this function, improving error handling for NOAA alerts.
- Introduced a lazy loading mechanism for `_nws_alerts_available` to manage alert availability based on NWS coverage status.
- Enhanced logging to provide clearer warnings when NWS alerts are unavailable due to HTTP errors.
- Introduced a new class `_SerializedCommands` to serialize host-to-radio commands, ensuring only one command is in-flight at a time to prevent USB-CDC buffer overruns and parser desynchronization.
- Added configuration options for `command_min_interval_ms` and `channel_fetch_interval_ms` to control pacing between commands and channel scans, improving firmware stability during reconnect sequences.
- Updated `ChannelManager` to utilize the new fetch interval during channel scans, and modified the reconnect delay to 10 seconds for gentler recovery from connection issues.
- Enhanced documentation in `config.ini.example` to reflect new settings and their purposes.
NEW_CONTACT adverts were always logged as newly discovered, even for
nodes heard many times before. Check the tracking DB (get_tracked_contact_row)
and the live device contact list (is_contact_on_device) to classify the
advert, log "known" vs "new" accordingly, and skip the
new_contact_discovered audit entry for contacts that are already known.
Also dedup the is_currently_tracked device-contact lookups onto the shared
is_contact_on_device() helper.
Behavior preserved: repeaters remain database-only, device-mode companion
adds are still left to the firmware, and no per-advert device round-trip is
introduced.
- Added new utility functions `espn_dates_for_local_day` and `filter_events_local_day` to handle local timezone date calculations and event filtering.
- Updated `ESPNClient` to support optional date parameters in `fetch_scoreboard_with_calendar` and `fetch_match_states` methods, allowing for more precise data retrieval.
- Modified `WorldCupCommand` and `WorldCupLiveService` to utilize the new date range functionality, ensuring accurate event reporting based on local time.
- Implemented unit tests to verify the correct behavior of date handling and event filtering in various scenarios.
- Added configuration options for announcing red cards, yellow cards, and match stoppages in `config.ini.example`.
- Updated `ESPNClient` to parse and include red and yellow card details in match state data.
- Enhanced `WorldCupLiveService` to announce red and yellow cards, including formatted messages for players and stoppage events.
- Implemented unit tests to validate the new card and stoppage announcement functionalities, ensuring accurate reporting during matches.
- Introduced a new configuration option `announce_disallowed` to allow notifications for goals overturned by VAR.
- Updated `WorldCupLiveService` to handle disallowed goals, including formatting for previously announced scorers.
- Enhanced `ESPNClient` to ensure accurate goal tracking and state management.
- Added unit tests to validate the new disallowed goal functionality and ensure correct behavior during matches.
- Updated ESPNClient to include a detailed list of scoring plays in the live match state, capturing clock, scorer, team ID, own goal, and penalty information.
- Modified WorldCupLiveService to track and announce goals, including formatting for new scorers and handling multiple goals since the last poll.
- Added unit tests to validate goal tracking functionality and ensure accurate announcements during matches.
- Added cache_ttl parameter to fetch_precip_series_nws to enable caching of results for improved performance.
- Implemented logic to reuse cached results based on location and cache expiration.
- Updated tests to ensure proper handling of cache_ttl without causing errors.
- Refactored WorldCupFastcastClient to streamline connection handling and improve readability.
The Open-Meteo forecast model smooths away scattered, pop-up convection, so the rain nowcast (the rain/snow command and the proactive push) could read 0.0 in / ~12% and stay silent while rain was actually falling. Observed near Nashville: Open-Meteo reported 0.0 in across the next 3 h while NWS's own gridpoint showed 65-74% probability with measurable QPF, and thunderstorms were occurring.
Add fetch_precip_series_nws(), which builds the same nowcast-series shape from the NWS gridpoint forecast (6-hour QPF + hourly PoP + weather type). Each hour's precip is its QPF share, zeroed when that hour's PoP is below a floor, so the predicted rain-start tracks the hourly probability instead of snapping to coarse 6-hour QPF boundaries. Both fetchers now prefer NWS for US points and fall back to Open-Meteo where NWS has no coverage (outside the US) or on failure, so the command and the push agree and the model's convective blind spot no longer silences the alert.
Pure helpers (_iso_duration_hours, _nws_hourly, _nws_weather_code) are unit-tested; the NWS-weather -> WMO-code mapping keeps bucket classification (rain/snow/thunder/...) identical to the Open-Meteo path.
- Introduced World Cup command and service settings in `config.ini.example`, `config.ini.minimal-example`, and `config.ini.quickstart` to enable FIFA World Cup score tracking and updates.
- Enhanced `espn_client.py` with methods to fetch tournament calendar and match states, supporting the new World Cup features.
- Updated `sports_command.py` to resolve World Cup nation names and integrate with the new command.
- Added translations for World Cup commands and messages in multiple languages, ensuring comprehensive support for users.
- Included mappings for World Cup nations in `sports_mappings.py` to facilitate accurate team identification.
- Introduced a new static method `_utc_iso_timestamp` to generate UTC ISO 8601 timestamps with a 'Z' suffix for compatibility.
- Updated timestamp generation in `log_packet` and status message to utilize the new method, ensuring consistent timestamp formatting across the service.
- Added a unit test to verify the correct format of the new timestamp method.
Builds on the merged rain/snow nowcast (#193): ten enhancements plus
end-to-end, proactive, and live-smoke test coverage. All new behavior is
config-gated or additive, so existing deployments are unaffected by default.
Enhancements
- Precip amount estimate "(est 0.2 in)" on the command and the proactive push;
snow shown as real depth (Open-Meteo snowfall, cm); freezing rain tagged "in ice".
- Bare country / US state resolves to its capital with a heads-up
(self-contained modules/region_capitals.py; no pycountry/us dependency).
- join_location() dedupes "Spain, Spain" / city-states.
- !snow alias + neutral !nowcast; each looks for its own precip family across
the window, else falls back with a "No snow, but rain ..." cross-type line.
- Keyword-aware help (help rain / help snow).
- Precip probability shown "(..., 70%)"; the proactive incoming alert is gated
to >= [Weather_Service] rain_nowcast_min_probability (default 50).
- Rain<->snow changeover line when the window holds both families.
- Borderline temperature tag (30-38F).
- Short-lived series cache shared by the command and the proactive poll.
Tests
- test_rain_command_e2e.py: drives RainCommand.execute() end to end and asserts
the exact rendered reply across dry/incoming/raining, snow depth, cross-type
mismatch, changeover, ice, temp tag, region capitals, config toggles, DM
budget, and keyword-aware help.
- test_rain_proactive_e2e.py: drives Weather_Service._check_rain_nowcast() for
the probability gate, snow depth, ending notice, and once-per-episode dedup.
- test_rain_live_smoke.py: opt-in (RAIN_LIVE_SMOKE=1) live Open-Meteo check for
upstream schema drift; skipped in CI.
- Shared scaffolding in tests/unit/_rain_harness.py.
New config: [Rain_Command] show_amount/amount_unit/show_probability/show_temp/
cache_seconds/zip_city_lookup; [Weather_Service] rain_nowcast_min_probability/
rain_nowcast_cache_seconds/rain_nowcast_show_amount/rain_nowcast_amount_unit.
ruff + mypy clean.
- Updated the Makefile and GitHub Actions workflow to install ruff version 0.15.15, ensuring consistent linting behavior across environments.
- Added a required-version entry for ruff in pyproject.toml to prevent drift in lint rules.
- Modified the base_service.py method to return None instead of passing, improving clarity.
- Removed an unnecessary blank line in the test_packet_capture_transport_reconnect.py file.
Add a rain nowcast (Open-Meteo 15-minutely precipitation — worldwide, no API
key) as both an on-demand `rain`/`nowcast` command and an opt-in Weather_Service
push that announces rain starting and stopping at the bot's position.
- modules/commands/rain_command.py — command plus pure, unit-tested
fetch/analyze/dedup/label helpers (also reused by the service)
- modules/service_plugins/weather_service.py — background poller mirroring the
existing weather-alert poll pattern; ships disabled (opt-in)
- Location labels resolve to "City, ST" (US) / "City, Country" (non-US)
- 34 unit tests; rain_command added to the strict-mypy module list
- Updated `contacts.html` to escape HTML for contact usernames and advertisement data, preventing potential XSS vulnerabilities.
- Modified `mesh.html` to escape node names and prefixes in tooltips, ensuring safe rendering in the vis-network.
- Removed unnecessary parameters from the `showDeleteConfirmation` method to derive the username directly from the contact data.
- Introduced a new utility function `public_key_has_prefix` to check if a public key starts with a specified prefix in a case-insensitive manner.
- Updated the `PathCommand` class to utilize the new utility function for public key prefix matching, enhancing code readability and maintainability.
- Refactored logic in `PathCommand` and `BotDataViewer` to streamline the handling of recent repeaters based on recency scores, improving clarity in the filtering process.
agessaman