We implement support for the DARC MoWaS service that distributes
emergency messages from the German BBK via the "Warnmultiplikator"
interface to Meshcore channels. This service is push-based and receives
alerts on /api/alert, which are then formatted similar to cell-broadcast
messages (without modification of the text) and re-distributed on
Meshcore.
Signed-off-by: Felix Moessbauer <felix.moessbauer@gmail.com>
- Updated config.ini.example and discord-bridge.md to reflect the ability to fan out a single MeshCore channel to multiple Discord servers using a comma- or whitespace-separated list of webhook URLs.
- Modified DiscordBridgeService to handle multiple webhooks per channel, including validation and logging improvements for better monitoring of configured webhooks.
- Updated the _cleanup_web_viewer and _cleanup_mesh_graph methods to avoid logging errors during shutdown, as the logger's stream may be closed at that time.
- Modified the shutdown method in MeshGraph to prevent logging of flushing errors, enhancing stability during the atexit process.
- Adjusted test configurations to use Path objects for bot_root and local_root, improving path handling in tests.
- Modified the RepeaterManager class to enable or disable auto-purge based on the 'auto_manage_contacts' setting from the bot's configuration.
- This change allows for more flexible management of contacts, aligning with the bot's operational mode.
- Changed the default IATA code from 'LOC' to 'XYZ' in `config.ini.example` and `packet-capture.md` to reflect updated routing requirements.
- Updated the `PacketCaptureService` to use 'XYZ' as the fallback IATA code, ensuring consistency across the application.
- Enhanced the ServicePluginLoader to add the modules directory to the path for local services, ensuring that utility modules can be resolved correctly.
- This change improves the loading mechanism for service plugins by allowing better integration of local utilities.
- Updated the ServicePluginLoader to include a built-in services directory in the module path, improving the loading mechanism for service plugins.
- Modified the class to accept BaseServicePlugin from both built-in and local services, allowing for greater flexibility in service integration.
- Enhanced the logic for identifying service classes, ensuring compatibility with multiple base service implementations.
- Updated the TraceCommand class to enhance the labeling of hops in the trace output.
- Replaced direct hash retrieval with a conditional approach to determine the source label for each hop, improving clarity in the output format.
- Ensured consistent labeling for the first and last hops, enhancing the readability of the trace results.
- Updated the command to handle 1-, 2-, or 3-byte hex prefixes, with longer input truncated to 3 bytes.
- Improved usage instructions and examples to clarify accepted prefix formats (2, 4, or 6 hex characters).
- Enhanced validation logic for prefix format, ensuring proper handling of hex strings and truncation.
- Updated documentation strings to reflect changes in prefix handling and expected input formats.
- Updated the MeshGraph class to prevent promoting a 1-byte edge to a 3-byte edge when the existing 1-byte edge lacks public keys, ensuring accurate observation attribution.
- Added a new test case to verify the behavior of edge promotion under specific conditions.
- Modified the BotDataViewer API to return the prefix length dynamically based on the edges, improving data consistency and user experience in the web viewer.
- Enhanced the mesh.html template to support displaying prefix byte counts, providing clearer information on node connections.
- Introduced a check in `load_service_from_path` to ensure the `local_services` module is created in `sys.modules` if it doesn't exist, facilitating proper relative and absolute imports for service plugins.
- This enhancement improves the loading mechanism for local service plugins, ensuring smoother integration and functionality.
- Added support for rotating log files in `core.py` using `RotatingFileHandler`, with a maximum size of 5 MB and up to 3 backup files.
- Updated `config.ini.example`, `config.ini.minimal-example`, and `config.ini.quickstart` to include descriptions of the new log rotation feature.
- Enhanced `data-retention.md` to clarify log file management and retention policies.
- Updated documentation in `data-retention.md` and `web-viewer.md` to clarify how stats are collected and displayed, including the new `collect_stats` configuration option.
- Modified `StatsCommand` in `stats_command.py` to introduce `collect_stats`, allowing message and command statistics to be recorded even when the `stats` command is disabled.
- Adjusted logic in `record_message`, `record_command_stats`, and `record_path_stats` methods to utilize the new `collect_stats` setting for improved flexibility in stats tracking.
- Updated `config.ini.example` and `config.ini.minimal-example` to include optional `local_dir_path` for specifying a custom local plugins directory.
- Enhanced `install-service.sh` to preserve the `local/` directory during installation and create its structure if it doesn't exist.
- Modified `CommandManager` and `MeshCoreBot` to resolve the local commands and services directory based on the new configuration.
- Added validation in `config_validation.py` to check the existence and readability of the specified local plugins path.
- Improved documentation in `local-plugins.md` to clarify the usage of local plugins and the new configuration options.
- Added a `join` method in `MessageScheduler` to wait for the scheduler thread to finish during shutdown.
- Updated the shutdown sequence in `core.py` to include a timeout for the `meshcore.disconnect()` method, with error handling for potential timeouts and exceptions.
- Improved logging to provide clearer information during the shutdown process.
- Added `send_channel_messages_chunked` method to `CommandManager` for sending multiple messages with appropriate rate-limit spacing.
- Introduced `send_response_chunked` method in `BaseCommand` to facilitate chunked responses in both channels and DMs.
- Updated `GreeterCommand` to utilize the new chunked response functionality, simplifying the greeting message sending process.
- Enhanced documentation in `local-plugins.md` with examples for using chunked message sending, improving clarity on handling long messages and rate limits.
- Updated MeshGraph to support multi-resolution storage and node identity, allowing for better handling of 1-byte, 2-byte, and 3-byte edges.
- Implemented logic for merging and promoting edges based on observation counts and specificity.
- Enhanced BotDataViewer API to dynamically adjust prefix length based on configuration, improving data representation.
- Updated mesh.html template to display prefix byte counts, enhancing user experience with clearer information on node connections.
- Added support for bytes per hop in the BotDataViewer, allowing for better path data representation.
- Updated the contacts template to display bytes per hop and adjusted path formatting based on this value.
- Improved the decode path functionality to utilize the correct bytes per hop for decoding paths, enhancing overall path handling and user experience.
- Updated `meshcore` dependency version to `2.2.14` in both `pyproject.toml` and `requirements.txt`.
- Added multi-byte path support in the `PathCommand`, allowing for 1-, 2-, and 3-byte-per-hop paths.
- Enhanced `MessageHandler` to utilize `routing_info` for accurate path extraction and validation.
- Improved path extraction methods in `MultitestCommand` and `TestCommand` to prefer `routing_info` for node IDs.
- Refactored path handling logic across various commands to ensure consistent multi-byte path processing.
- Updated `.gitignore` to include local configuration and plugin directories, allowing users to add custom commands and services without modifying core code.
- Enhanced `config.ini.example` with instructions for using local plugins and added sections for local service configurations.
- Refactored `PluginLoader` and `ServicePluginLoader` to support loading local commands and services from specified directories, improving extensibility.
- Updated `mkdocs.yml` to include documentation for local plugins and the check-in API.
- Added tests to verify the discovery and loading of local plugins, ensuring functionality and preventing name collisions with built-in plugins.
- Updated the logic in `TraceCommand` to avoid capping the `path_nodes` after building the reciprocal path, ensuring that the return path is preserved.
- Improved clarity in the code by adding a comment to explain the rationale behind the change.
- Added support for `{hops}` and `{hops_label}` placeholders in `config.ini.example` and `core.py`, allowing for dynamic display of total hop count and pluralized labels in messages.
- Updated `format_keyword_response_with_placeholders` function in `utils.py` to extract and format hop count from message data, improving message clarity and detail.
- Ensured backward compatibility by providing default values for hop-related placeholders when not set.
- Added configuration options to bridge the bot's own channel responses to Discord and Telegram, allowing command replies to be sent to respective channels.
- Updated `DiscordBridgeService` and `TelegramBridgeService` to register listeners for bot-sent messages, ensuring they are included in the bridging process.
- Enhanced `CommandManager` to invoke listeners with a synthetic event when a message is successfully sent, providing context about the message.
- Updated `config.ini.example` to reflect the new `bridge_bot_responses` option for both services.
- Added tests to verify the correct invocation of listeners upon successful message sending.
- Added support for channel-specific triggers in the CommandManager, allowing triggers to be restricted to designated channels via `channel.<key>` or `channels.<key>` configuration.
- Updated `config.ini.example` to include examples of channel restrictions for triggers.
- Enhanced tests to verify that triggers only match in specified channels, ensuring proper functionality of the new feature.
- Updated the profanity filter to check for hate symbols (e.g., swastika Unicode) in addition to word-based profanity, ensuring comprehensive message filtering.
- Modified the `censor` function to replace hate symbols with `***`, maintaining functionality even when the `better-profanity` package is unavailable.
- Updated documentation in `discord-bridge.md` and `telegram-bridge.md` to reflect the new hate symbol handling features.
- Added tests to verify detection and censorship of hate symbols, ensuring robustness of the profanity filter.
- Introduced `filter_profanity` configuration option in `config.ini.example` to handle profanity in bridged messages: options include `drop` (default), `censor`, or `off`.
- Updated `requirements.txt` to include `better-profanity` for profanity filtering functionality.
- Enhanced `DiscordBridgeService` and `TelegramBridgeService` to implement the profanity filter logic, allowing messages with profanity to be dropped or censored before bridging.
- Updated documentation in `discord-bridge.md` and `telegram-bridge.md` to reflect the new profanity handling feature.
- Updated `MeshGraph` to support multi-resolution storage of edges, allowing prefixes of 2, 4, or 6 hex chars without truncation.
- Implemented prefix matching logic to ensure distinct links are maintained and accurately retrieved based on prefix queries.
- Refactored methods in `MessageHandler` and `PathCommand` to accommodate variable prefix lengths during graph lookups.
- Enhanced tests to validate prefix match functionality and edge management in the mesh graph.
- Commented out the 'help' command output in `config.ini.example` for clarity on overrides.
- Refactored prefix handling in `BotDataViewer` to use a dynamically retrieved `prefix_hex_chars` value instead of a hardcoded reference, improving flexibility and maintainability.
- Ensured consistent prefix handling across multiple instances in the `BotDataViewer` class.
- Updated the import statement for time zone handling in the `format_elapsed_display` function to use `timezone.utc` instead of `UTC` from `datetime`.
- Improved clarity and consistency in time zone management within the function.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Added `flood_scope` option to `[Channels]` section in `config.ini.example` for scoped flooding of channel messages.
- Updated `config-validation.md` to reflect the new `flood_scope` feature.
- Modified `CommandManager` to support optional flood scope during message sending, restoring global flood settings afterward.
- Removed unused imports from `core.py` related to meshcore-cli.
- Updated `_format_path_for_display` method to support multi-byte path formatting based on `bot.prefix_hex_chars`, ensuring compatibility with both legacy and new path formats.
- Improved handling of descriptive text in paths and added fallback logic for legacy paths.
- Added a synopsis format for path display in the stats command, providing a concise representation when message limits are exceeded.
- Updated translation files for multiple languages to include the new synopsis format for path data.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Update prefix command to accept BOTH legacy 2-char prefixes and
configured prefix_hex_chars (e.g. 4-char) during firmware transition
- Replace strict length validation with dual-length validation (2 or N)
- Ensure prefix lookups work with either input length via LIKE matching
- Update related SQL prefix extraction to use configured prefix length
- Add fallback handling in path parsing for legacy 2-char route data
Notes:
- This is an interim compatibility change to support mixed networks
where RF path data is still 1-byte while bot config may be 2-byte.
- Needs additional testing across real multi-hop scenarios and mixed
bot configurations.
- Translation updates are incomplete: only English strings were updated;
other translation files still need review.
- Behavior and UX may need refinement after real-world testing.
- Updated `config.ini.example` to include an `output_format` option for the trace command, allowing users to choose between inline and vertical formats for trace results.
- Enhanced the `TraceCommand` class to handle the new output format, implementing methods for both inline and vertical formatting of trace results.
- Improved documentation within the code to clarify the behavior of the new output format feature.
- Updated the trace command's description and help text to clarify the behavior of the `trace` and `tracer` commands regarding path handling.
- Modified the `_format_trace_result` method to include the sender's ID in the response for better context in trace results.
- Adjusted the logic in the `execute` method to ensure proper path handling for tracer commands and when no path is provided.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Introduced `[Earthquake_Service]` section in `config.ini.example` to enable earthquake alerts with customizable parameters such as polling interval, time window, and minimum magnitude.
- Updated `service-plugins.md` to include documentation for the new Earthquake Service, detailing its functionality and default settings.
- Introduced `[Earthquake_Service]` section in `config.ini.example` to enable earthquake alerts with customizable parameters such as polling interval, time window, and minimum magnitude.
- Updated `service-plugins.md` to include documentation for the new Earthquake Service, detailing its functionality and default settings.
- Introduced `[Trace_Command]` section in `config.ini.example` to enable link diagnostics with customizable parameters such as maximum hops, trace mode, and retry settings.
- Updated `command-reference.md` to include usage instructions and examples for the new `trace` and `tracer` commands, detailing their functionality for manual and round-trip tracing.
- Enhanced `mesh_graph.py` to support 2-byte trace confirmation for improved path weighting.
- Modified `message_handler.py` to update the mesh graph based on trace data, ensuring accurate edge representation in the graph.
- Added translations for the new trace commands in `en.json` to support user interaction in multiple languages.
- Updated SQLite connection timeout from 30 seconds to 60 seconds across multiple methods to reduce lock contention.
- Enabled Write-Ahead Logging (WAL) mode for better concurrent access between the bot and web viewer.
- Refactored packet insertion logic to include retry mechanism for handling locked database scenarios.
- Removed custom _DuplicateAwareConfigParser in favor of standard configparser.ConfigParser.
- Simplified configuration loading process while maintaining existing functionality.
- Added a new FAQ section addressing common issues when moving a database to a new install, including schema mismatches, stale operations, and timeout errors.
- Updated error logging in `feed_manager.py` and `scheduler.py` to use `logger.exception` for better traceback visibility during message queue and channel operation errors.
- Improved command help text generation in `command_manager.py` to ensure proper formatting and context filtering based on message length.
- Enhanced command availability checks in `cmd_command.py` and `help_command.py` to respect channel-specific overrides and improve user experience.
Felix Moessbauer