- Updated the MeshGraph class to prevent promoting a 1-byte edge to a 3-byte edge when the existing 1-byte edge lacks public keys, ensuring accurate observation attribution.
- Added a new test case to verify the behavior of edge promotion under specific conditions.
- Modified the BotDataViewer API to return the prefix length dynamically based on the edges, improving data consistency and user experience in the web viewer.
- Enhanced the mesh.html template to support displaying prefix byte counts, providing clearer information on node connections.
- Updated MeshGraph to support multi-resolution storage and node identity, allowing for better handling of 1-byte, 2-byte, and 3-byte edges.
- Implemented logic for merging and promoting edges based on observation counts and specificity.
- Enhanced BotDataViewer API to dynamically adjust prefix length based on configuration, improving data representation.
- Updated mesh.html template to display prefix byte counts, enhancing user experience with clearer information on node connections.
- Added support for bytes per hop in the BotDataViewer, allowing for better path data representation.
- Updated the contacts template to display bytes per hop and adjusted path formatting based on this value.
- Improved the decode path functionality to utilize the correct bytes per hop for decoding paths, enhancing overall path handling and user experience.
- Updated `MeshGraph` to support multi-resolution storage of edges, allowing prefixes of 2, 4, or 6 hex chars without truncation.
- Implemented prefix matching logic to ensure distinct links are maintained and accurately retrieved based on prefix queries.
- Refactored methods in `MessageHandler` and `PathCommand` to accommodate variable prefix lengths during graph lookups.
- Enhanced tests to validate prefix match functionality and edge management in the mesh graph.
- Commented out the 'help' command output in `config.ini.example` for clarity on overrides.
- Refactored prefix handling in `BotDataViewer` to use a dynamically retrieved `prefix_hex_chars` value instead of a hardcoded reference, improving flexibility and maintainability.
- Ensured consistent prefix handling across multiple instances in the `BotDataViewer` class.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Update prefix command to accept BOTH legacy 2-char prefixes and
configured prefix_hex_chars (e.g. 4-char) during firmware transition
- Replace strict length validation with dual-length validation (2 or N)
- Ensure prefix lookups work with either input length via LIKE matching
- Update related SQL prefix extraction to use configured prefix length
- Add fallback handling in path parsing for legacy 2-char route data
Notes:
- This is an interim compatibility change to support mixed networks
where RF path data is still 1-byte while bot config may be 2-byte.
- Needs additional testing across real multi-hop scenarios and mixed
bot configurations.
- Translation updates are incomplete: only English strings were updated;
other translation files still need review.
- Behavior and UX may need refinement after real-world testing.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Updated SQLite connection timeout from 30 seconds to 60 seconds across multiple methods to reduce lock contention.
- Enabled Write-Ahead Logging (WAL) mode for better concurrent access between the bot and web viewer.
- Refactored packet insertion logic to include retry mechanism for handling locked database scenarios.
- Replaced the `abbreviate_noaa` method calls with a new `_noaa_period_display_name` method to improve the display of forecast period names, including handling federal holidays.
- Updated the `BotDataViewer` class to include the bot name in the returned configuration dictionary, ensuring consistent naming across the application.
- Made minor adjustments to HTML templates for better layout and added footer links for project visibility.
- Introduced a new method to retrieve the database path, allowing for a fallback to the [Bot] section if the [Web_Viewer] db_path is unset.
- Updated various parts of the codebase to utilize this new method, ensuring consistent database path resolution across the application.
- Enhanced the configuration example to clarify the optional nature of the db_path setting for the web viewer.
- Improved logic for determining the database path by prioritizing the [Web_Viewer] section and falling back to the [Bot] section if necessary.
- Updated the packet stream initialization to use the resolved database path from the constructor, ensuring consistency across the application.
- Updated the configuration handling for the web viewer to use the [Bot] db_path when [Web_Viewer] db_path is unset, ensuring consistent database access.
- Added warnings in the logging to notify users when the web viewer database path differs from the bot database path, guiding them to configure it correctly.
- Refactored database path resolution logic across multiple modules for better maintainability and clarity.
- Added error handling in the template context processor to ensure templates render without raising exceptions, logging any failures.
- Enhanced the configuration path resolution to support relative paths when starting the web viewer as a subprocess, improving flexibility.
- Implemented logging for unhandled exceptions in Flask routes to aid in debugging and provide clearer error reporting.
- Updated the API endpoint to accept an optional query parameter `since` for filtering contact data based on the time frame (24h, 7d, 30d, 90d, or all).
- Modified the `_get_tracking_data` method to implement filtering logic based on the `since` parameter, improving performance by limiting the data retrieved.
- Enhanced the contacts HTML template to include a dropdown for selecting the time frame, with the default set to the last 30 days.
- Implemented local storage functionality to remember the user's selected time frame across sessions.
- Changed the default database path in config.ini.example and related files from 'bot_data.db' to 'meshcore_bot.db' for consistency across modules.
- Adjusted final hop proximity scoring parameters in config.ini.example to reflect updated distance normalization and thresholds, enhancing scoring accuracy for LoRa networks.
- Updated documentation to clarify the shared database usage between the bot and web viewer.
- Moved the assignment of `db_path` to be loaded from the configuration file, enhancing flexibility and maintainability of the database path management in the BotDataViewer class.
- Added new configuration options for path selection presets and proximity methods in config.ini.example, allowing users to customize routing behavior.
- Implemented a new database table for storing observed paths, enabling better tracking of paths from advertisements and messages.
- Updated MessageHandler to store complete paths in the observed_paths table, improving path validation and selection accuracy.
- Enhanced the PathCommand class to utilize new proximity and recency settings, optimizing repeater selection based on user-defined criteria.
- Improved web viewer functionality to display multiple paths for contacts, enhancing user experience and interaction with path data.
- Added a new mesh graph feature for improved path validation, allowing for enhanced routing accuracy.
- Introduced configuration options for recency decay half-life and graph-based validation settings in config.ini.example.
- Updated the PathCommand class to utilize graph-based selection methods, combining graph and geographic scores for better repeater selection.
- Implemented new methods in MessageHandler to update the mesh graph with advertisement paths and trace packet data.
- Created a new database table for mesh connections to support graph-based path validation.
- Enhanced web viewer integration to display mesh graph updates in real-time, improving user interaction and monitoring capabilities.
- Updated `send_dm` and `send_channel_message` methods to include an optional `command_id` for tracking message repeats.
- Integrated transmission tracking to record and manage repeat messages, improving message handling and response accuracy.
- Enhanced `capture_command` method in `BotIntegration` to store repeat information for better analysis in the web viewer.
- Added favicon support and improved web viewer templates to display repeat information effectively.
- Implemented JavaScript updates in the web viewer to handle command updates and display repeat counts dynamically.
- Updated `config.ini.example` to include a new option for additional hashtag channels to decode in the packet stream.
- Modified `BotDataViewer` to retrieve and display additional decode-only channels from the configuration.
- Improved packet handling in `message_handler.py` to capture full packet data for web viewer integration.
- Enhanced the web viewer's JavaScript to support detailed packet analysis and display, including color-coded hex breakdowns and improved user interface elements.
- Added new styles and scripts to the web viewer templates for better visual representation of packet data and improved user experience.
- Added a rotating file handler for logging to manage log file size and backups.
- Implemented thread safety for connected clients using a lock to prevent race conditions.
- Introduced a method to clean up stale clients based on inactivity, improving resource management.
- Updated logging messages to reflect changes in logging setup and client connection status.
- Enhanced error logging to include stack traces for better debugging.
- Optimized the cleanup of old packet stream data by implementing smaller batch deletions to reduce lock contention.
- Adjusted database connection settings for improved concurrency and added handling for database busy errors.
- Removed periodic cleanup from BotIntegration, delegating this responsibility to the web viewer subprocess to avoid contention.
- Refactored bot's main execution flow to use asyncio.run() with improved signal handling for graceful shutdown on Unix systems.
- Added new dependencies in pyproject.toml: urllib3, paho-mqtt, cryptography, and pynacl for enhanced functionality.
- Updated Nix flake to include flake-parts for better modularization and added translation path for NixOS module compatibility.
- Improved argument parsing in web viewer to maintain clarity and consistency.
This commit adds comprehensive Nix/NixOS support and modernizes the
Python packaging structure to enable declarative system-wide deployments.
Added complete Nix flake infrastructure in nix/:
- packages.nix: Package definition with all dependencies
- nixos-module.nix: NixOS module for declarative service configuration
- nixos-test.nix: Comprehensive NixOS VM tests (basic + web viewer)
- shell.nix: Development shell with all build tools
The NixOS module (services.meshcore-bot) provides:
- Automatic user/group creation (meshcore-bot:meshcore-bot)
- Serial port access (dialout group membership)
- Systemd service with security hardening
- Automatic directory management via StateDirectory/LogsDirectory
- INI config generation from Nix attribute sets
- Sensible defaults for system paths
Example NixOS configuration:
services.meshcore-bot = {
enable = true;
settings = {
Connection.connection_type = "serial";
Connection.serial_port = "/dev/ttyUSB0";
Bot.bot_name = "MyBot";
};
};
Migrated from requirements.txt to pyproject.toml:
- Defined proper package metadata and dependencies
- Created entry points: meshcore-bot, meshcore-viewer
- Enables standard 'pip install -e .' workflow
- Maintains compatibility with existing setups
Changes to existing code are minimal and only where necessary:
1. **Added --config parameter** (meshcore_bot.py, app.py)
- Wrapped main logic in main() function for entry point
- Added argparse to accept --config parameter
- Required for NixOS to pass generated config from /nix/store
- Backwards compatible: defaults to 'config.ini' in current directory
- No changes to core bot logic
2. **Fixed web viewer subprocess spawning** (modules/web_viewer/integration.py)
- Changed from sys.executable to 'meshcore-viewer' entry point
- Ensures Nix wrapper script sets up correct PYTHONPATH
- Resolves Flask module import issues in Nix environment
- No functional change for traditional installations
Note: The dev branch's resolve_path() utility already handles both
relative and absolute paths correctly, making it compatible with
NixOS system paths (/var/lib, /var/log) without additional changes.
All NixOS VM tests passing:
- nixos-module-basic: Service startup, file creation, TCP connection
- nixos-module-webviewer: Flask available, port listening, HTTP responses
No breaking changes to existing workflows. The bot continues to work
exactly as before when run with traditional Python methods.
- Enhanced the handle_connect, handle_disconnect, handle_subscribe_commands, handle_subscribe_packets, and handle_ping methods to include better error handling and logging.
- Added checks for client_id to prevent issues during connection and disconnection events.
- Implemented database file existence and accessibility checks in the polling loop, with detailed logging for errors.
- Improved handling of database connection errors, including exponential backoff for persistent issues, ensuring more robust database interactions.
- Implemented periodic updates of system health status to the database every 30 seconds.
- Added a new method to aggregate and return comprehensive health status of all components, including core connection, database, services, and web viewer.
- Introduced an API endpoint at /api/system-health to retrieve system health data, providing a structured response for health status and component details.
- Enhanced error handling for health data storage and retrieval processes, ensuring robust logging and feedback in case of issues.
- Simplified database connection management by using context managers to ensure connections are properly closed.
- Enhanced error handling during MQTT client disconnection in packet capture service, logging specific exceptions.
- Updated message handling in MessageHandler to capture command data for web viewer integration, improving response tracking.
- Replaced the validate_safe_path function with a new resolve_path utility to simplify database path resolution in BotDataViewer, BotIntegration, and MapUploaderService.
- Updated the logic to ensure that both relative and absolute paths are handled correctly, enhancing the robustness of database connections.
- Improved code readability and maintainability by centralizing path resolution logic.
- Centralized placeholder handling in utils instead of individual function handlers
- Added db_path configuration options in config.ini.example for both the main bot and web viewer
- Updated BotDataViewer and BotIntegration classes to retrieve the database path from the new Web_Viewer section
- Set a timeout of 30 seconds for SQLite connections in MessageScheduler and BotIntegration to improve reliability.
- Added validation for the database path in BotIntegration, ensuring it is safe and relative to the bot root, enhancing robustness.
- Introduced a bot_root attribute in BotDataViewer for path validation.
- Added validation for the database path using the validate_safe_path function to ensure paths are safe and relative to the bot root.
- Enhanced error handling to log warnings when invalid paths are detected, defaulting to 'bot_data.db' if necessary.
- This change improves the robustness of database connections across the application.
- Updated MessageHandler to extract and store path information from packet_info and routing_info, improving data tracking.
- Added a new API endpoint in the web viewer for decoding path hex strings to repeater names.
- Enhanced the contacts template to display path information with tooltips, improving user experience.
- Implemented tooltip functionality for path data in the web viewer, allowing users to view detailed repeater information on hover.
