Updated the BotDataViewer class to utilize a context manager for database connections, enhancing resource management. Additionally, refactored test files to implement a centralized approach for managing SQLite connections, ensuring proper cleanup after tests. This change improves code maintainability and reliability across the application.
Make outbound send suppression consistent by honoring both radio-offline and zombie states across command-manager and scheduler paths. Preserve strict SSRF defaults while adding explicit private-feed URL opt-ins, persist allow_local_smtp from notifications config writes, reconcile zombie alert setting precedence, and replace deprecated UTC timestamp calls with timezone-aware UTC usage.
Moved radio-related configuration options from the [Bot] section to a new [Connection] section in config.ini.example for better organization. Updated the core logic to reference the new configuration paths, ensuring that radio health monitoring and alert settings are correctly handled. This change enhances clarity and maintainability of the configuration structure.
Updated the path validation logic in security_utils.py and web_viewer/app.py to include additional dangerous prefixes, specifically targeting private directories. This change aims to strengthen security by preventing access to sensitive system paths. Additionally, modified the test for posting feeds to mock the external URL validation, ensuring consistent behavior during tests.
Clean up residual cherry-pick conflict markers and keep SMTP guidance in config templates brief while preserving full behavior in code and tests.
Made-with: Cursor
Two incompatibilities surfaced after rebasing onto upstream/dev in code
introduced by #147:
- tests/integration/test_flood_scope_reply.py: add bot.is_radio_zombie = False
mock; the zombie-detection guard added to send_channel_message in #147
causes the call to short-circuit without it, failing the scope assertion;
also removes unused `call` import
- modules/web_viewer/app.py: replace ~50-line inline _get_version_info()
with 5-line delegate to resolve_runtime_version(); removes now-unnecessary
import subprocess; the resolve_runtime_version import is now actually used
Do not merge until #147 is merged.
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
- base.html: persistent danger banner appears on every web page when
is_radio_zombie is true; shows datetime zombie was detected; includes
"Restart Bot Processing" button (POST /api/admin/zombie-recover) that
clears _radio_zombie_detected and _radio_fail_count on the live bot
object and removes the persisted DB flag; banner turns green on success
- config.html: new "Zombie Radio Alert" card with enable/disable toggle
and alert-email field; "Save" writes to bot_metadata (immediate,
survives restarts); "Save to config.ini" also persists values to
config.ini and keeps the in-memory config in sync; card shows
current config.ini values as baseline defaults
- app.py: inject_template_vars context processor now provides
radio_zombie and radio_zombie_since to all templates; added
GET/POST /api/config/zombie-alert endpoints (GET returns both
bot_metadata and config.ini values; POST supports write_to_config
flag); added POST /api/admin/zombie-recover endpoint; stored
config_path on self for write-back use
- scheduler.py: send_zombie_alert_email now prefers bot_metadata
(zombie.alert_enabled, zombie.alert_email) over config.ini so web
UI changes take effect without a restart; uses isinstance(..., str)
guard so mock/None values safely fall through to config.ini defaults
- Updated `app.py` to ensure the project root is correctly added to `sys.path` when the script is executed directly, allowing for proper module imports.
- Removed redundant code that previously handled project root path insertion, streamlining the import process.
- Updated `config.ini.example` to clarify password requirements for web viewer authentication, emphasizing the need for a password when binding to non-loopback interfaces.
- Introduced a new function `normalized_web_viewer_password` in `integration.py` to standardize password retrieval and validation, handling various empty and null placeholder cases.
- Enhanced error logging in `core.py` to use `logger.error` for web viewer integration failures, improving visibility of issues.
- Modified `app.py` to utilize the new password normalization function, ensuring consistent password handling across the application.
- Added tests in `test_web_viewer_integration.py` to validate password normalization and error logging behavior when the web viewer is configured without a password.
- Increased the per-user rate limit from 5 to 30 seconds across multiple configuration files to reduce response frequency.
- Added the version command to the configuration examples and updated help text to include the new command.
- Refactored version information retrieval in the bot and web viewer to utilize a shared runtime resolver for consistency.
- Improved documentation in README.md to reflect changes in commands and configuration options.
- Added calculations for contacts and incoming packets with multibyte path evidence over the last 7 days in `app.py`, improving data accuracy.
- Introduced new methods for handling multibyte path chunks and counting packets from JSON data, enhancing backend functionality.
- Updated `contacts.html` and `index.html` templates to display multibyte path encoding badges and tooltips, improving user interface clarity.
- Enhanced CSS for path encoding badges to differentiate between multibyte and one-byte paths, ensuring better visual representation.
These changes improve the overall user experience and data representation in the Bot Data Viewer.
- Updated type hints to use `|` for union types instead of `Optional` where applicable, enhancing readability and consistency.
- Cleaned up import statements by removing unnecessary imports and organizing them for better clarity.
- Adjusted function signatures in various modules to reflect the new type hinting style, improving type safety and code maintainability.
These changes contribute to a more modern and consistent codebase, aligning with current Python typing practices.
- Added a regex pattern to validate SQLite column definitions, preventing SQL injection in the _add_column() function.
- Introduced a new validation function to ensure safe column definitions are used.
- Updated repeater_manager.py to use list() for iterating over contacts, ensuring compatibility with potential changes in the underlying data structure.
- Enhanced error handling in the MessageScheduler for better debugging during scheduler shutdown and message sending.
These changes improve the security and reliability of database operations and enhance the robustness of the repeater management system.
- Added a function to strip ANSI color codes from log lines for better display in SocketIO web clients, improving log readability.
- Implemented dark mode styles for dropdown menus and other UI components to enhance user experience in dark theme.
- Updated the contacts template to include a new overflow menu for additional actions, improving accessibility and usability.
- Enhanced the login page with a more visually appealing layout and improved theme handling to prevent flash of unstyled content.
- Refined log level toggles in the logs template for better user interaction and visibility of log levels.
These changes improve the overall functionality and aesthetics of the web viewer.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Introduced validation for SQLite journal modes in DBManager, defaulting to "WAL" for invalid inputs and logging warnings.
- Added a new utility function in security_utils for validating SQL identifiers to prevent SQL injection.
- Updated BotDataViewer to utilize the new journal mode validation and SQL identifier checks, ensuring safer database operations.
- Enhanced test coverage for restore functionality, including checks for backup directory configuration and path traversal prevention.
- Introduced a new `_utc_now` function in `maintenance.py` to standardize UTC time retrieval, replacing deprecated `utcnow()` calls.
- Updated all instances of `datetime.utcnow()` in `maintenance.py` to use the new `_utc_now` function for consistency.
- Modified `app.py` in the web viewer to use `datetime.now(timezone.utc)` instead of `datetime.utcnow()`, ensuring timezone awareness.
- Enhanced MQTT test suite with new v2 callback methods for improved connection handling and error reporting.
- Introduced `{field|auto}` placeholder in message formats to fill remaining characters up to `max_message_length`, improving message customization.
- Implemented logic in `FeedManager` to handle multiple `{field|auto}` placeholders, logging a warning if more than one is present.
- Updated `BotDataViewer` to utilize the new auto field feature, ensuring compatibility with existing message formatting.
- Added unit tests to validate the behavior of the new auto field functionality, including handling of message length constraints and multiple placeholders.
- Added configuration options for URL shortening in `config.ini.example` and updated documentation in `FEEDS.md`.
- Enhanced `FeedManager` to support URL shortening based on new settings, allowing for both global and per-link shortening.
- Refactored message formatting logic to incorporate URL shortening features, ensuring compatibility with existing link handling.
- Introduced new utility functions for encoding path length bytes in `utils.py`, improving path management in message handling.
- Added unit tests to validate the new URL shortening functionality and ensure proper behavior under various conditions.
- Introduced `within_days` and `within_weeks` operators for filtering items based on their timestamps, allowing for more dynamic date range conditions.
- Updated documentation in `FEEDS.md` to include examples and usage of the new operators.
- Refactored filter evaluation logic in `FeedManager` and `BotDataViewer` to utilize a shared function for condition checking, improving code maintainability.
- Added unit tests to validate the new date-based filtering functionality.
- Updated the web viewer documentation to replace the cache data section with a new config panel overview, enhancing clarity on configuration options.
- Removed the legacy cache management route, redirecting to the new configuration panel for better user experience.
- Enhanced the configuration page to include structured settings with categorized topics and database tools, improving navigation and usability.
- Updated tests to reflect changes in cache management and ensure proper functionality of the new configuration routes.
- Introduced a new `maintenance` module to handle data retention, log rotation, and nightly email tasks.
- Updated the `scheduler` to utilize the `MaintenanceRunner` for executing maintenance tasks, improving code organization and clarity.
- Enhanced documentation to reflect changes in logging configuration and data retention processes.
- Adjusted tests to accommodate the refactored scheduler methods and ensure proper functionality.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
Auth (BUG-001):
- Optional password via web_viewer_password in [Web_Viewer]; /login and
/logout; Flask session guard on all routes and SocketIO handlers
Contact management and export:
- Star contacts of any type; purge-preview + purge inactive contacts
- GET /api/export/contacts and /api/export/paths: CSV/JSON with time-range
Config tab and maintenance:
- /config page: SMTP, log rotation, DB backup settings in bot_metadata
- Nightly email digest (uptime, contacts, DB size, log errors); SMTP
timeout=30s; pre-rotation log attachment hook
- GET /api/maintenance/status: Maintenance Status card
DB backup, restore, and purge:
- POST /api/maintenance/backup_now; GET /api/maintenance/list_backups;
POST /api/maintenance/restore (SQLite magic-byte validation)
- POST /api/maintenance/purge: remove rows older than threshold
- Scheduled backups: daily/weekly/manual with retention pruning
- Config save validates db_backup_dir exists; 400 on missing path
Live streaming and realtime monitoring:
- Live Activity panel: colour-coded SocketIO feed with pause/clear
- capture_channel_message() feeds packet_stream; message_data event
- /realtime page: three independent stream panels; [#channel] prefix
- /logs page: subscribe_logs/log_line; log-tail thread; level colouring
- History replay: last 50/50/200 items on connect
- Werkzeug 3.1 WebSocket fix: _apply_werkzeug_websocket_fix()
- BUG-029: db_path resolved via config_base = Path(config_path).parent;
stored as self._config_base; dead _get_db_path() removed
Scroll/filter controls and connected agents:
- Scroll-to-top/bottom on Live Activity and all realtime panels
- Type-filter checkboxes (Packets/Commands/Messages) with applyFilters()
- GET /api/connected_clients: agent count clickable; Bootstrap modal
- Updated the MeshGraph class to prevent promoting a 1-byte edge to a 3-byte edge when the existing 1-byte edge lacks public keys, ensuring accurate observation attribution.
- Added a new test case to verify the behavior of edge promotion under specific conditions.
- Modified the BotDataViewer API to return the prefix length dynamically based on the edges, improving data consistency and user experience in the web viewer.
- Enhanced the mesh.html template to support displaying prefix byte counts, providing clearer information on node connections.
- Updated MeshGraph to support multi-resolution storage and node identity, allowing for better handling of 1-byte, 2-byte, and 3-byte edges.
- Implemented logic for merging and promoting edges based on observation counts and specificity.
- Enhanced BotDataViewer API to dynamically adjust prefix length based on configuration, improving data representation.
- Updated mesh.html template to display prefix byte counts, enhancing user experience with clearer information on node connections.
- Added support for bytes per hop in the BotDataViewer, allowing for better path data representation.
- Updated the contacts template to display bytes per hop and adjusted path formatting based on this value.
- Improved the decode path functionality to utilize the correct bytes per hop for decoding paths, enhancing overall path handling and user experience.
- Updated `MeshGraph` to support multi-resolution storage of edges, allowing prefixes of 2, 4, or 6 hex chars without truncation.
- Implemented prefix matching logic to ensure distinct links are maintained and accurately retrieved based on prefix queries.
- Refactored methods in `MessageHandler` and `PathCommand` to accommodate variable prefix lengths during graph lookups.
- Enhanced tests to validate prefix match functionality and edge management in the mesh graph.
- Commented out the 'help' command output in `config.ini.example` for clarity on overrides.
- Refactored prefix handling in `BotDataViewer` to use a dynamically retrieved `prefix_hex_chars` value instead of a hardcoded reference, improving flexibility and maintainability.
- Ensured consistent prefix handling across multiple instances in the `BotDataViewer` class.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Update prefix command to accept BOTH legacy 2-char prefixes and
configured prefix_hex_chars (e.g. 4-char) during firmware transition
- Replace strict length validation with dual-length validation (2 or N)
- Ensure prefix lookups work with either input length via LIKE matching
- Update related SQL prefix extraction to use configured prefix length
- Add fallback handling in path parsing for legacy 2-char route data
Notes:
- This is an interim compatibility change to support mixed networks
where RF path data is still 1-byte while bot config may be 2-byte.
- Needs additional testing across real multi-hop scenarios and mixed
bot configurations.
- Translation updates are incomplete: only English strings were updated;
other translation files still need review.
- Behavior and UX may need refinement after real-world testing.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Updated SQLite connection timeout from 30 seconds to 60 seconds across multiple methods to reduce lock contention.
- Enabled Write-Ahead Logging (WAL) mode for better concurrent access between the bot and web viewer.
- Refactored packet insertion logic to include retry mechanism for handling locked database scenarios.
- Replaced the `abbreviate_noaa` method calls with a new `_noaa_period_display_name` method to improve the display of forecast period names, including handling federal holidays.
- Updated the `BotDataViewer` class to include the bot name in the returned configuration dictionary, ensuring consistent naming across the application.
- Made minor adjustments to HTML templates for better layout and added footer links for project visibility.
- Introduced a new method to retrieve the database path, allowing for a fallback to the [Bot] section if the [Web_Viewer] db_path is unset.
- Updated various parts of the codebase to utilize this new method, ensuring consistent database path resolution across the application.
- Enhanced the configuration example to clarify the optional nature of the db_path setting for the web viewer.
- Improved logic for determining the database path by prioritizing the [Web_Viewer] section and falling back to the [Bot] section if necessary.
- Updated the packet stream initialization to use the resolved database path from the constructor, ensuring consistency across the application.
- Updated the configuration handling for the web viewer to use the [Bot] db_path when [Web_Viewer] db_path is unset, ensuring consistent database access.
- Added warnings in the logging to notify users when the web viewer database path differs from the bot database path, guiding them to configure it correctly.
- Refactored database path resolution logic across multiple modules for better maintainability and clarity.
- Added error handling in the template context processor to ensure templates render without raising exceptions, logging any failures.
- Enhanced the configuration path resolution to support relative paths when starting the web viewer as a subprocess, improving flexibility.
- Implemented logging for unhandled exceptions in Flask routes to aid in debugging and provide clearer error reporting.
- Updated the API endpoint to accept an optional query parameter `since` for filtering contact data based on the time frame (24h, 7d, 30d, 90d, or all).
- Modified the `_get_tracking_data` method to implement filtering logic based on the `since` parameter, improving performance by limiting the data retrieved.
- Enhanced the contacts HTML template to include a dropdown for selecting the time frame, with the default set to the last 30 days.
- Implemented local storage functionality to remember the user's selected time frame across sessions.
Stacy Olivas