- Changed single quotes to HTML entities in JSON placeholders within the feeds.html file to ensure proper rendering.
- Updated the assignment of PREFIX_HEX_CHARS in mesh.html to parse the value as an integer, enhancing type safety and clarity.
These changes enhance the user interface and code maintainability in the web viewer templates.
- Updated type hints to use `|` for union types instead of `Optional` where applicable, enhancing readability and consistency.
- Cleaned up import statements by removing unnecessary imports and organizing them for better clarity.
- Adjusted function signatures in various modules to reflect the new type hinting style, improving type safety and code maintainability.
These changes contribute to a more modern and consistent codebase, aligning with current Python typing practices.
- Added a regex pattern to validate SQLite column definitions, preventing SQL injection in the _add_column() function.
- Introduced a new validation function to ensure safe column definitions are used.
- Updated repeater_manager.py to use list() for iterating over contacts, ensuring compatibility with potential changes in the underlying data structure.
- Enhanced error handling in the MessageScheduler for better debugging during scheduler shutdown and message sending.
These changes improve the security and reliability of database operations and enhance the robustness of the repeater management system.
- Introduced a responsive toolbar for filtering and searching contacts, improving usability on mobile devices.
- Added a dropdown for timespan selection, allowing users to filter contacts based on different timeframes.
- Updated styles to ensure consistent alignment and height for toolbar elements, enhancing the overall appearance and functionality.
These changes improve the user experience for managing contacts in the web viewer.
- Introduced a mobile-friendly layout for the contacts section, including a new mobile stack for displaying contact cards.
- Added sorting options for contacts on mobile devices, allowing users to sort by various criteria such as name, device type, and distance.
- Implemented a select-all checkbox for mobile view to improve user interaction with contact selection.
- Updated event listeners to handle changes in selection and sorting, ensuring a seamless experience across devices.
These changes improve usability and accessibility for mobile users in the web viewer.
- Added a function to strip ANSI color codes from log lines for better display in SocketIO web clients, improving log readability.
- Implemented dark mode styles for dropdown menus and other UI components to enhance user experience in dark theme.
- Updated the contacts template to include a new overflow menu for additional actions, improving accessibility and usability.
- Enhanced the login page with a more visually appealing layout and improved theme handling to prevent flash of unstyled content.
- Refined log level toggles in the logs template for better user interaction and visibility of log levels.
These changes improve the overall functionality and aesthetics of the web viewer.
- Added 'X-Requested-With' header to various API requests in channel_operations.js, cache.html, config.html, contacts.html, feeds.html, greeter.html, mesh.html, radio.html, and other templates to improve request handling and prevent potential issues with cross-origin requests.
- Ensured consistent header usage across all relevant fetch calls to enhance security and compatibility.
These changes improve the robustness of API interactions within the web viewer.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Introduced validation for SQLite journal modes in DBManager, defaulting to "WAL" for invalid inputs and logging warnings.
- Added a new utility function in security_utils for validating SQL identifiers to prevent SQL injection.
- Updated BotDataViewer to utilize the new journal mode validation and SQL identifier checks, ensuring safer database operations.
- Enhanced test coverage for restore functionality, including checks for backup directory configuration and path traversal prevention.
- Added new CSS styles for live message metadata badges and layout adjustments for better visual organization.
- Reorganized the HTML structure of the real-time message and packet stream sections for improved responsiveness and clarity.
- Updated JavaScript functions to support new channel key normalization and hashing for enhanced message handling.
- Modified test cases to reflect changes in message entry templates and ensure proper functionality of new features.
- Introduced a new `_utc_now` function in `maintenance.py` to standardize UTC time retrieval, replacing deprecated `utcnow()` calls.
- Updated all instances of `datetime.utcnow()` in `maintenance.py` to use the new `_utc_now` function for consistency.
- Modified `app.py` in the web viewer to use `datetime.now(timezone.utc)` instead of `datetime.utcnow()`, ensuring timezone awareness.
- Enhanced MQTT test suite with new v2 callback methods for improved connection handling and error reporting.
- Introduced `{field|auto}` placeholder in message formats to fill remaining characters up to `max_message_length`, improving message customization.
- Implemented logic in `FeedManager` to handle multiple `{field|auto}` placeholders, logging a warning if more than one is present.
- Updated `BotDataViewer` to utilize the new auto field feature, ensuring compatibility with existing message formatting.
- Added unit tests to validate the behavior of the new auto field functionality, including handling of message length constraints and multiple placeholders.
- Added configuration options for URL shortening in `config.ini.example` and updated documentation in `FEEDS.md`.
- Enhanced `FeedManager` to support URL shortening based on new settings, allowing for both global and per-link shortening.
- Refactored message formatting logic to incorporate URL shortening features, ensuring compatibility with existing link handling.
- Introduced new utility functions for encoding path length bytes in `utils.py`, improving path management in message handling.
- Added unit tests to validate the new URL shortening functionality and ensure proper behavior under various conditions.
- Updated `pyproject.toml` to include JavaScript files for the web viewer.
- Added a new script reference in `base.html` for channel operations.
- Improved the channel creation process in `feeds.html` with enhanced UI elements and error handling.
- Refactored channel index retrieval in `radio.html` to utilize a centralized method for better maintainability.
- Implemented asynchronous channel statistics loading to improve responsiveness during channel operations.
- Introduced `within_days` and `within_weeks` operators for filtering items based on their timestamps, allowing for more dynamic date range conditions.
- Updated documentation in `FEEDS.md` to include examples and usage of the new operators.
- Refactored filter evaluation logic in `FeedManager` and `BotDataViewer` to utilize a shared function for condition checking, improving code maintainability.
- Added unit tests to validate the new date-based filtering functionality.
- Updated the web viewer documentation to replace the cache data section with a new config panel overview, enhancing clarity on configuration options.
- Removed the legacy cache management route, redirecting to the new configuration panel for better user experience.
- Enhanced the configuration page to include structured settings with categorized topics and database tools, improving navigation and usability.
- Updated tests to reflect changes in cache management and ensure proper functionality of the new configuration routes.
- Introduced a new `maintenance` module to handle data retention, log rotation, and nightly email tasks.
- Updated the `scheduler` to utilize the `MaintenanceRunner` for executing maintenance tasks, improving code organization and clarity.
- Enhanced documentation to reflect changes in logging configuration and data retention processes.
- Adjusted tests to accommodate the refactored scheduler methods and ensure proper functionality.
- Expanded package data in `pyproject.toml` to include additional static files for the web viewer.
- Removed testing dependencies from `requirements.txt`, noting that they are now included in pyproject extras.
- Updated the base HTML template to reference the manifest from the Flask static folder.
- Modified the build script to exclude `.cursor` files and updated documentation URLs to the correct repository.
- Changed the installation command in the build script to use `requirements.txt` instead of the previous method.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
Auth (BUG-001):
- Optional password via web_viewer_password in [Web_Viewer]; /login and
/logout; Flask session guard on all routes and SocketIO handlers
Contact management and export:
- Star contacts of any type; purge-preview + purge inactive contacts
- GET /api/export/contacts and /api/export/paths: CSV/JSON with time-range
Config tab and maintenance:
- /config page: SMTP, log rotation, DB backup settings in bot_metadata
- Nightly email digest (uptime, contacts, DB size, log errors); SMTP
timeout=30s; pre-rotation log attachment hook
- GET /api/maintenance/status: Maintenance Status card
DB backup, restore, and purge:
- POST /api/maintenance/backup_now; GET /api/maintenance/list_backups;
POST /api/maintenance/restore (SQLite magic-byte validation)
- POST /api/maintenance/purge: remove rows older than threshold
- Scheduled backups: daily/weekly/manual with retention pruning
- Config save validates db_backup_dir exists; 400 on missing path
Live streaming and realtime monitoring:
- Live Activity panel: colour-coded SocketIO feed with pause/clear
- capture_channel_message() feeds packet_stream; message_data event
- /realtime page: three independent stream panels; [#channel] prefix
- /logs page: subscribe_logs/log_line; log-tail thread; level colouring
- History replay: last 50/50/200 items on connect
- Werkzeug 3.1 WebSocket fix: _apply_werkzeug_websocket_fix()
- BUG-029: db_path resolved via config_base = Path(config_path).parent;
stored as self._config_base; dead _get_db_path() removed
Scroll/filter controls and connected agents:
- Scroll-to-top/bottom on Live Activity and all realtime panels
- Type-filter checkboxes (Packets/Commands/Messages) with applyFilters()
- GET /api/connected_clients: agent count clickable; Bootstrap modal
- Updated the MeshGraph class to prevent promoting a 1-byte edge to a 3-byte edge when the existing 1-byte edge lacks public keys, ensuring accurate observation attribution.
- Added a new test case to verify the behavior of edge promotion under specific conditions.
- Modified the BotDataViewer API to return the prefix length dynamically based on the edges, improving data consistency and user experience in the web viewer.
- Enhanced the mesh.html template to support displaying prefix byte counts, providing clearer information on node connections.
- Updated MeshGraph to support multi-resolution storage and node identity, allowing for better handling of 1-byte, 2-byte, and 3-byte edges.
- Implemented logic for merging and promoting edges based on observation counts and specificity.
- Enhanced BotDataViewer API to dynamically adjust prefix length based on configuration, improving data representation.
- Updated mesh.html template to display prefix byte counts, enhancing user experience with clearer information on node connections.
- Added support for bytes per hop in the BotDataViewer, allowing for better path data representation.
- Updated the contacts template to display bytes per hop and adjusted path formatting based on this value.
- Improved the decode path functionality to utilize the correct bytes per hop for decoding paths, enhancing overall path handling and user experience.
- Updated `MeshGraph` to support multi-resolution storage of edges, allowing prefixes of 2, 4, or 6 hex chars without truncation.
- Implemented prefix matching logic to ensure distinct links are maintained and accurately retrieved based on prefix queries.
- Refactored methods in `MessageHandler` and `PathCommand` to accommodate variable prefix lengths during graph lookups.
- Enhanced tests to validate prefix match functionality and edge management in the mesh graph.
- Commented out the 'help' command output in `config.ini.example` for clarity on overrides.
- Refactored prefix handling in `BotDataViewer` to use a dynamically retrieved `prefix_hex_chars` value instead of a hardcoded reference, improving flexibility and maintainability.
- Ensured consistent prefix handling across multiple instances in the `BotDataViewer` class.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Update prefix command to accept BOTH legacy 2-char prefixes and
configured prefix_hex_chars (e.g. 4-char) during firmware transition
- Replace strict length validation with dual-length validation (2 or N)
- Ensure prefix lookups work with either input length via LIKE matching
- Update related SQL prefix extraction to use configured prefix length
- Add fallback handling in path parsing for legacy 2-char route data
Notes:
- This is an interim compatibility change to support mixed networks
where RF path data is still 1-byte while bot config may be 2-byte.
- Needs additional testing across real multi-hop scenarios and mixed
bot configurations.
- Translation updates are incomplete: only English strings were updated;
other translation files still need review.
- Behavior and UX may need refinement after real-world testing.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Updated SQLite connection timeout from 30 seconds to 60 seconds across multiple methods to reduce lock contention.
- Enabled Write-Ahead Logging (WAL) mode for better concurrent access between the bot and web viewer.
- Refactored packet insertion logic to include retry mechanism for handling locked database scenarios.
- Added a new FAQ section addressing common issues when moving a database to a new install, including schema mismatches, stale operations, and timeout errors.
- Updated error logging in `feed_manager.py` and `scheduler.py` to use `logger.exception` for better traceback visibility during message queue and channel operation errors.
- Improved command help text generation in `command_manager.py` to ensure proper formatting and context filtering based on message length.
- Enhanced command availability checks in `cmd_command.py` and `help_command.py` to respect channel-specific overrides and improve user experience.
- Introduced a circuit breaker mechanism to manage consecutive connection failures when sending updates to the web viewer, preventing log flooding during outages.
- Added methods to track send attempts and determine when to skip sending based on the circuit breaker state.
- Updated logging to provide feedback on circuit status and failures, improving troubleshooting capabilities.
- Refined message handling to ensure efficient communication with the web viewer, reducing unnecessary requests during downtime.
- Changed the return type of the `db_path` method in the `BotIntegration` class from a Path object to a string, ensuring consistent data type handling for database paths.
- Shortened the title in the mesh graph visualization from "Mesh Graph Visualization" to "Mesh Graph" for conciseness.
- Removed the subtitle from the real-time monitoring header to streamline the display.
- Replaced the `abbreviate_noaa` method calls with a new `_noaa_period_display_name` method to improve the display of forecast period names, including handling federal holidays.
- Updated the `BotDataViewer` class to include the bot name in the returned configuration dictionary, ensuring consistent naming across the application.
- Made minor adjustments to HTML templates for better layout and added footer links for project visibility.
- Updated the logic for storing and displaying bot uptime to handle cases where the bot is not running or has no start time in the database, ensuring a more accurate representation of the bot's status.
- Enhanced the updateUptime method to provide clearer feedback when the bot is down, improving user experience.
- Refactored the live uptime counter to only run when valid uptime data is available, preventing unnecessary updates when the bot is inactive.
- Introduced a new method to retrieve the database path, allowing for a fallback to the [Bot] section if the [Web_Viewer] db_path is unset.
- Updated various parts of the codebase to utilize this new method, ensuring consistent database path resolution across the application.
- Enhanced the configuration example to clarify the optional nature of the db_path setting for the web viewer.
- Improved logic for determining the database path by prioritizing the [Web_Viewer] section and falling back to the [Bot] section if necessary.
- Updated the packet stream initialization to use the resolved database path from the constructor, ensuring consistency across the application.
- Updated the configuration handling for the web viewer to use the [Bot] db_path when [Web_Viewer] db_path is unset, ensuring consistent database access.
- Added warnings in the logging to notify users when the web viewer database path differs from the bot database path, guiding them to configure it correctly.
- Refactored database path resolution logic across multiple modules for better maintainability and clarity.
- Added error handling in the template context processor to ensure templates render without raising exceptions, logging any failures.
- Enhanced the configuration path resolution to support relative paths when starting the web viewer as a subprocess, improving flexibility.
- Implemented logging for unhandled exceptions in Flask routes to aid in debugging and provide clearer error reporting.
- Replaced the "Last 7 Days" option with "Last 48 Hours" and "Last 72 Hours" in both edge and node filters for better granularity.
- Enhanced the layout of the "Show Only Starred" checkbox for improved user experience.
- Introduced filtered node maps for efficient edge drawing, ensuring edges are only drawn between nodes within the selected timeframe.
- Added a button for bulk deletion of selected contacts, enhancing user experience by allowing multiple contacts to be deleted at once.
- Introduced a checkbox for selecting all contacts on the current page, with corresponding updates to the bulk delete button visibility and count.
- Updated the contacts HTML template to accommodate the new bulk delete feature and improved layout for better usability.
- Enhanced the ModernContactsManager class to manage selected contact IDs and handle bulk delete actions effectively.
- Updated the API endpoint to accept an optional query parameter `since` for filtering contact data based on the time frame (24h, 7d, 30d, 90d, or all).
- Modified the `_get_tracking_data` method to implement filtering logic based on the `since` parameter, improving performance by limiting the data retrieved.
- Enhanced the contacts HTML template to include a dropdown for selecting the time frame, with the default set to the last 30 days.
- Implemented local storage functionality to remember the user's selected time frame across sessions.
- Changed the default database path in config.ini.example and related files from 'bot_data.db' to 'meshcore_bot.db' for consistency across modules.
- Adjusted final hop proximity scoring parameters in config.ini.example to reflect updated distance normalization and thresholds, enhancing scoring accuracy for LoRa networks.
- Updated documentation to clarify the shared database usage between the bot and web viewer.
- Added '*.key' to .gitignore to prevent tracking of key files.
- Changed the default database path in config.ini.example from 'bot_data.db' to 'meshcore_bot.db' for clarity.
- Enhanced documentation files with front matter for better rendering and organization, including Discord Bridge, Feed Management, Map Uploader, Packet Capture, Path Command Configuration, Repeater Management Commands, and Weather Service.
