Correctness:
- {path_distance} was always blank in production. The resolution code that
builds repeater_info dropped latitude/longitude in every branch, so the
calculator could never find a coordinate. My tests passed hand-built
dicts straight to the calculator and never exercised the builder, which
is why they stayed green. Coordinates are now carried through all four
construction sites, and the new tests drive _lookup_repeater_names via
its lookup_func hook so the real builder runs.
- The #80 route guard was defeated two ways in the channel handler. When
the RF data was an uncorrelated fallback, control fell through to the
raw-hex and routing_info fallbacks below, which took the route from the
unrelated packet anyway; the guard had actually made that path
reachable. message.routing_info was also assigned unconditionally, and
the path command reads it. "Not attributable" is now a terminal branch
and the routing_info hand-off checks provenance.
- Same fix was incomplete for DMs: routing_info was captured and turned
into path_info before the provenance check ran, so the later check only
declined to overwrite an already-wrong value. Guarded at the source.
- Rendering could transmit for real. Capture only intercepts
send_response, but advert calls send_advert() directly and
send_response_chunked never checked capture_sink. Chunked sends are now
captured, and rendering is opt-in via BaseCommand.render_safe (default
False) instead of a denylist that cannot be complete. This also closes
the DM-only leak: schedule is not marked safe, so {cmd:schedule} can no
longer broadcast configuration to a channel.
- Multi-part rendered output was rejoined into one oversized send.
Scheduled messages are now split to the RF body budget and sent through
send_channel_messages_chunked, on character boundaries so multi-byte
text is not corrupted.
- _last_path_distance_km is instance state that was only set on success,
so an invalid path request could show the previous request's distance.
Reset at the start of every execute().
- Stale-contact retries were only counted on non-OK results, so timeouts
and exceptions left a contact eligible forever and could recreate the
storm. All failed attempts count now.
Web viewer:
- A failed config reload was reported as a successful save. The API and
UI now distinguish "saved and active" from "saved, restart needed".
- Preview count was unbounded; clamped to 1-20.
- update_ini_values is a read-modify-replace with no locking, so two
concurrent viewer saves could lose one. Serialised behind a lock.
The seed check added in c1cbdf9 only guarded stale-contact cleanup, but
three other paths read the same raw last_advert and drew the same wrong
conclusion from it:
_get_repeaters_for_purging sorts by apparent age, oldest first
_get_companions_for_purging scores by days_inactive, most inactive first
purge_old_repeaters removes anything older than a cutoff
In all three an unset clock reads as maximum age, so a node that had
never been time-synced was the first candidate for eviction regardless
of whether it was active. Unlike the stale-contact path, these do not
just log — they remove contacts.
Same rule everywhere now: an unset device clock means staleness is
unknown, and unknown staleness is not grounds for removal.
Corrects the timestamp guard shipped in 5b07ee2, which assumed the
reporter's "722 days ago" contacts were genuine mid-2024 observations.
They were not.
MeshCore seeds an unset clock with a hardcoded time rather than zero:
1715770351 (15 May 2024) in VolatileRTCClock, and RTC_TIME_MIN
1772323200 (1 Mar 2026) on the NRF52 and ESP32 RTC paths. Measured from
the date in the issue, 1715770351 is exactly 722 days — every affected
contact reported the same figure because they were all sitting on the
same seed.
So those contacts were never stale. Their clocks were unset, and the bot
was reading that as extreme age: they sorted to the top of the staleness
list, consumed the whole per-sweep removal budget, and the bot kept
trying to evict nodes that may well have been active. The previous
2020-then-2024-01-01 floor sat below both seeds and never fired.
Now matched against the seeds themselves, plus anything at or below the
earliest (which still covers a raw 0 decoding to 1970). A device running
unsynced for a while reports seed + uptime and remains undetectable;
noted in the docstring rather than guessed at.
A failed remove_contact leaves the contact on the device, so the next
cleanup sweep selected it again and logged the same warning again. With
the contact list stuck near its limit the sweeps kept coming, which is
the reported flood of hundreds of "Failed to remove stale contact"
warnings that only a restart cleared — and the restart only helped
because it reset the in-memory contact list, not because anything was
fixed.
Refusals are now counted per public key. After 3 consecutive failures
the contact is excluded from future sweeps, with one summary warning
saying the list may stay near its limit and that the contact needs
removing from the companion app. A successful removal clears the count,
so a transient failure costs nothing.
Also stop treating unset and future last_seen values as staleness. An
unset timestamp parses as 1970, and since candidates are sorted by
staleness descending, those entries sorted to the top and consumed the
whole max_remove budget every sweep — starving the contacts that could
actually have been removed. A genuinely old contact (the reporter's 722
days) is still selected; that is a real observation, not a bad
timestamp.
- Introduced a locking mechanism to ensure journal mode is initialized once per configuration section, preventing redundant database operations.
- Added a validation function to check for required repeater tables, raising an error with actionable messages if any are missing.
- Updated the DBManager to handle journal modes more efficiently, ensuring that only persistent modes are applied across connections.
- Refactored the RepeaterManager to utilize the new validation function, improving error handling during initialization.
- Enhanced tests to cover new journal mode behaviors and validate repeater table existence, ensuring robust database management.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
Changed the default setting for `auto_manage_contacts` from `false` to `device` across configuration files and updated related documentation. This ensures that the device handles auto-addition of contacts while the bot manages capacity. Adjusted comments and documentation to reflect this change for clarity and consistency.
NEW_CONTACT adverts were always logged as newly discovered, even for
nodes heard many times before. Check the tracking DB (get_tracked_contact_row)
and the live device contact list (is_contact_on_device) to classify the
advert, log "known" vs "new" accordingly, and skip the
new_contact_discovered audit entry for contacts that are already known.
Also dedup the is_currently_tracked device-contact lookups onto the shared
is_contact_on_device() helper.
Behavior preserved: repeaters remain database-only, device-mode companion
adds are still left to the firmware, and no per-advert device round-trip is
introduced.
- Updated the `track_contact_advertisement` method to return a structured result indicating success and duplicate packet status, improving clarity in handling contact advertisements.
- Adjusted the `MessageHandler` to utilize the new result structure, allowing for better decision-making based on tracking outcomes.
- Enhanced unit tests to cover new behaviors and ensure correct handling of duplicate packets and advertisement tracking results.
- Modified the SQL insert statement in the RepeaterManager class to include `public_key` and `name` fields, ensuring compatibility with the new schema.
- Updated corresponding test cases to reflect changes in the database operation, maintaining the integrity of the purging log functionality.
- Added a new migration to include a 'details' column in the 'purging_log' table for improved logging of actions.
- Refactored the `log_purging_action` method in `RepeaterManager` to handle both new and legacy schemas, ensuring compatibility with the updated database structure.
- Updated `MessageHandler` to utilize the new logging method, replacing direct database updates with a unified logging approach.
- Added tests to verify the presence of the new column and the correct functionality of the logging mechanism across different scenarios.
- Enhanced configuration options for `auto_manage_contacts` to support 'device' mode, allowing firmware to handle companion auto-addition and favourite hygiene.
- Updated `MessageHandler` to track new companions based on the `auto_manage_contacts` setting, with distinct behaviors for 'false', 'device', and 'bot' modes.
- Introduced scheduled jobs in `MessageScheduler` for device mode to manage firmware preferences and favourite keys with specified delays.
- Modified `RepeaterManager` to skip companion auto-purge in device mode, ensuring firmware manages contact slots effectively.
- Added tests to validate new behaviors and configurations, ensuring robust handling of contact management across different modes.
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
- Introduced locking mechanisms to prevent overlapping auto-purge runs and duplicate purge attempts for specific keys.
- Added methods to manage in-flight purge attempts, ensuring that concurrent calls do not interfere with each other.
- Updated the `check_and_auto_purge` method to utilize the new locking logic, enhancing the reliability of the auto-purge process.
- Expanded test coverage to validate the new concurrency controls, ensuring correct behavior under concurrent execution scenarios.
These changes improve the stability and efficiency of the repeater management system during purge operations.
- Added a regex pattern to validate SQLite column definitions, preventing SQL injection in the _add_column() function.
- Introduced a new validation function to ensure safe column definitions are used.
- Updated repeater_manager.py to use list() for iterating over contacts, ensuring compatibility with potential changes in the underlying data structure.
- Enhanced error handling in the MessageScheduler for better debugging during scheduler shutdown and message sending.
These changes improve the security and reliability of database operations and enhance the robustness of the repeater management system.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
BUG-001: web viewer login/session auth (in web viewer commit)
BUG-002: db_manager ALTER TABLE for missing channel_operations and
feed_message_queue columns on startup
BUG-015: scheduler thread blocked on future.result(); replaced all
blocking waits with add_done_callback (fire-and-forget)
BUG-016: reboot_radio sends meshcore.commands.reboot() before disconnect
BUG-017: radio disconnect uses asyncio.wait_for(timeout=10)
BUG-022: custom asyncio loop exception handler suppresses IndexError
from meshcore parser at DEBUG level
BUG-024: last_db_backup_run updated after each run; 2-min startup
window; last-run seeded from DB on restart
BUG-025: send_channel_message retries up to 2 times (2s delay) on
no_event_received via _is_no_event_received() helper
BUG-026: split_text_into_chunks() and get_max_message_length() added
to CommandManager; keyword dispatch uses send_response_chunked()
BUG-028: byte_data = b"" initialised before try block in
decode_meshcore_packet to prevent UnboundLocalError in except handler
TraceCommand: path nodes reversed and return path truncated; fixed
format_elapsed_display: UTC normalisation before elapsed computation (#75)
RepeaterManager: auto_manage_contacts guard before any purge logic (#50)
Command aliases: [Aliases] config section injects shorthands at startup
JSON logging: _JsonFormatter; json_logging = true in [Logging]
Structured JSON logging compatible with Loki, Elasticsearch, Splunk
Discord bridge, Telegram bridge, and all service plugins updated
MeshGraph edge promotion logic corrected
Shutdown: scheduler and meshcore disconnect joined cleanly; log spam fixed
All modules: ruff and mypy cleanup applied (type annotations, imports)
- Modified the RepeaterManager class to enable or disable auto-purge based on the 'auto_manage_contacts' setting from the bot's configuration.
- This change allows for more flexible management of contacts, aligning with the bot's operational mode.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Updated the repeater_manager table schema to include a new column `is_starred` with a default value of 0, allowing for the tracking of starred entries.
- This change enhances the functionality of the repeater manager by enabling users to mark specific entries as starred.
- Enhanced .gitignore to allow test files in the tests/ directory and committed pytest.ini for test discovery.
- Added checks for missing sections in configuration files, specifically for Admin_ACL and Banned_Users, to prevent errors during bot startup.
- Updated generate_website.py and command_manager.py to handle cases where required sections are absent, returning empty lists instead of raising exceptions.
- Introduced optional dependencies for testing in pyproject.toml, ensuring a smoother development experience.
- Improved localization handling in core.py to default to English when the Localization section is missing, enhancing user experience.
- Updated the database migration logic to handle missing tables gracefully, ensuring that migrations for existing installations do not block the process.
- Added checks for the existence of specific columns in the `repeater_contacts`, `complete_contact_tracking`, and `mesh_connections` tables, allowing for more robust schema updates.
- Introduced logging for migration errors to improve visibility and troubleshooting during the migration process.
- Ensured that the migration runs seamlessly for users accessing the web viewer without prior bot startup, enhancing user experience.
- Changed the default database path in config.ini.example and related files from 'bot_data.db' to 'meshcore_bot.db' for consistency across modules.
- Adjusted final hop proximity scoring parameters in config.ini.example to reflect updated distance normalization and thresholds, enhancing scoring accuracy for LoRa networks.
- Updated documentation to clarify the shared database usage between the bot and web viewer.
- Added new configuration options for path selection presets and proximity methods in config.ini.example, allowing users to customize routing behavior.
- Implemented a new database table for storing observed paths, enabling better tracking of paths from advertisements and messages.
- Updated MessageHandler to store complete paths in the observed_paths table, improving path validation and selection accuracy.
- Enhanced the PathCommand class to utilize new proximity and recency settings, optimizing repeater selection based on user-defined criteria.
- Improved web viewer functionality to display multiple paths for contacts, enhancing user experience and interaction with path data.
- Added a new mesh graph feature for improved path validation, allowing for enhanced routing accuracy.
- Introduced configuration options for recency decay half-life and graph-based validation settings in config.ini.example.
- Updated the PathCommand class to utilize graph-based selection methods, combining graph and geographic scores for better repeater selection.
- Implemented new methods in MessageHandler to update the mesh graph with advertisement paths and trace packet data.
- Created a new database table for mesh connections to support graph-based path validation.
- Enhanced web viewer integration to display mesh graph updates in real-time, improving user interaction and monitoring capabilities.
- Introduced a new database table 'unique_advert_packets' for tracking unique advert packets by their hash.
- Enhanced the RepeaterManager to handle unique packet tracking during daily advertisement statistics.
- Updated StatsCommand to include a new subcommand for displaying the leaderboard of nodes with the most unique advert packets in the last 24 hours.
- Modified translations to support the new advert statistics feature, ensuring user-friendly command descriptions and error messages.
- Implemented a mechanism in MessageHandler to resolve location names from the database using public keys, providing a more user-friendly display of locations.
- Added fallback logic to ensure coordinates are used if no resolved location is found.
- Updated comments in RepeaterManager to clarify the order of preference for city extraction and introduced county as a fallback for rural areas, improving location accuracy.
- Added configuration options for companion contact purging, including thresholds for inactivity based on direct messages and advertisements.
- Enhanced the RepeaterManager to support automatic purging of companions when contact limits are exceeded.
- Updated PathCommand to utilize configurable recency/proximity weighting for improved path routing decisions.
- Introduced new commands for purging companions and updated existing commands to handle companion purging logic.
- Added API endpoint for manual geocoding of contacts and improved web viewer functionality for geocoding contacts.
- Added a new 'daily_stats' table for tracking daily advertisement statistics.
- Implemented methods to track daily advertisement counts and retrieve statistics over specified date ranges.
- Updated existing advertisement tracking logic to utilize the new daily statistics.
- Modified web viewer to display advertisement metrics using the new daily tracking data.
- Improved path command logic to prioritize database queries over API cache for path decoding.
- Add Flask + Flask-SocketIO web viewer with dashboard (modules/web_viewer/app.py and related)
- Add web viewer templates: index, realtime, tracking (contacts), cache, purging, stats (modules/web_viewer/templates/)
- Add integration hooks and utility functions for web viewer (modules/web_viewer/integration.py, modules/utils.py)
- Add command to launch web viewer from bot CLI (modules/commands/webviewer_command.py)
- Update .gitignore: ignore db/log files, test scripts, and web viewer artifacts
- Add restart_viewer.sh helper script for standalone web viewer restart/troubleshooting
- Add guidance and documentation for modern viewer in WEB_VIEWER.md and docs/
- Various code structure and import improvements to core bot and command modules to support integration
- Add ACL support for sensitive commands
- Example config updates
Benefits:
- Decouples monitoring/UI from bot core process
- Enables real-time browser dashboard and unified contact/repeater tracking
- Easier integration, dev, and troubleshooting
agessaman