_get_city_from_coordinates and _get_full_location_from_coordinates carried the same city chain, county fallback and large-city neighborhood lookup; _get_state_country_from_coordinates and the full-location path the same state chain; and all three the same packet-hash cache-window check. They are now _city_from_address, _state_from_address and _recent_geocode_age. A side-by-side run of the old and new module over 432 combinations of address shape, stored data and packet-hash cache state gave identical results, log calls and cache writes.
#315 changed populate_missing_geolocation_data, which S1 had deleted as dead. It is public, and local plugins may call public manager and command methods, so it comes back in dev's form, along with 11 other public methods S1 removed: CommandManager.get_available_commands_list, GreeterCommand.get_recent_greeted_users, WxCommand.abbreviate_alert_title and get_current_conditions, MessageHandler.discover_message_path, and RepeaterManager.add_discovered_contact, get_daily_advertisement_stats, get_nodes_per_day_stats, get_purging_stats, restore_repeater, test_meshcore_cli_commands and toggle_auto_add. All are verbatim from dev; every self. attribute they use still exists. A scan of every public name in modules/ on dev against this branch now finds nothing missing.
Moving the geocode decision ahead of the await meant every advert that arrived while another was still geocoding read the same uncommitted row and geocoded again; six adverts for one contact made nine or more Nominatim requests instead of two, at the 1 req/s budget. track_contact_advertisement now holds a per-contact asyncio lock across the read, geocode and write, which restores the old one-at-a-time behavior for a contact while different contacts still proceed independently. The lock entry is dropped when its last holder leaves.
Also: geocoding_cache reads in the worker-thread helpers use .get() so the loop-thread expiry pass cannot raise KeyError between the check and the read, the write-block duplicate guard calls _is_duplicate_advert_packet, and the guard test loses leftover scaffolding.
track_contact_advertisement, _background_geocoding, scan_and_catalog_repeaters and populate_missing_geolocation_data are async but called rate-limited Nominatim synchronously (with the rate limiter's time.sleep), so the whole bot stalled for each lookup. The async-path guard test only scanned commands and service plugins, so it never saw this file.
The calls now go through asyncio.to_thread. track_contact_advertisement decides whether to geocode, and does it, before its write block, which stays free of awaits so concurrent adverts cannot interleave between its reads and writes; the duplicate-packet guard runs both before geocoding and again inside the write block. The guard test now scans repeater_manager.py and knows the state/country and full-location helpers; a new test shows advert geocoding runs off the loop thread and fails without the change.
handle_contact_message built path_info from the contact table, the payload's path_len and a correlated RF packet's routing_info, then unconditionally replaced it (and path_len) before constructing the MeshMessage; the RF lookup it did was repeated after the message was built. The only effect of the removed block was a pair of INFO log lines reporting a route the DM never carried. The helpers it called (find_recent_rf_data, decode_meshcore_packet) mutate nothing.
handle_new_contact's 'unknown contact type' fallback could not run: the branch above returns on both of its paths whenever the bot has a repeater_manager, and every statement in the fallback was guarded by that same hasattr. RepeaterManager.add_discovered_contact was called only from there.
Every name removed here has zero references across modules, tests, docs, local plugins, scripts and generate_website.py, and none is part of the documented plugin surface (db_manager, command_manager send methods, channel_manager stay untouched). About 1,160 lines:
- multitest: four formatter helpers that only referenced each other (the flat and nested variants are the ones in use)
- path: _get_api_cache_data, _filter_recent_repeaters
- repeater command: _handle_contact_stats, _get_geocoding_status
- airplanes: _send_split_response; greeter: get_recent_greeted_users; plugin_loader: validate_plugin
- wx: get_weather_for_zipcode, abbreviate_alert_title, get_current_conditions
- message_handler: discover_message_path, and an ADVERT debug line behind a self.debug attribute that is never set
- utils: _get_node_location_and_key_from_db
- repeater_manager: test_meshcore_cli_commands, toggle_auto_add, restore_repeater, get_purging_stats, populate_missing_geolocation_data, get_daily_advertisement_stats, get_nodes_per_day_stats
- command_manager: get_available_commands_list and get_plugin_metadata (HelpCommand and PluginLoader have the live versions)
- base_command: _load_command_prefix, and a no-op .replace('_', '_')
- scheduler: the first of two _run_data_retention definitions, shadowed by the second since it was added
- core: the write-only last_clock_sync_time, and _setup_routing_capture, which only logged a line
Full suite, ruff and mypy pass, and the characterization golden files are unchanged.
Correctness:
- {path_distance} was always blank in production. The resolution code that
builds repeater_info dropped latitude/longitude in every branch, so the
calculator could never find a coordinate. My tests passed hand-built
dicts straight to the calculator and never exercised the builder, which
is why they stayed green. Coordinates are now carried through all four
construction sites, and the new tests drive _lookup_repeater_names via
its lookup_func hook so the real builder runs.
- The #80 route guard was defeated two ways in the channel handler. When
the RF data was an uncorrelated fallback, control fell through to the
raw-hex and routing_info fallbacks below, which took the route from the
unrelated packet anyway; the guard had actually made that path
reachable. message.routing_info was also assigned unconditionally, and
the path command reads it. "Not attributable" is now a terminal branch
and the routing_info hand-off checks provenance.
- Same fix was incomplete for DMs: routing_info was captured and turned
into path_info before the provenance check ran, so the later check only
declined to overwrite an already-wrong value. Guarded at the source.
- Rendering could transmit for real. Capture only intercepts
send_response, but advert calls send_advert() directly and
send_response_chunked never checked capture_sink. Chunked sends are now
captured, and rendering is opt-in via BaseCommand.render_safe (default
False) instead of a denylist that cannot be complete. This also closes
the DM-only leak: schedule is not marked safe, so {cmd:schedule} can no
longer broadcast configuration to a channel.
- Multi-part rendered output was rejoined into one oversized send.
Scheduled messages are now split to the RF body budget and sent through
send_channel_messages_chunked, on character boundaries so multi-byte
text is not corrupted.
- _last_path_distance_km is instance state that was only set on success,
so an invalid path request could show the previous request's distance.
Reset at the start of every execute().
- Stale-contact retries were only counted on non-OK results, so timeouts
and exceptions left a contact eligible forever and could recreate the
storm. All failed attempts count now.
Web viewer:
- A failed config reload was reported as a successful save. The API and
UI now distinguish "saved and active" from "saved, restart needed".
- Preview count was unbounded; clamped to 1-20.
- update_ini_values is a read-modify-replace with no locking, so two
concurrent viewer saves could lose one. Serialised behind a lock.
The seed check added in c1cbdf9 only guarded stale-contact cleanup, but
three other paths read the same raw last_advert and drew the same wrong
conclusion from it:
_get_repeaters_for_purging sorts by apparent age, oldest first
_get_companions_for_purging scores by days_inactive, most inactive first
purge_old_repeaters removes anything older than a cutoff
In all three an unset clock reads as maximum age, so a node that had
never been time-synced was the first candidate for eviction regardless
of whether it was active. Unlike the stale-contact path, these do not
just log — they remove contacts.
Same rule everywhere now: an unset device clock means staleness is
unknown, and unknown staleness is not grounds for removal.
Corrects the timestamp guard shipped in 5b07ee2, which assumed the
reporter's "722 days ago" contacts were genuine mid-2024 observations.
They were not.
MeshCore seeds an unset clock with a hardcoded time rather than zero:
1715770351 (15 May 2024) in VolatileRTCClock, and RTC_TIME_MIN
1772323200 (1 Mar 2026) on the NRF52 and ESP32 RTC paths. Measured from
the date in the issue, 1715770351 is exactly 722 days — every affected
contact reported the same figure because they were all sitting on the
same seed.
So those contacts were never stale. Their clocks were unset, and the bot
was reading that as extreme age: they sorted to the top of the staleness
list, consumed the whole per-sweep removal budget, and the bot kept
trying to evict nodes that may well have been active. The previous
2020-then-2024-01-01 floor sat below both seeds and never fired.
Now matched against the seeds themselves, plus anything at or below the
earliest (which still covers a raw 0 decoding to 1970). A device running
unsynced for a while reports seed + uptime and remains undetectable;
noted in the docstring rather than guessed at.
A failed remove_contact leaves the contact on the device, so the next
cleanup sweep selected it again and logged the same warning again. With
the contact list stuck near its limit the sweeps kept coming, which is
the reported flood of hundreds of "Failed to remove stale contact"
warnings that only a restart cleared — and the restart only helped
because it reset the in-memory contact list, not because anything was
fixed.
Refusals are now counted per public key. After 3 consecutive failures
the contact is excluded from future sweeps, with one summary warning
saying the list may stay near its limit and that the contact needs
removing from the companion app. A successful removal clears the count,
so a transient failure costs nothing.
Also stop treating unset and future last_seen values as staleness. An
unset timestamp parses as 1970, and since candidates are sorted by
staleness descending, those entries sorted to the top and consumed the
whole max_remove budget every sweep — starving the contacts that could
actually have been removed. A genuinely old contact (the reporter's 722
days) is still selected; that is a real observation, not a bad
timestamp.
- Introduced a locking mechanism to ensure journal mode is initialized once per configuration section, preventing redundant database operations.
- Added a validation function to check for required repeater tables, raising an error with actionable messages if any are missing.
- Updated the DBManager to handle journal modes more efficiently, ensuring that only persistent modes are applied across connections.
- Refactored the RepeaterManager to utilize the new validation function, improving error handling during initialization.
- Enhanced tests to cover new journal mode behaviors and validate repeater table existence, ensuring robust database management.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
Changed the default setting for `auto_manage_contacts` from `false` to `device` across configuration files and updated related documentation. This ensures that the device handles auto-addition of contacts while the bot manages capacity. Adjusted comments and documentation to reflect this change for clarity and consistency.
NEW_CONTACT adverts were always logged as newly discovered, even for
nodes heard many times before. Check the tracking DB (get_tracked_contact_row)
and the live device contact list (is_contact_on_device) to classify the
advert, log "known" vs "new" accordingly, and skip the
new_contact_discovered audit entry for contacts that are already known.
Also dedup the is_currently_tracked device-contact lookups onto the shared
is_contact_on_device() helper.
Behavior preserved: repeaters remain database-only, device-mode companion
adds are still left to the firmware, and no per-advert device round-trip is
introduced.
- Updated the `track_contact_advertisement` method to return a structured result indicating success and duplicate packet status, improving clarity in handling contact advertisements.
- Adjusted the `MessageHandler` to utilize the new result structure, allowing for better decision-making based on tracking outcomes.
- Enhanced unit tests to cover new behaviors and ensure correct handling of duplicate packets and advertisement tracking results.
- Modified the SQL insert statement in the RepeaterManager class to include `public_key` and `name` fields, ensuring compatibility with the new schema.
- Updated corresponding test cases to reflect changes in the database operation, maintaining the integrity of the purging log functionality.
- Added a new migration to include a 'details' column in the 'purging_log' table for improved logging of actions.
- Refactored the `log_purging_action` method in `RepeaterManager` to handle both new and legacy schemas, ensuring compatibility with the updated database structure.
- Updated `MessageHandler` to utilize the new logging method, replacing direct database updates with a unified logging approach.
- Added tests to verify the presence of the new column and the correct functionality of the logging mechanism across different scenarios.
- Enhanced configuration options for `auto_manage_contacts` to support 'device' mode, allowing firmware to handle companion auto-addition and favourite hygiene.
- Updated `MessageHandler` to track new companions based on the `auto_manage_contacts` setting, with distinct behaviors for 'false', 'device', and 'bot' modes.
- Introduced scheduled jobs in `MessageScheduler` for device mode to manage firmware preferences and favourite keys with specified delays.
- Modified `RepeaterManager` to skip companion auto-purge in device mode, ensuring firmware manages contact slots effectively.
- Added tests to validate new behaviors and configurations, ensuring robust handling of contact management across different modes.
Add SSRF host validation to maintenance.py send_nightly_email and
scheduler.py send_zombie_alert_email using validate_external_url().
New allow_local_smtp config key permits private-IP SMTP for local
relay setups.
Add sanitize_name() to security_utils and apply it to all log calls
in message_handler, repeater_manager, path_command, solarforecast_command,
command_manager, and discord_bridge_service to prevent log injection.
Move nightly email logic from duplicate scheduler._send_nightly_email()
into the canonical maintenance.py implementation, removing the duplicate.
Update tests to call maintenance.send_nightly_email() directly.
Add validate_external_url allow_private parameter with support for
loopback, RFC1918, CGN, and link-local address ranges.
- Introduced locking mechanisms to prevent overlapping auto-purge runs and duplicate purge attempts for specific keys.
- Added methods to manage in-flight purge attempts, ensuring that concurrent calls do not interfere with each other.
- Updated the `check_and_auto_purge` method to utilize the new locking logic, enhancing the reliability of the auto-purge process.
- Expanded test coverage to validate the new concurrency controls, ensuring correct behavior under concurrent execution scenarios.
These changes improve the stability and efficiency of the repeater management system during purge operations.
- Added a regex pattern to validate SQLite column definitions, preventing SQL injection in the _add_column() function.
- Introduced a new validation function to ensure safe column definitions are used.
- Updated repeater_manager.py to use list() for iterating over contacts, ensuring compatibility with potential changes in the underlying data structure.
- Enhanced error handling in the MessageScheduler for better debugging during scheduler shutdown and message sending.
These changes improve the security and reliability of database operations and enhance the robustness of the repeater management system.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
BUG-001: web viewer login/session auth (in web viewer commit)
BUG-002: db_manager ALTER TABLE for missing channel_operations and
feed_message_queue columns on startup
BUG-015: scheduler thread blocked on future.result(); replaced all
blocking waits with add_done_callback (fire-and-forget)
BUG-016: reboot_radio sends meshcore.commands.reboot() before disconnect
BUG-017: radio disconnect uses asyncio.wait_for(timeout=10)
BUG-022: custom asyncio loop exception handler suppresses IndexError
from meshcore parser at DEBUG level
BUG-024: last_db_backup_run updated after each run; 2-min startup
window; last-run seeded from DB on restart
BUG-025: send_channel_message retries up to 2 times (2s delay) on
no_event_received via _is_no_event_received() helper
BUG-026: split_text_into_chunks() and get_max_message_length() added
to CommandManager; keyword dispatch uses send_response_chunked()
BUG-028: byte_data = b"" initialised before try block in
decode_meshcore_packet to prevent UnboundLocalError in except handler
TraceCommand: path nodes reversed and return path truncated; fixed
format_elapsed_display: UTC normalisation before elapsed computation (#75)
RepeaterManager: auto_manage_contacts guard before any purge logic (#50)
Command aliases: [Aliases] config section injects shorthands at startup
JSON logging: _JsonFormatter; json_logging = true in [Logging]
Structured JSON logging compatible with Loki, Elasticsearch, Splunk
Discord bridge, Telegram bridge, and all service plugins updated
MeshGraph edge promotion logic corrected
Shutdown: scheduler and meshcore disconnect joined cleanly; log spam fixed
All modules: ruff and mypy cleanup applied (type annotations, imports)
- Modified the RepeaterManager class to enable or disable auto-purge based on the 'auto_manage_contacts' setting from the bot's configuration.
- This change allows for more flexible management of contacts, aligning with the bot's operational mode.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced a new utility function `decode_path_len_byte` to decode RF packet path length bytes, supporting both legacy and multi-byte paths.
- Updated various modules to utilize the new decoding logic, ensuring compatibility with configured prefix lengths.
- Modified database schemas to include `bytes_per_hop` and `out_bytes_per_hop` columns for better path management.
- Enhanced path parsing and validation across commands and services to accommodate variable prefix lengths.
- Improved logging and error handling for path-related operations, ensuring robustness during transitions.
- Introduced a new `[Data_Retention]` section in `config.ini.example` to manage retention periods for various database tables, including packet stream, daily stats, and observed paths.
- Updated `mkdocs.yml` and `configuration.md` to include documentation for the new data retention settings.
- Implemented data retention cleanup methods in `mesh_graph.py`, `repeater_manager.py`, and `scheduler.py` to enforce retention policies automatically.
- Enhanced the web viewer's data cleanup logic to utilize the new retention settings, ensuring efficient database management.
- Updated the repeater_manager table schema to include a new column `is_starred` with a default value of 0, allowing for the tracking of starred entries.
- This change enhances the functionality of the repeater manager by enabling users to mark specific entries as starred.
- Enhanced .gitignore to allow test files in the tests/ directory and committed pytest.ini for test discovery.
- Added checks for missing sections in configuration files, specifically for Admin_ACL and Banned_Users, to prevent errors during bot startup.
- Updated generate_website.py and command_manager.py to handle cases where required sections are absent, returning empty lists instead of raising exceptions.
- Introduced optional dependencies for testing in pyproject.toml, ensuring a smoother development experience.
- Improved localization handling in core.py to default to English when the Localization section is missing, enhancing user experience.
- Updated the database migration logic to handle missing tables gracefully, ensuring that migrations for existing installations do not block the process.
- Added checks for the existence of specific columns in the `repeater_contacts`, `complete_contact_tracking`, and `mesh_connections` tables, allowing for more robust schema updates.
- Introduced logging for migration errors to improve visibility and troubleshooting during the migration process.
- Ensured that the migration runs seamlessly for users accessing the web viewer without prior bot startup, enhancing user experience.
- Changed the default database path in config.ini.example and related files from 'bot_data.db' to 'meshcore_bot.db' for consistency across modules.
- Adjusted final hop proximity scoring parameters in config.ini.example to reflect updated distance normalization and thresholds, enhancing scoring accuracy for LoRa networks.
- Updated documentation to clarify the shared database usage between the bot and web viewer.
- Added new configuration options for path selection presets and proximity methods in config.ini.example, allowing users to customize routing behavior.
- Implemented a new database table for storing observed paths, enabling better tracking of paths from advertisements and messages.
- Updated MessageHandler to store complete paths in the observed_paths table, improving path validation and selection accuracy.
- Enhanced the PathCommand class to utilize new proximity and recency settings, optimizing repeater selection based on user-defined criteria.
- Improved web viewer functionality to display multiple paths for contacts, enhancing user experience and interaction with path data.
- Added a new mesh graph feature for improved path validation, allowing for enhanced routing accuracy.
- Introduced configuration options for recency decay half-life and graph-based validation settings in config.ini.example.
- Updated the PathCommand class to utilize graph-based selection methods, combining graph and geographic scores for better repeater selection.
- Implemented new methods in MessageHandler to update the mesh graph with advertisement paths and trace packet data.
- Created a new database table for mesh connections to support graph-based path validation.
- Enhanced web viewer integration to display mesh graph updates in real-time, improving user interaction and monitoring capabilities.
- Introduced a new database table 'unique_advert_packets' for tracking unique advert packets by their hash.
- Enhanced the RepeaterManager to handle unique packet tracking during daily advertisement statistics.
- Updated StatsCommand to include a new subcommand for displaying the leaderboard of nodes with the most unique advert packets in the last 24 hours.
- Modified translations to support the new advert statistics feature, ensuring user-friendly command descriptions and error messages.
- Implemented a mechanism in MessageHandler to resolve location names from the database using public keys, providing a more user-friendly display of locations.
- Added fallback logic to ensure coordinates are used if no resolved location is found.
- Updated comments in RepeaterManager to clarify the order of preference for city extraction and introduced county as a fallback for rural areas, improving location accuracy.
- Added configuration options for companion contact purging, including thresholds for inactivity based on direct messages and advertisements.
- Enhanced the RepeaterManager to support automatic purging of companions when contact limits are exceeded.
- Updated PathCommand to utilize configurable recency/proximity weighting for improved path routing decisions.
- Introduced new commands for purging companions and updated existing commands to handle companion purging logic.
- Added API endpoint for manual geocoding of contacts and improved web viewer functionality for geocoding contacts.
- Added a new 'daily_stats' table for tracking daily advertisement statistics.
- Implemented methods to track daily advertisement counts and retrieve statistics over specified date ranges.
- Updated existing advertisement tracking logic to utilize the new daily statistics.
- Modified web viewer to display advertisement metrics using the new daily tracking data.
- Improved path command logic to prioritize database queries over API cache for path decoding.
agessaman