Port the observer firmware's neighbours feature into the bot's packet capture
service, by way of meshcore-packet-capture (upstream PRs #42/#43). On a long
interval the bot asks which repeaters it hears directly and records each
confirmed link with its measured SNR.
This is the strongest link evidence the bot collects: a first-party RF
measurement between two full 32-byte public keys. Path inference works from
1-3 byte prefixes with no keys, and complete_contact_tracking.hop_count
over-claims zero-hop (800 claimed vs 68 corroborated on the live database).
modules/neighbors_discovery.py keeps upstream's public names so its fixes and
tests stay portable. Two deliberate divergences:
- No command_lock plumbing. _SerializedCommands in modules/core.py already
serialises and paces every radio command, strictly more than upstream's
reentrant lock did.
- neighbors_collect_scopes defaults off. Upstream's zero-hop scope probe
relies on a neighbour not being a known contact; this bot tracks contacts,
and for a repeater with no stored path the library reaches zero-hop by
calling change_contact_path() then reset_path() -- mutating the device's
contact table per neighbour. Scope requests also hold the radio lock for
their whole round trip (~25s), stalling bot replies. The default cycle is
one command plus a passive listen window, during which the bot stays
responsive.
Evidence lands in neighbor_links and neighbor_observations (migration 22)
rather than mesh_connections, which cannot persist provenance. The viewer
exposes it as evidence=neighbors on /api/mesh/edges and a Neighbours Only
mode on the mesh page, with populated public keys and real SNR; confirmed
neighbours also relabel edges in the combined view and count as
provenance-trusted when framing the initial map.
neighbors_enabled is the single switch. Every enabled broker publishes once
it is on (mqttN_neighbors defaults true; set false to hold one back). The
topic derives from each broker's packets topic with the last segment swapped,
so a templated broker gets meshcore/{IATA}/{PUBLIC_KEY}/neighbors -- the
topic the firmware uses -- instead of an unrelated flat one. A derived
location-routed topic is skipped with a warning when no iata is set, rather
than publishing into meshcore/XYZ/... on a shared namespace. Snapshots are
non-retained: heard_secs_ago is relative to publish time, so a retained copy
would read as current days later.
Also adds a DM-gated `neighbors` command (the 12h interval floor makes
waiting for the scheduler impractical), which acks immediately and reports in
a second message once the window closes.
Requires meshcore >= 2.3.8 for send_node_discover_req / req_regions_sync.
- Introduced a locking mechanism to ensure journal mode is initialized once per configuration section, preventing redundant database operations.
- Added a validation function to check for required repeater tables, raising an error with actionable messages if any are missing.
- Updated the DBManager to handle journal modes more efficiently, ensuring that only persistent modes are applied across connections.
- Refactored the RepeaterManager to utilize the new validation function, improving error handling during initialization.
- Enhanced tests to cover new journal mode behaviors and validate repeater table existence, ensuring robust database management.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
- Rewrite test_subscribe_packets/messages_emits_status_ack to match the
silent subscription UX from 1ee84f2.
- Reconcile Python version: requires-python>=3.10, ruff target py310, CI
matrix adds 3.13, pyupgrade UP0xx ignored pending a separate typing-rewrite
PR; fix two B905 zip(strict=...) lints.
- Issue #80 fix in find_recent_rf_data: return None when correlation_key is
provided but unmatched; prefer the longest observed path among samples
sharing a packet_hash; narrow the no-key fallback to a configurable
rf_fallback_window (default 2s).
- Issue #161: lower shipped max_response_hops default 10 -> 7.
- Add CHANGELOG.md, restructure BUGS.md around a ## v0.9.0 Fixed Bugs
table, prune crossed-out duplicate outstanding rows, and add a
Deferred-from-v0.9.0 triage section to TODO.md.
- Untrack coverage.json and add it to .gitignore.
Made-with: Cursor
- Updated type hints to use `|` for union types instead of `Optional` where applicable, enhancing readability and consistency.
- Cleaned up import statements by removing unnecessary imports and organizing them for better clarity.
- Adjusted function signatures in various modules to reflect the new type hinting style, improving type safety and code maintainability.
These changes contribute to a more modern and consistent codebase, aligning with current Python typing practices.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Introduced validation for SQLite journal modes in DBManager, defaulting to "WAL" for invalid inputs and logging warnings.
- Added a new utility function in security_utils for validating SQL identifiers to prevent SQL injection.
- Updated BotDataViewer to utilize the new journal mode validation and SQL identifier checks, ensuring safer database operations.
- Enhanced test coverage for restore functionality, including checks for backup directory configuration and path traversal prevention.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced `execute_query_on_connection` and `execute_update_on_connection` methods in DBManager to facilitate executing queries and updates on existing database connections.
- Enhanced the `_recalculate_distance_if_needed`, `_get_location_by_public_key`, and `_get_location_by_prefix` methods in MeshGraph to support optional database connection and location caching, improving performance during batch operations.
- Added new configuration options for path selection presets and proximity methods in config.ini.example, allowing users to customize routing behavior.
- Implemented a new database table for storing observed paths, enabling better tracking of paths from advertisements and messages.
- Updated MessageHandler to store complete paths in the observed_paths table, improving path validation and selection accuracy.
- Enhanced the PathCommand class to utilize new proximity and recency settings, optimizing repeater selection based on user-defined criteria.
- Improved web viewer functionality to display multiple paths for contacts, enhancing user experience and interaction with path data.
- Added a new mesh graph feature for improved path validation, allowing for enhanced routing accuracy.
- Introduced configuration options for recency decay half-life and graph-based validation settings in config.ini.example.
- Updated the PathCommand class to utilize graph-based selection methods, combining graph and geographic scores for better repeater selection.
- Implemented new methods in MessageHandler to update the mesh graph with advertisement paths and trace packet data.
- Created a new database table for mesh connections to support graph-based path validation.
- Enhanced web viewer integration to display mesh graph updates in real-time, improving user interaction and monitoring capabilities.
- Introduced a new database table 'unique_advert_packets' for tracking unique advert packets by their hash.
- Enhanced the RepeaterManager to handle unique packet tracking during daily advertisement statistics.
- Updated StatsCommand to include a new subcommand for displaying the leaderboard of nodes with the most unique advert packets in the last 24 hours.
- Modified translations to support the new advert statistics feature, ensuring user-friendly command descriptions and error messages.
- Introduced a new get_connection method in DBManager to standardize database connection creation with proper configuration.
- Updated all instances in GreeterCommand where sqlite3.connect was used to utilize the new get_connection method, improving code consistency and maintainability.
- Implemented periodic updates of system health status to the database every 30 seconds.
- Added a new method to aggregate and return comprehensive health status of all components, including core connection, database, services, and web viewer.
- Introduced an API endpoint at /api/system-health to retrieve system health data, providing a structured response for health status and component details.
- Enhanced error handling for health data storage and retrieval processes, ensuring robust logging and feedback in case of issues.
- Modified database connection calls in DBManager, FeedManager, and MessageScheduler to convert db_path to a string and set a timeout of 30 seconds, improving reliability and performance of database operations.
- Enhanced error logging to include detailed information about database path existence and permissions, aiding in debugging and ensuring smoother operation.
- Add Flask + Flask-SocketIO web viewer with dashboard (modules/web_viewer/app.py and related)
- Add web viewer templates: index, realtime, tracking (contacts), cache, purging, stats (modules/web_viewer/templates/)
- Add integration hooks and utility functions for web viewer (modules/web_viewer/integration.py, modules/utils.py)
- Add command to launch web viewer from bot CLI (modules/commands/webviewer_command.py)
- Update .gitignore: ignore db/log files, test scripts, and web viewer artifacts
- Add restart_viewer.sh helper script for standalone web viewer restart/troubleshooting
- Add guidance and documentation for modern viewer in WEB_VIEWER.md and docs/
- Various code structure and import improvements to core bot and command modules to support integration
- Add ACL support for sensitive commands
- Example config updates
Benefits:
- Decouples monitoring/UI from bot core process
- Enables real-time browser dashboard and unified contact/repeater tracking
- Easier integration, dev, and troubleshooting
agessaman