Closes#279.
A MeshCore client with no region configured sends every channel message as
a plain FLOOD, which every repeater on the mesh rebroadcasts. This adds two
things: free observation of how much of that the bot hears, and an opt-in
warning to the senders.
Observation classifies each channel message as scoped, global or unknown and
tallies it per channel per local day. It costs one upsert and no airtime, and
it is what lets an operator see the size of the problem before deciding to
spend airtime on it. The classification runs ahead of the flood_scopes
allowlist, because an unscoped message is exactly what that allowlist drops.
Warnings only ever fire on positive RF evidence of an unscoped FLOOD. Absence
of correlation is not proof that a sender omitted a region, so it classifies
as unknown and stays quiet. They are off by default, start in dry run, and are
fenced by min_unscoped_messages, a per-sender cooldown, a mesh-wide cooldown
and a daily cap on attempts. Dry run consumes the same budget it previews, so
the log is what going live would put on the air, not an upper bound. All three
limits read from the database, so a restart cannot release a burst.
Channel-delivered warnings go out at global scope on purpose: the recipient is
by definition outside any region the bot replies under.
New Settings -> Region Warnings page in the web viewer covers all of it. Its
dark-mode striped rows exposed a pre-existing base.html bug where Bootstrap's
light-theme text color survived on a dark row background (~1.3:1); fixed there
for every table in the app.
Port the observer firmware's neighbours feature into the bot's packet capture
service, by way of meshcore-packet-capture (upstream PRs #42/#43). On a long
interval the bot asks which repeaters it hears directly and records each
confirmed link with its measured SNR.
This is the strongest link evidence the bot collects: a first-party RF
measurement between two full 32-byte public keys. Path inference works from
1-3 byte prefixes with no keys, and complete_contact_tracking.hop_count
over-claims zero-hop (800 claimed vs 68 corroborated on the live database).
modules/neighbors_discovery.py keeps upstream's public names so its fixes and
tests stay portable. Two deliberate divergences:
- No command_lock plumbing. _SerializedCommands in modules/core.py already
serialises and paces every radio command, strictly more than upstream's
reentrant lock did.
- neighbors_collect_scopes defaults off. Upstream's zero-hop scope probe
relies on a neighbour not being a known contact; this bot tracks contacts,
and for a repeater with no stored path the library reaches zero-hop by
calling change_contact_path() then reset_path() -- mutating the device's
contact table per neighbour. Scope requests also hold the radio lock for
their whole round trip (~25s), stalling bot replies. The default cycle is
one command plus a passive listen window, during which the bot stays
responsive.
Evidence lands in neighbor_links and neighbor_observations (migration 22)
rather than mesh_connections, which cannot persist provenance. The viewer
exposes it as evidence=neighbors on /api/mesh/edges and a Neighbours Only
mode on the mesh page, with populated public keys and real SNR; confirmed
neighbours also relabel edges in the combined view and count as
provenance-trusted when framing the initial map.
neighbors_enabled is the single switch. Every enabled broker publishes once
it is on (mqttN_neighbors defaults true; set false to hold one back). The
topic derives from each broker's packets topic with the last segment swapped,
so a templated broker gets meshcore/{IATA}/{PUBLIC_KEY}/neighbors -- the
topic the firmware uses -- instead of an unrelated flat one. A derived
location-routed topic is skipped with a warning when no iata is set, rather
than publishing into meshcore/XYZ/... on a shared namespace. Snapshots are
non-retained: heard_secs_ago is relative to publish time, so a retained copy
would read as current days later.
Also adds a DM-gated `neighbors` command (the 12h interval floor makes
waiting for the scheduler impractical), which acks immediately and reports in
a second message once the window closes.
Requires meshcore >= 2.3.8 for send_node_discover_req / req_regions_sync.
- Introduced a locking mechanism to ensure journal mode is initialized once per configuration section, preventing redundant database operations.
- Added a validation function to check for required repeater tables, raising an error with actionable messages if any are missing.
- Updated the DBManager to handle journal modes more efficiently, ensuring that only persistent modes are applied across connections.
- Refactored the RepeaterManager to utilize the new validation function, improving error handling during initialization.
- Enhanced tests to cover new journal mode behaviors and validate repeater table existence, ensuring robust database management.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
- Rewrite test_subscribe_packets/messages_emits_status_ack to match the
silent subscription UX from 1ee84f2.
- Reconcile Python version: requires-python>=3.10, ruff target py310, CI
matrix adds 3.13, pyupgrade UP0xx ignored pending a separate typing-rewrite
PR; fix two B905 zip(strict=...) lints.
- Issue #80 fix in find_recent_rf_data: return None when correlation_key is
provided but unmatched; prefer the longest observed path among samples
sharing a packet_hash; narrow the no-key fallback to a configurable
rf_fallback_window (default 2s).
- Issue #161: lower shipped max_response_hops default 10 -> 7.
- Add CHANGELOG.md, restructure BUGS.md around a ## v0.9.0 Fixed Bugs
table, prune crossed-out duplicate outstanding rows, and add a
Deferred-from-v0.9.0 triage section to TODO.md.
- Untrack coverage.json and add it to .gitignore.
Made-with: Cursor
- Updated type hints to use `|` for union types instead of `Optional` where applicable, enhancing readability and consistency.
- Cleaned up import statements by removing unnecessary imports and organizing them for better clarity.
- Adjusted function signatures in various modules to reflect the new type hinting style, improving type safety and code maintainability.
These changes contribute to a more modern and consistent codebase, aligning with current Python typing practices.
- Updated TODO.md with the latest status and coverage metrics.
- Added date and datetime adapters for SQLite in db_manager.py to improve date handling.
- Refactored repeater_manager.py to ensure atomic database operations within transactions, enhancing data integrity.
- Implemented CSRF protection and security headers in web_viewer/app.py to safeguard against cross-origin requests and improve response security.
- Enforced authentication for non-loopback interface binding in web_viewer/integration.py to prevent unauthorized access.
These changes enhance the overall security, reliability, and maintainability of the application.
- Introduced validation for SQLite journal modes in DBManager, defaulting to "WAL" for invalid inputs and logging warnings.
- Added a new utility function in security_utils for validating SQL identifiers to prevent SQL injection.
- Updated BotDataViewer to utilize the new journal mode validation and SQL identifier checks, ensuring safer database operations.
- Enhanced test coverage for restore functionality, including checks for backup directory configuration and path traversal prevention.
- Added `_apply_sqlite_pragmas` method in `DBManager` to configure SQLite connection settings such as foreign keys, busy timeout, and journal mode.
- Updated `connection` methods in `DBManager` and `BotDataViewer` to utilize the new pragma settings.
- Introduced validation functions in `db_migrations.py` to ensure proper identifier formats and table existence checks.
- Created new migration functions for managing `packet_stream` and repeater-related tables, ensuring they are created and indexed correctly.
- Removed redundant table initialization code from `RepeaterManager` and `BotDataViewer`, relying on migrations for table setup.
- Enhanced tests to verify the creation of repeater tables and indexes during migrations.
- Replaced direct SQLite connection calls with a context manager in various modules to ensure proper resource management and prevent file descriptor leaks.
- Introduced a new `connection` method in `DBManager` to standardize connection handling.
- Updated all relevant database interactions in modules such as `feed_manager`, `scheduler`, `commands`, and others to utilize the new connection method.
- Improved code readability and maintainability by consolidating connection logic.
- Introduced `execute_query_on_connection` and `execute_update_on_connection` methods in DBManager to facilitate executing queries and updates on existing database connections.
- Enhanced the `_recalculate_distance_if_needed`, `_get_location_by_public_key`, and `_get_location_by_prefix` methods in MeshGraph to support optional database connection and location caching, improving performance during batch operations.
- Added new configuration options for path selection presets and proximity methods in config.ini.example, allowing users to customize routing behavior.
- Implemented a new database table for storing observed paths, enabling better tracking of paths from advertisements and messages.
- Updated MessageHandler to store complete paths in the observed_paths table, improving path validation and selection accuracy.
- Enhanced the PathCommand class to utilize new proximity and recency settings, optimizing repeater selection based on user-defined criteria.
- Improved web viewer functionality to display multiple paths for contacts, enhancing user experience and interaction with path data.
- Added a new mesh graph feature for improved path validation, allowing for enhanced routing accuracy.
- Introduced configuration options for recency decay half-life and graph-based validation settings in config.ini.example.
- Updated the PathCommand class to utilize graph-based selection methods, combining graph and geographic scores for better repeater selection.
- Implemented new methods in MessageHandler to update the mesh graph with advertisement paths and trace packet data.
- Created a new database table for mesh connections to support graph-based path validation.
- Enhanced web viewer integration to display mesh graph updates in real-time, improving user interaction and monitoring capabilities.
- Introduced a new database table 'unique_advert_packets' for tracking unique advert packets by their hash.
- Enhanced the RepeaterManager to handle unique packet tracking during daily advertisement statistics.
- Updated StatsCommand to include a new subcommand for displaying the leaderboard of nodes with the most unique advert packets in the last 24 hours.
- Modified translations to support the new advert statistics feature, ensuring user-friendly command descriptions and error messages.
- Introduced a new get_connection method in DBManager to standardize database connection creation with proper configuration.
- Updated all instances in GreeterCommand where sqlite3.connect was used to utilize the new get_connection method, improving code consistency and maintainability.
- Implemented periodic updates of system health status to the database every 30 seconds.
- Added a new method to aggregate and return comprehensive health status of all components, including core connection, database, services, and web viewer.
- Introduced an API endpoint at /api/system-health to retrieve system health data, providing a structured response for health status and component details.
- Enhanced error handling for health data storage and retrieval processes, ensuring robust logging and feedback in case of issues.
- Modified database connection calls in DBManager, FeedManager, and MessageScheduler to convert db_path to a string and set a timeout of 30 seconds, improving reliability and performance of database operations.
- Enhanced error logging to include detailed information about database path existence and permissions, aiding in debugging and ensuring smoother operation.
- Add Flask + Flask-SocketIO web viewer with dashboard (modules/web_viewer/app.py and related)
- Add web viewer templates: index, realtime, tracking (contacts), cache, purging, stats (modules/web_viewer/templates/)
- Add integration hooks and utility functions for web viewer (modules/web_viewer/integration.py, modules/utils.py)
- Add command to launch web viewer from bot CLI (modules/commands/webviewer_command.py)
- Update .gitignore: ignore db/log files, test scripts, and web viewer artifacts
- Add restart_viewer.sh helper script for standalone web viewer restart/troubleshooting
- Add guidance and documentation for modern viewer in WEB_VIEWER.md and docs/
- Various code structure and import improvements to core bot and command modules to support integration
- Add ACL support for sensitive commands
- Example config updates
Benefits:
- Decouples monitoring/UI from bot core process
- Enables real-time browser dashboard and unified contact/repeater tracking
- Easier integration, dev, and troubleshooting
agessaman