Plan rule 3c said every module this refactor creates is type-checked from its first commit; web_viewer/tracking.py was on the ignore list because it moved out of app.py with app's existing errors, and mypy skipped the bodies of untyped helpers in all the new modules. tracking.py is off the ignore list, and the 30 new modules get check_untyped_defs. The 25 errors that surfaced were all missing annotations: container annotations (paths_by_key, tracking, server_stats, result, daily_by_role, feed params), casts for page/page_size where pagination guarantees ints, FeedSubscriptionsMixin's two host attributes, and an ignore for Flask-SocketIO's request.sid. No runtime change.
_get_tracking_data, _get_contact_detail and _calculate_distance move to modules/web_viewer/tracking.py (ContactTrackingMixin); the database statistics, table info and description, optimization and cache view move to modules/web_viewer/database_admin.py (DatabaseAdminMixin). All nine bodies are verbatim; BotDataViewer inherits both. tracking.py joins app.py on mypy's suppressed list, since the moved code carries the type errors mypy never checked in app.py. app.py keeps validate_sql_identifier and humanize_span imported so those module attributes still resolve. No test patched anything these methods use. app.py drops to about 5,800 lines, from 9,095 at the start of the refactor.
create_default_config held a 340-line config.ini as a string literal in core.py. The text now lives in a packaged template (added to package-data, which the packaging test checks) and the method writes it the same way. The file written on first run is byte-identical to before, checked against the literal's value from the previous commit.
The ACK fix I wrote for the radio-lock work is now upstream and released
(meshcore_py#108, v2.3.13). Before it, send_msg_with_retry reported ACKed DMs as
failures: the ACK subscription was registered only after send_msg returned, so
an ACK queued right behind MSG_SENT was dispatched with no listener, and each
attempt accepted only its own ACK code, so a late ACK answering an earlier
attempt was ignored. The bot logged "no ACK received after retries" and skipped
the delivery bookkeeping for messages the recipient had in fact received. Pin
2.3.14, which also fixes send_cmd's destination-type handling.
Two fixes from re-reading the merged radio-lock change:
A region-scoped channel message now restores global flood even when
set_flood_scope raises. The restore only ran in the finally around the send, so
a set that raised left the device pinned to that region and every later send
went out under it.
Drop the __setattr__ left behind when _SerializedCommands became
_serialize_command_frames; it sat after the function's return, unreachable, and
still referenced the proxy's _commands slot.
Closes#279.
A MeshCore client with no region configured sends every channel message as
a plain FLOOD, which every repeater on the mesh rebroadcasts. This adds two
things: free observation of how much of that the bot hears, and an opt-in
warning to the senders.
Observation classifies each channel message as scoped, global or unknown and
tallies it per channel per local day. It costs one upsert and no airtime, and
it is what lets an operator see the size of the problem before deciding to
spend airtime on it. The classification runs ahead of the flood_scopes
allowlist, because an unscoped message is exactly what that allowlist drops.
Warnings only ever fire on positive RF evidence of an unscoped FLOOD. Absence
of correlation is not proof that a sender omitted a region, so it classifies
as unknown and stays quiet. They are off by default, start in dry run, and are
fenced by min_unscoped_messages, a per-sender cooldown, a mesh-wide cooldown
and a daily cap on attempts. Dry run consumes the same budget it previews, so
the log is what going live would put on the air, not an upper bound. All three
limits read from the database, so a restart cannot release a burst.
Channel-delivered warnings go out at global scope on purpose: the recipient is
by definition outside any region the bot replies under.
New Settings -> Region Warnings page in the web viewer covers all of it. Its
dark-mode striped rows exposed a pre-existing base.html bug where Bootstrap's
light-theme text color survived on a dark row background (~1.3:1); fixed there
for every table in the app.
- Extracted `modules/alert_format.py` as the single NWS alert formatter, replacing four copies of the event-type abbreviation table and two of the time compactor. `!wx alerts` and the proactive `WeatherService` broadcasts now localize from one code path, so a Russian bot no longer answers `!wx alerts` in English while its proactive alerts are Russian.
- Stopped leaking translation key paths into mesh broadcasts. `Translator.translate` returns the dotted key when a lookup misses in both the locale and the English fallback, which is right for development but reached the air in production: an unclassifiable NWS title rendered as `⚪Hazardous services.weather_service.event_types.Unknown`, an unmapped WMO code as `services.weather_service.weather_descriptions.4`, and an oddly-cased `wind_speed_unit` as `services.weather_service.wind_speed_units.KMH`. `alert_format.translate_or()` carries an English default at each site, and `WeatherService` now normalizes and validates its three `[Weather]` unit settings the way `GlobalWxCommand` already did.
- Fixed alert expiry rendering in every locale. The formatter rendered a timestamp to a string and re-parsed its own output with `(\d+)(AM|PM)` against a hardcoded English month list, so translated months took the wrong branch and truncated mid-string. Times now carry parsed parts and render through a per-locale `common.alerts.time_12h` template — the space before AM/PM was correct (Russian writes "6 дня", not "6дня"); the downstream regex was the bug.
- Restored month abbreviation. `_compact_time` iterated over abbreviations and replaced them in the string instead of mapping full names, so English stopped shortening "June 28" and Russian replaced the "Jun" inside "June", leaving a stray Latin "e" (`июнe 28`). Reuses the existing `common.date_time.month_abbreviations` rather than the duplicate `services.weather_service.months` block.
- Made `!gwx` display units follow `[Weather]` config instead of the response language. Visibility switched on `base_language != 'en'`, so `language = ru` with the default `temperature_unit = fahrenheit` printed Fahrenheit beside kilometers, and `en-GB` was forced to miles. Pressure is a locale convention rather than a metric/imperial split, so each catalog names its own via `commands.gwx.pressure_unit` — previously every non-English locale inherited mmHg from the English catalog, whose `pressure_mmhg` string contained Russian text, giving German and French users Cyrillic pressure units.
- Let localized `H`/`L` labels reach a standard install. `config.ini.example` shipped the three `temperature_*_format` keys uncommented with literal `H:`/`L:`, and a config value always beats the new locale-aware default, so a Russian bot built from the documented example still rendered `H:47°C L:33°C`. The example now uses the `{high_label}`/`{low_label}` placeholders, which were documented in the docstring but not in the file.
- Routed high/low labels through the reply's translator. `_format_high_low` passed `bot.translator`, so with `auto_detect_language` on, an English-default bot answering a Russian sender localized the rest of the line but not `H:`/`L:`. Added `BaseCommand.response_translator` for this, replacing `wx_international`'s reach into the private `_response_translator` ContextVar.
- Fixed a byte-budget overrun in `!gwx`. The guard on the extra conditions block compared a character count against a byte-derived budget while the rest of the function used `_count_display_width`; Cyrillic is two bytes per character, so the block was appended after the budget was spent.
- Reverted nine `commands.gwx` English rewordings that were not localization work, including the configuration hint in `mqtt_weather_no_subscriber` — dropping it left a mis-configured operator with no pointer to the two keys they need.
- Fixed the Russian `visibility` string, which said "км" on the miles key — the same locale/config conflation as the code bug, in the data. Shortened the Russian event-type abbreviations, which were full words consuming a quarter of the 130-byte budget at two bytes per character.
- Added `commands.wx.hourly_not_available`, missing from every catalog so `!wx hourly` printed the raw key path. Predates this branch; found while auditing every translation key the weather modules reference.
- Moved alert strings to `common.alerts.*` and wind directions to `common.wind_directions.*`, since a command and a service both read them.
ruff now checks the whole tree in CI and in `make lint` instead of only
modules/ and tests/. scripts/ and the root-level scripts ship to users
too and were going unlinted; the wider scope flags nothing today, which
makes now the cheap time to widen it.
Coverage floor raised 35 to 50 against an actual 53.21%.
meshcore-cli was the only dependency with no version floor. It is still
required — meshcore_cli.next_cmd backs contact and channel operations
with no equivalent in the meshcore library — so it is pinned rather than
dropped, and the reason is recorded next to it.
Removes the dead [tool.pytest.ini_options] block: pytest.ini takes
precedence, so those settings were never in effect and could only drift
from the ones that are.
Port the observer firmware's neighbours feature into the bot's packet capture
service, by way of meshcore-packet-capture (upstream PRs #42/#43). On a long
interval the bot asks which repeaters it hears directly and records each
confirmed link with its measured SNR.
This is the strongest link evidence the bot collects: a first-party RF
measurement between two full 32-byte public keys. Path inference works from
1-3 byte prefixes with no keys, and complete_contact_tracking.hop_count
over-claims zero-hop (800 claimed vs 68 corroborated on the live database).
modules/neighbors_discovery.py keeps upstream's public names so its fixes and
tests stay portable. Two deliberate divergences:
- No command_lock plumbing. _SerializedCommands in modules/core.py already
serialises and paces every radio command, strictly more than upstream's
reentrant lock did.
- neighbors_collect_scopes defaults off. Upstream's zero-hop scope probe
relies on a neighbour not being a known contact; this bot tracks contacts,
and for a repeater with no stored path the library reaches zero-hop by
calling change_contact_path() then reset_path() -- mutating the device's
contact table per neighbour. Scope requests also hold the radio lock for
their whole round trip (~25s), stalling bot replies. The default cycle is
one command plus a passive listen window, during which the bot stays
responsive.
Evidence lands in neighbor_links and neighbor_observations (migration 22)
rather than mesh_connections, which cannot persist provenance. The viewer
exposes it as evidence=neighbors on /api/mesh/edges and a Neighbours Only
mode on the mesh page, with populated public keys and real SNR; confirmed
neighbours also relabel edges in the combined view and count as
provenance-trusted when framing the initial map.
neighbors_enabled is the single switch. Every enabled broker publishes once
it is on (mqttN_neighbors defaults true; set false to hold one back). The
topic derives from each broker's packets topic with the last segment swapped,
so a templated broker gets meshcore/{IATA}/{PUBLIC_KEY}/neighbors -- the
topic the firmware uses -- instead of an unrelated flat one. A derived
location-routed topic is skipped with a warning when no iata is set, rather
than publishing into meshcore/XYZ/... on a shared namespace. Snapshots are
non-retained: heard_secs_ago is relative to publish time, so a retained copy
would read as current days later.
Also adds a DM-gated `neighbors` command (the 12h interval floor makes
waiting for the scheduler impractical), which acks immediately and reports in
a second message once the window closes.
Requires meshcore >= 2.3.8 for send_node_discover_req / req_regions_sync.
- Added a tracked `LICENSE` file and updated `pyproject.toml` to include license metadata.
- Enhanced `CHANGELOG.md` with recent changes and clarifications.
- Updated `config.ini.example` and related documentation to reflect clamping behavior for numeric limits in `[Feed_Manager]`.
- Improved startup validation to suggest corrections for unknown and misspelled keys.
- Refactored geocoding and HTTP request handling to run off-thread, preventing event loop stalls.
- Added thread safety to cache management in geocoding functions to avoid race conditions.
Promote the pending 0.9.4 release to 1.0.0 and polish the release
metadata before tagging.
- Bump the version in pyproject.toml and the packaging docs assertion.
- Reframe the CHANGELOG entry as the first stable release, and add the
user-visible changes that shipped without entries: flexible command
prefixes, per-channel flood scope, packet-capture payload decoding,
startup config validation, MOWAS region scoping, hops/RSSI
placeholders, help-command channel overrides, webhook readiness, and
NWS coverage detection.
- Backfill CHANGELOG entries for 0.9.1, 0.9.2, and 0.9.3, which were
tagged but never documented, and add compare links for each.
- Credit external contributors for 0.9.1 and 1.0.0.
- Document the opt-in sender-language detection in the upgrade guide,
alongside the existing optional-extras sections.
- build-deb.sh: fail loudly when the version cannot be read from
pyproject.toml instead of falling back to a hardcoded 0.9.1, which
would stamp a release build with an apparent downgrade.
Reworked implementation of the multilingual-response feature (issue #218)
as a reusable, framework-level capability instead of logic bolted onto
HelloCommand.
- modules/lang_detector.py: keyword-first detection (reliable for the short
greetings typical on mesh) with optional langdetect for longer text;
results are constrained to languages that actually have a translation file.
- core.py: cache Translator instances by language (get_translator) so
per-message switching never re-reads files, and expose available_languages()
derived from translations/*.json. Store translation_path; keep cache/path
consistent on config reload.
- base_command.py: task-local translator override via a ContextVar (safe under
asyncio concurrency, no global mutation), plus detect_response_language() and
a respond_in_sender_language() context manager any command can opt into.
- hello_command.py: build the reply inside respond_in_sender_language(); the
only awaited call (send_response) stays outside the override window.
- Gated by [Localization] auto_detect_language (default false); langdetect is
an optional [lang] extra, not a hard dependency.
Tests: unit coverage for the detector and the base-command wiring.
Three independent wins, measured against a 1.4 GB production database
(714k observed_paths, 38.7k mesh_connections):
- Migration 0014: partial covering index on observed_paths for
bytes_per_hop >= 2. The multibyte evidence query was a full table
scan (2.2 s); with the index it reads only the index (77 ms). The
optional days filter is covered too via last_seen in slot 2.
- Compress responses with flask-compress when the client supports it.
/api/mesh/edges alone is 16.3 MB of JSON uncompressed, 3.9 MB
gzipped. Falls back gracefully (with a warning) if the package is
not installed.
- Fetch stats, edges, and nodes concurrently on the mesh page instead
of three serialized awaits. Node prefixes are now computed client-side
from public_key at the edge prefix length, which removes the
edges-before-nodes ordering dependency. Rendering still waits for
stats so initial framing can use the bot location.
Also call map.invalidateSize() before the initial fitBounds: if the
map was created while its container had no layout (e.g. a background
tab), Leaflet's cached zero width made fitBounds zoom to max.
- Updated the Makefile and GitHub Actions workflow to install ruff version 0.15.15, ensuring consistent linting behavior across environments.
- Added a required-version entry for ruff in pyproject.toml to prevent drift in lint rules.
- Modified the base_service.py method to return None instead of passing, improving clarity.
- Removed an unnecessary blank line in the test_packet_capture_transport_reconnect.py file.
Add a rain nowcast (Open-Meteo 15-minutely precipitation — worldwide, no API
key) as both an on-demand `rain`/`nowcast` command and an opt-in Weather_Service
push that announces rain starting and stopping at the bot's position.
- modules/commands/rain_command.py — command plus pure, unit-tested
fetch/analyze/dedup/label helpers (also reused by the service)
- modules/service_plugins/weather_service.py — background poller mirroring the
existing weather-alert poll pattern; ships disabled (opt-in)
- Location labels resolve to "City, ST" (US) / "City, Country" (non-US)
- 34 unit tests; rain_command added to the strict-mypy module list
- Changed the documentation URL in the meshcore-bot.service file to point to the correct repository.
- Updated the project version in pyproject.toml from 0.9.1 to 0.9.2 for consistency in versioning.
- Modified the User-Agent string in dadjoke_command.py to reflect the new repository URL.
Updated the project version in pyproject.toml to 0.9.1 and adjusted the version retrieval logic in build-deb.sh to reflect this change. This ensures consistency across the project versioning.
- Rewrite test_subscribe_packets/messages_emits_status_ack to match the
silent subscription UX from 1ee84f2.
- Reconcile Python version: requires-python>=3.10, ruff target py310, CI
matrix adds 3.13, pyupgrade UP0xx ignored pending a separate typing-rewrite
PR; fix two B905 zip(strict=...) lints.
- Issue #80 fix in find_recent_rf_data: return None when correlation_key is
provided but unmatched; prefer the longest observed path among samples
sharing a packet_hash; narrow the no-key fallback to a configurable
rf_fallback_window (default 2s).
- Issue #161: lower shipped max_response_hops default 10 -> 7.
- Add CHANGELOG.md, restructure BUGS.md around a ## v0.9.0 Fixed Bugs
table, prune crossed-out duplicate outstanding rows, and add a
Deferred-from-v0.9.0 triage section to TODO.md.
- Untrack coverage.json and add it to .gitignore.
Made-with: Cursor
Added a new PendingMessageEntry TypedDict to improve the structure of pending messages. Updated type hints throughout the MessageHandler class for better clarity and consistency, including the initialization method and message processing functions. This refactor enhances code readability and maintainability without altering existing functionality. Additionally, included the new message_handler module in the pyproject.toml for better module organization.
Introduced optional timeout settings in the configuration for various web viewer operations, including edge and node post timeouts, SQLite connection timeout, and requeue timeout. Updated the web viewer integration to utilize these settings, enhancing flexibility and reliability. Added commands to inspect the resolved configuration with sensitive keys redacted, and updated documentation accordingly.
Bumped the meshcore dependency version from 2.2.31 to 2.3.6 in both pyproject.toml and requirements.txt to ensure compatibility with the latest features and improvements. This change helps maintain the project's dependency integrity.
- Introduced a new `maintenance` module to handle data retention, log rotation, and nightly email tasks.
- Updated the `scheduler` to utilize the `MaintenanceRunner` for executing maintenance tasks, improving code organization and clarity.
- Enhanced documentation to reflect changes in logging configuration and data retention processes.
- Adjusted tests to accommodate the refactored scheduler methods and ensure proper functionality.
- Expanded package data in `pyproject.toml` to include additional static files for the web viewer.
- Removed testing dependencies from `requirements.txt`, noting that they are now included in pyproject extras.
- Updated the base HTML template to reference the manifest from the Flask static folder.
- Modified the build script to exclude `.cursor` files and updated documentation URLs to the correct repository.
- Changed the installation command in the build script to use `requirements.txt` instead of the previous method.
ruff: fix import order, Dict→dict, and unused variable in discord_bridge_service.py
mypy: add types-requests to test deps to resolve import-untyped warnings;
add per-module ignore_errors overrides for modules not yet brought up to
strict typing standard
ShellCheck: remove unused vars (SC2034) in install-service.sh and
uninstall-service.sh; split local declare/assign (SC2155) in
install-service.sh and restart_viewer.sh; replace ls|grep (SC2010) with
glob loops in docker-setup.sh; capture $? immediately after heredoc (SC2320)
Add four jobs to .github/workflows/test.yml:
- lint: ruff check modules/ tests/ — zero violations enforced
- typecheck: mypy modules/ with incremental strict mode; per-module
disallow_untyped_defs where applicable
- lint-frontend: ESLint (eslint-plugin-html) + HTMLHint on templates/
- lint-shell: ShellCheck --severity=warning on all .sh files
Add [tool.ruff] and [tool.mypy] sections to pyproject.toml.
Add .eslintrc.json, .htmlhintrc, package.json for frontend tooling.
- pytest-timeout>=2.1.0 added; timeout=30s per test prevents runaway
tests from hanging CI
- asyncio_mode=auto in pyproject.toml [tool.pytest.ini_options]; async
tests run without per-test markers
- fail_under=27 in [tool.coverage.report] as the enforced coverage
floor; target 40% tracked in TASK-14
- CI test matrix updated for Python 3.9, 3.11, 3.12
- Updated `meshcore` dependency version to `2.2.14` in both `pyproject.toml` and `requirements.txt`.
- Added multi-byte path support in the `PathCommand`, allowing for 1-, 2-, and 3-byte-per-hop paths.
- Enhanced `MessageHandler` to utilize `routing_info` for accurate path extraction and validation.
- Improved path extraction methods in `MultitestCommand` and `TestCommand` to prefer `routing_info` for node IDs.
- Refactored path handling logic across various commands to ensure consistent multi-byte path processing.
- Added new pages to the documentation navigation, including Docker, Upgrade, and Guides sections.
- Updated mkdocs.yml to include the mkdocs-exclude plugin for better content management.
- Modified pyproject.toml to include mkdocs-exclude as a dependency.
- Adjusted GitHub Actions workflow to install the new dependency.
- Removed outdated link from the index.md documentation page.
- Enhanced .gitignore to allow test files in the tests/ directory and committed pytest.ini for test discovery.
- Added checks for missing sections in configuration files, specifically for Admin_ACL and Banned_Users, to prevent errors during bot startup.
- Updated generate_website.py and command_manager.py to handle cases where required sections are absent, returning empty lists instead of raising exceptions.
- Introduced optional dependencies for testing in pyproject.toml, ensuring a smoother development experience.
- Improved localization handling in core.py to default to English when the Localization section is missing, enhancing user experience.
- Deleted COMMANDS.md, DOCKER.md, WEB_VIEWER.md, and other related documentation files to streamline the project structure.
- Updated README.md and other documentation references to point to the new locations of the command and Docker instructions.
- Added optional dependencies for documentation generation in pyproject.toml.
- Refactored bot's main execution flow to use asyncio.run() with improved signal handling for graceful shutdown on Unix systems.
- Added new dependencies in pyproject.toml: urllib3, paho-mqtt, cryptography, and pynacl for enhanced functionality.
- Updated Nix flake to include flake-parts for better modularization and added translation path for NixOS module compatibility.
- Improved argument parsing in web viewer to maintain clarity and consistency.
This commit adds comprehensive Nix/NixOS support and modernizes the
Python packaging structure to enable declarative system-wide deployments.
Added complete Nix flake infrastructure in nix/:
- packages.nix: Package definition with all dependencies
- nixos-module.nix: NixOS module for declarative service configuration
- nixos-test.nix: Comprehensive NixOS VM tests (basic + web viewer)
- shell.nix: Development shell with all build tools
The NixOS module (services.meshcore-bot) provides:
- Automatic user/group creation (meshcore-bot:meshcore-bot)
- Serial port access (dialout group membership)
- Systemd service with security hardening
- Automatic directory management via StateDirectory/LogsDirectory
- INI config generation from Nix attribute sets
- Sensible defaults for system paths
Example NixOS configuration:
services.meshcore-bot = {
enable = true;
settings = {
Connection.connection_type = "serial";
Connection.serial_port = "/dev/ttyUSB0";
Bot.bot_name = "MyBot";
};
};
Migrated from requirements.txt to pyproject.toml:
- Defined proper package metadata and dependencies
- Created entry points: meshcore-bot, meshcore-viewer
- Enables standard 'pip install -e .' workflow
- Maintains compatibility with existing setups
Changes to existing code are minimal and only where necessary:
1. **Added --config parameter** (meshcore_bot.py, app.py)
- Wrapped main logic in main() function for entry point
- Added argparse to accept --config parameter
- Required for NixOS to pass generated config from /nix/store
- Backwards compatible: defaults to 'config.ini' in current directory
- No changes to core bot logic
2. **Fixed web viewer subprocess spawning** (modules/web_viewer/integration.py)
- Changed from sys.executable to 'meshcore-viewer' entry point
- Ensures Nix wrapper script sets up correct PYTHONPATH
- Resolves Flask module import issues in Nix environment
- No functional change for traditional installations
Note: The dev branch's resolve_path() utility already handles both
relative and absolute paths correctly, making it compatible with
NixOS system paths (/var/lib, /var/log) without additional changes.
All NixOS VM tests passing:
- nixos-module-basic: Service startup, file creation, TCP connection
- nixos-module-webviewer: Flask available, port listening, HTTP responses
No breaking changes to existing workflows. The bot continues to work
exactly as before when run with traditional Python methods.
agessaman