Four code paths that produced no result, or a wrong one, without ever
logging an error. Each looked correct in review and failed only against
real inputs.
- packet_capture: read RAW_DATA from the "payload" field. meshcore's
reader dispatches RAW_DATA as {"SNR", "RSSI", "payload"}, but the
handler looked for "data" and "raw_hex" and returned early when it
found neither, so every RAW_DATA event was dropped with no log line.
Existing tests passed because they fabricated {"data": ...}. The
sibling RX_LOG handler already preferred "payload".
- packet_capture: key both RF correlation caches on uppercase hex.
RX_LOG_DATA cached prefixes in the lowercase the wire supplies, while
RAW_DATA uppercased before looking up, so the two never correlated and
no RAW_DATA packet ever picked up cached SNR/RSSI. This was masked
until now by the drop above. The event's own SNR/RSSI are also folded
into the lowercase spelling the rest of the pipeline reads, via an
explicit None check so a legitimate 0 is not discarded.
- path_inference: run the bot path-validation bonus. Its scoring block
sat inside an `if (len(decoded_path_hex) % path_n) != 0` branch, and
for a uniform-width path that remainder is always 0, so the body was
unreachable. A candidate with a perfect historical path match scored
0.0 instead of 0.3. The web sibling was indented correctly.
- path_inference: compare path segments case-insensitively in the web
variant. decoded_nodes came from the uppercased path_context while
stored_nodes were lowercased, so a segment match was impossible for
any path containing a hex letter; a perfect match scored 0.15 rather
than 0.3. This raises some web decode confidence values, so the
module docstring's byte-for-byte parity note now records the
deliberate exception. No node resolves to a different repeater purely
from case.
Also resolves node hop position by index rather than by value, so a
1-byte prefix appearing twice in one path no longer gives every
occurrence the neighbours and final-hop status of the first. This is
threaded through the shared engine and used by the web decode path,
which returns a list. The bot `path` command deliberately does not pass
it: repeater_info is keyed by node_id, so resolving each occurrence
separately would let the later hop overwrite the earlier one's display.
Fixing that needs repeater_info re-keyed by hop index.
Four defects in how configuration is written and displayed. All are
reachable without authentication when web_viewer_password is unset,
which is an explicitly supported setup.
- ini_writer: reject sections, keys and values that cannot survive the
round-trip. update_ini_values() wrote free text straight into the
file, so a newline in any value ended the key's line and everything
after it was re-parsed as INI on the next load. Saving a greeting of
"Welcome!\n[Injected]\npwned = true" through the plugin settings
endpoint created a real new section; a repeated section name bricked
startup with DuplicateSectionError. The check lives at the writer
because three separate callers reach it, and raises IniValueError so a
bad payload is a 400 instead of a half-written file. DEFAULT is
refused as well: its keys apply to every section, and
ConfigParser.add_section('DEFAULT') raises.
- settings_store, web_viewer: persist to disk before mirroring into the
in-memory config. The mirror ran first, so a value rejected by the
writer stayed live in the running process despite never reaching
disk. Both routes taking free-form input (the zombie and offline
alert emails) had the same ordering. The zombie route also caught
only OSError, so an IniValueError there escaped as a 500 rather than
the intended 400.
- config_snapshot: redact Discord webhook URLs. Neither
discord_webhook_urls nor the [DiscordBridge] bridge.<channel> keys
matched any redaction rule, so --show-config and /admin/config
printed live webhook secrets in full; anyone holding one can post to
the channel. Telegram's api_token was already redacted. The
bridge. prefix needs its own rule because the varying part is the
channel name, leaving no fixed stem for the substring match.
- mqtt_weather, packet_capture: verify TLS certificates by default.
Both called tls_set(cert_reqs=ssl.CERT_NONE) unconditionally, so the
broker username and password sent immediately afterwards were
readable by anyone able to intercept the connection.
BREAKING: brokers presenting self-signed certificates now fail to
connect until tls_insecure (mqtt_weather) or mqttN_tls_insecure
(packet_capture) is set to true. Both log a warning while enabled.
- Updated the NominatimRateLimiter to ensure thread-safe request recording and management.
- Refactored the wait_for_request and wait_and_request methods to share a single reservation mechanism for both async and sync callers.
- Enhanced the rate-limited geocoding functions to utilize the new request reservation method, ensuring proper request handling without unnecessary recording.
- Added tests to verify the correct behavior of the new request handling logic and its interaction between async and sync contexts.
- Added a tracked `LICENSE` file and updated `pyproject.toml` to include license metadata.
- Enhanced `CHANGELOG.md` with recent changes and clarifications.
- Updated `config.ini.example` and related documentation to reflect clamping behavior for numeric limits in `[Feed_Manager]`.
- Improved startup validation to suggest corrections for unknown and misspelled keys.
- Refactored geocoding and HTTP request handling to run off-thread, preventing event loop stalls.
- Added thread safety to cache management in geocoding functions to avoid race conditions.
Reworked implementation of the multilingual-response feature (issue #218)
as a reusable, framework-level capability instead of logic bolted onto
HelloCommand.
- modules/lang_detector.py: keyword-first detection (reliable for the short
greetings typical on mesh) with optional langdetect for longer text;
results are constrained to languages that actually have a translation file.
- core.py: cache Translator instances by language (get_translator) so
per-message switching never re-reads files, and expose available_languages()
derived from translations/*.json. Store translation_path; keep cache/path
consistent on config reload.
- base_command.py: task-local translator override via a ContextVar (safe under
asyncio concurrency, no global mutation), plus detect_response_language() and
a respond_in_sender_language() context manager any command can opt into.
- hello_command.py: build the reply inside respond_in_sender_language(); the
only awaited call (send_response) stays outside the override window.
- Gated by [Localization] auto_detect_language (default false); langdetect is
an optional [lang] extra, not a hard dependency.
Tests: unit coverage for the detector and the base-command wiring.
- Enhanced the `/api/contacts` endpoint to support pagination, allowing users to specify `page` and `page_size` parameters.
- Added search functionality to filter contacts based on a search term.
- Implemented sorting options for the contacts list, enabling users to sort by various fields.
- Updated the frontend to reflect these changes, including a new pagination UI and improved loading states.
- Introduced caching for multibyte hop chunks to optimize performance when retrieving contact badge evidence.
- Added tests to ensure the new features work as expected and maintain existing functionality.
Finish the lint/type cleanup so all CI gates pass:
- location.py: declare lat/lon as Optional[float] up front. The
fallback/coordinates/repeater branches bind plain floats while the
zipcode/city branches bind float | None from best-effort geocoding, so
a single Optional declaration keeps mypy from pinning the first binding.
- rain_command.py: sort imports to ruff/isort order and mark the
US_STATE_ABBRS / titlecase_location re-exports noqa: F401. They are
listed in __all_location_reexports__ rather than __all__, so ruff does
not recognize them as intentional re-exports on its own.
- test_location_characterization.py: drop a stray blank line (ruff E303).
Resolve the feeds.html conflict by keeping the branch's XSS-safe DOM
construction of the feed-details view and re-adding dev's per-feed and
reset-all error buttons via addEventListener instead of inline onclick,
since /feeds is a nonce-CSP page where inline handlers are blocked.
Update tests/test_feed_manager_extended.py::TestPollFeedPosting to patch
the SafeUrlPolicy.validate_async path (this branch's SSRF refactor) rather
than the removed module-level validate_external_url symbol, matching the
idiom already used throughout that test file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_check_address now canonicalizes IPv4-mapped IPv6 addresses (e.g.
::ffff:169.254.169.254) to their embedded IPv4 target before the metadata
and non-unicast checks. Previously the mapped form was a distinct address
object absent from _METADATA_ADDRESSES with is_reserved=False/is_global=False,
so under allow_private=True it slipped past every check and the socket layer
still dialed the mapped IPv4 metadata endpoint. test_allow_private_does_not_
allow_metadata is parametrized over the plain and mapped spellings.
Also cast SafeAiohttpResolver.resolve's ResolveResult host/port to str/int to
satisfy aiohttp's TypedDict (fixes the two mypy errors in the strict-overrides
CI gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Added asynchronous support for location resolution, improving performance and responsiveness.
- Introduced detailed error handling for invalid latitude and longitude inputs, providing specific feedback to users.
- Implemented a caching mechanism for geocoding results to optimize repeated lookups.
- Refactored the AQI command to streamline location handling and improve clarity in error messages.
- Expanded unit tests to cover new geocoding features and error scenarios, ensuring robust functionality.
- Introduced a shared API for location resolution using `modules.location`, allowing for better handling of place lookups (coordinates, ZIP codes, city names).
- Updated geocoding functions to accept both strings and structured dictionaries, improving flexibility in location queries.
- Refactored existing commands (AQI, Rain, Wx) to utilize the new location resolution methods, streamlining the codebase and enhancing maintainability.
- Removed redundant location handling logic from commands, centralizing functionality in the new location module.
- Added tests to ensure proper classification and resolution of various location formats, including multi-word international cities and ZIP codes with surrounding whitespace.
- Added max_posts_per_check to limit the number of items posted per check, defaulting to max_items_per_check for backward compatibility.
- Updated logic in FeedManager to process new items, ensuring that only the specified number of posts are made while examining a larger set of items.
- Enhanced emoji selection to prioritize per-item emojis from the API, falling back to heuristics based on feed names when absent.
- Introduced new truncation and substring functions for formatting messages, improving text handling in feed outputs.
- Added API endpoints to reset feed error counts globally or for individual feeds, enhancing error management capabilities in the web viewer.
- Added a mechanism to check the byte length of direct messages (DMs) before sending, ensuring they do not exceed the firmware's maximum limit.
- Implemented a method to split oversized messages into smaller chunks, preserving UTF-8 encoding and avoiding mid-codepoint splits.
- Enhanced logging to provide warnings when messages are auto-split, including details on the number of chunks created.
- Updated the DM sending logic to handle both single and split messages efficiently.
- Introduced a new static method for splitting text into UTF-8 chunks, improving message handling across the application.
- Added a mechanism to check the byte length of direct messages (DMs) before sending, ensuring they do not exceed the firmware's maximum limit.
- Implemented a method to split oversized messages into smaller chunks, preserving UTF-8 encoding and avoiding mid-codepoint splits.
- Enhanced logging to provide warnings when messages are auto-split, including details on the number of chunks created.
- Updated the DM sending logic to handle both single and split messages efficiently.
- Introduced a new static method for splitting text into UTF-8 chunks, improving message handling across the application.
- Added a type ignore comment for the assignment of optionxform in config_schema.py to suppress type checker warnings.
- Updated path inference logic to return None when bot latitude or longitude is not set, ensuring safer handling of missing values.
- Enhanced the decoding of path nodes to check for None before appending repeater details, preventing potential errors.
Three independent wins, measured against a 1.4 GB production database
(714k observed_paths, 38.7k mesh_connections):
- Migration 0014: partial covering index on observed_paths for
bytes_per_hop >= 2. The multibyte evidence query was a full table
scan (2.2 s); with the index it reads only the index (77 ms). The
optional days filter is covered too via last_seen in slot 2.
- Compress responses with flask-compress when the client supports it.
/api/mesh/edges alone is 16.3 MB of JSON uncompressed, 3.9 MB
gzipped. Falls back gracefully (with a warning) if the package is
not installed.
- Fetch stats, edges, and nodes concurrently on the mesh page instead
of three serialized awaits. Node prefixes are now computed client-side
from public_key at the edge prefix length, which removes the
edges-before-nodes ordering dependency. Rendering still waits for
stats so initial framing can use the bot location.
Also call map.invalidateSize() before the initial fitBounds: if the
map was created while its container had no layout (e.g. a background
tab), Leaflet's cached zero width made fitBounds zoom to max.
Nodes occasionally carry wrong GPS, which stretched the initial
fit-to-all-nodes view across continents. /api/mesh/stats now exposes
the bot's configured position ([Bot] bot_latitude/bot_longitude), and
the mesh page BFSes the multi-byte-evidence subgraph from repeaters
within 30 km of the bot (nearest one as fallback), framing the initial
map view to that connected component. Falls back to the old fit when
the bot position is unset or the component is trivially small; other
mesh islands stay on the map, just outside the initial frame.
- Hot connections mode now also colors repeater dots with viridis by
their total observation count across touching edges (log-normalized,
same dark-mode ramp compression as edges); node popup gains a Path
Observations row and the heat legend a clarifying note
- Density pass: smaller dot base/cap (5..13 vs 6..16), edge widths
capped at 5px, and both dots and lines scale down to 55% as the map
zooms out (full size from zoom 12 in), rescaled live on zoomend
- Node degree and observation totals are now precomputed once per
filter pass (computeNodeStats) instead of per-node scans over all
edges, using the same endpoint resolution as the renderers
- Multi-byte evidence is now the default mode for the mesh graph
- Mesh display theme: follows the site theme until toggled on the page,
then remembered independently (localStorage); dark CARTO basemap,
dark Leaflet popups/controls/legends, dark graph canvas
- Hot connections: optional viridis edge coloring by log-scaled
observation count with a gradient legend; the near-black cold end of
the ramp is compressed away in dark mode so cold edges stay visible
- Single-row header (title + stat chips + icon controls) replaces the
title and stats-card rows; footer hidden on this page to maximize the
visualization viewport
- Mobile: page scrolls naturally instead of starving the fixed-height
flex container (which stranded the absolutely-positioned legends)
The old slider defaulted to floor(avg observations) — on heavy-tailed data
the average sits above the 75th percentile, so the first visit hid most of
the graph. Worse, the default (e.g. 98) exceeded the slider's max of 50:
the browser clamped the control while the label kept showing the unclamped
value, so the label lied about the active filter.
- slider position now maps logarithmically onto 1..max(observation_count),
rescaled whenever edge data loads, giving fine control at the low end
and full reach into the tail
- first-visit default is the median observation count of the loaded edges
- the persisted value is the threshold itself, so it stays meaningful when
the scale changes (e.g. switching evidence modes)
- label shows the threshold plus live 'shown/total edges' feedback
Single-byte repeater identity guessing has been persistently unreliable,
and mesh_connections flattens away evidence provenance (confirmed_2byte
never persists). observed_paths retains it: bytes_per_hop marks which
paths carry unambiguous multi-byte hops.
- /api/mesh/edges?evidence=multibyte derives edges purely from unique
multi-byte path observations (consecutive hop pairs, aggregated with a
distinct-path count); 2-byte edges coalesce into a 3-byte edge only on
a unique prefix match, mirroring MeshGraph.add_edge
- default mode tags each edge evidence=multibyte|singlebyte by key length
- Evidence filter on the mesh page (persisted with the other filters);
single-byte edges render dashed in map and graph views, with an
evidence line in tooltips and a legend entry
- client-side haversine fallback for edges without a stored distance
- prefix-compatible edge/node matching so degree sizing and node details
work when edge and node prefixes differ in resolution
- /api/mesh/stats reports multibyte_edges; shown on the Edges stat card
Adds device-level companion settings to /radio, headlined by the response
path hash size (mode 0-2 = 1-3 bytes per hop) wired to the existing
firmware config endpoints. New sections: Identity & Adverts (name, advert
lat/lon, location policy, zero-hop/flood advert buttons), Mesh Behavior
(extra ACKs, telemetry permissions), and write-only Advanced Tuning
(RX delay base, airtime factor, sent x1000 per the wire format).
Backend: GET /api/radio/params now returns the full SELF_INFO node fields;
POST accepts the new fields with validation; new POST /api/radio/advert;
scheduler ops handle the writes (other-params group as one read-modify-write
frame so partial updates never clobber device state).
Config-managed settings are not writable from the panel: new-contact mode
is owned by [Bot] auto_manage_contacts and shown read-only; the node name
is locked while bot_name + auto_update_device_name manage it. loop.detect
is removed from the firmware endpoints entirely - it is repeater/room-server
CLI config, and companion custom vars map to sensor settings.
Also: dark-mode styling for disabled form fields (base.html), TASK-01 guard
test updated for the deliberate reintroduction, 23 endpoint/template tests.
The legacy [Section] enabled alias map (e.g. [Jokes] joke_enabled,
[Stats] stats_enabled) was duplicated in BaseCommand.get_config_value and
modules/settings_schema.py, so runtime behavior and the web settings UI
could drift apart. Both now read config_schema.LEGACY_ENABLED_ALIASES.
- meshcore_bot.py: run config validation on normal startup and print
errors/warnings (capped at 15) — misspelled sections/keys previously
failed silently, the top source of configuration confusion
- ini_writer: append new keys directly under a section's last entry
instead of after its trailing blank line
- add .claude/launch.json for browser-preview of the standalone viewer
Verified end-to-end in the browser: /plugins renders 43 command cards,
opt-in commands show off, dynamic-row add/save writes config.ini with a
timestamped backup and preserved comments.
- reload_config: clear all sections before re-reading so keys deleted from
config.ini (e.g. rows removed in the settings UI) don't survive in memory
(ConfigParser.read merges instead of replacing)
- reload_config: re-instantiate command plugins so settings cached in
__init__ (including 'enabled') take effect on reload
- /api/plugins save: treat dynamic_sections / repeating_blocks absent from
the payload as 'don't touch' instead of 'delete all managed keys'
- settings view: respect per-plugin settings_enabled_default so opt-in
commands (Announcements, Greeter) display as off when unconfigured
- ruff: drop unused import, organize command_manager imports
- add tests for ini_writer, settings_schema, settings_store and the
/api/plugins endpoints (38 cases) plus reload regression tests
- Updated `config.ini.example`, `config.ini.minimal-example`, and `config.ini.quickstart` to clarify connection type precedence and required keys for serial, BLE, and TCP connections.
- Added detailed comments and examples for each connection type, ensuring users understand how to configure their connection settings effectively.
- Introduced a new `[Service_Overrides]` section in `config.ini.example` for alternative service plugin implementations, improving extensibility.
- Enhanced documentation in `configuration.md` to reflect these changes and provide clearer guidance on connection options and templates.
- Modified the core service to start the webhook service before establishing a radio connection, reducing the window for connection refusals.
- Implemented a readiness check in the webhook service to return a 503 status when the bot is not connected, ensuring clear communication to external callers.
- Added unit tests to verify the webhook service's behavior when the bot's connection status changes, including rate limiting and readiness responses.
Add optional payload decoding to the packet capture service. GRP_TXT
channel messages are decrypted (sender/text), ADVERTs are parsed
(name/role/lat-lon), and a nested "decoded" object is attached to each
packet alongside the unchanged raw fields.
- Comprehensive channel key store: bot's configured radio channels plus
decode_hashtag_channels, [Channels_List], decode_channel_keys, and the
built-in default Public key.
- Publishing the decoded object to MQTT is off by default and
configurable per broker via mqttN_include_decoded.
- Configurable packet-log rotation (off/size/time) for historical dumps.
The decoder lives in a standalone, dependency-free module
(modules/meshcore_payload_decode.py) so it can be shared verbatim with
the meshcore-packet-capture project.
Closes#197Closes#35
- Updated logging statements in the `MessageHandler` class to use `debug` level instead of `info` for `RAW_DATA` and `NEW_CONTACT` events, reducing log verbosity while maintaining essential information.
- This change enhances the clarity of logs by reserving `info` level for more significant events.
- Resolves issue #214
- Added a `--strict` option to `validate_config.py` to fail on "Unknown section" messages, aimed at CI validation of example configs.
- Updated GitHub Actions workflow to validate shipped example configs against the canonical section list using the new strict mode.
- Expanded the `CANONICAL_NON_COMMAND_SECTIONS` in `config_validation.py` to include additional sections for improved validation accuracy.
- Introduced a regression test to ensure example configs do not trigger unknown section warnings, maintaining consistency with the canonical list.
- Updated `config.ini.example` to support single, multiple, and decorative command prefixes, improving flexibility in command invocation.
- Refactored `CommandManager` and `BaseCommand` to utilize new command prefix parsing logic, allowing for better handling of multiple prefixes and optional bare commands.
- Enhanced documentation to clarify command prefix configuration options and behavior, ensuring users understand how to set and use prefixes effectively.
- Added unit tests to verify the correct functionality of command prefix handling, including support for permissive and strict prefix modes.
- Updated configuration examples to include new `{hops}` and `{hops_label}` placeholders for path command replies, enhancing response detail.
- Modified `BaseCommand` to support hop count retrieval and formatting, ensuring accurate display of hop information in responses.
- Enhanced unit tests to verify correct formatting of hop-related placeholders in response messages, improving overall command functionality.
- Updated the `[Rain_Command]` section in `config.ini.example` to include support for snow alongside rain, improving the command's functionality.
- Enhanced documentation for the rain command to reflect the new snow alias and clarify response behavior based on the selected keyword.
- Added a new `collect_stats` option in the `[Stats_Command]` section, allowing stats collection to be enabled independently of the user-facing command, with updated documentation to explain its behavior.
- Improved the web viewer documentation to clarify how stats are collected and displayed, ensuring users understand the configuration options.
- Updated the `CommandManager` to respect the `channels` override for the help command, ensuring it checks channel permissions directly when processing help requests.
- Refactored the logic to fall back to global `monitor_channels` only when no specific help command is loaded, improving channel-specific access control.
- Added unit tests to verify the correct behavior of the help command in disallowed channels, ensuring it only responds in permitted contexts.
- Updated the `CommandManager` to respect the `help_enabled` flag for the help command, ensuring it only responds when the command is enabled.
- Refactored help request processing to streamline checks for help keywords and channel restrictions.
- Added unit tests to verify the behavior of the help command when enabled and disabled, ensuring proper suppression of responses when the command is not active.
- Deleted the `[Custom_Syntax]` section from `config.ini.example` and `core.py`, simplifying the configuration by removing outdated custom syntax patterns and examples.
- This change streamlines the configuration files, focusing on essential settings and improving clarity for users.
- Enhanced `CommandManager` to persist normalized message content, preventing prefix stripping from interfering with command matching.
- Updated `BaseCommand` to skip prefix handling if already normalized, ensuring all commands can be matched correctly.
- Modified `HelpCommand` to ensure unused commands are listed even when command statistics are present, improving user accessibility to all commands.
- Added unit tests to verify the correct behavior of command matching and help command functionality under various scenarios.
- Introduced per-channel flood scope configuration in `config.ini.example`, allowing for more granular control over message routing.
- Updated `CommandManager` to normalize channel names and resolve flood scopes based on channel-specific settings, improving message delivery accuracy.
- Enhanced `MeshMessage` to consider channel-specific flood scopes when determining the effective outgoing flood scope.
- Added unit tests to verify the new flood scope resolution logic and ensure correct behavior across various scenarios.
- Added detailed diagnostics for database initialization failures in `DBManager`, improving error logging with filesystem checks.
- Implemented a new method `_resolve_viewer_db_path` in `WebViewerIntegration` to determine the database path for the viewer subprocess.
- Introduced `_preflight_database_path` to validate the viewer database path, ensuring the parent directory exists, is writable, and is a directory.
- Added unit tests to verify logging behavior for missing or invalid database paths in both `DBManager` and `WebViewerIntegration`.
- Enhanced the `TestCommand` class to include the `rssi` value in the response format, allowing for more detailed output.
- Updated the example configuration to reflect the new placeholder usage for `rssi`.
- Added a unit test to verify that the `rssi` placeholder expands correctly in the response message.
- Modified `WxCommand` to use its own keywords along with any aliases from `BaseCommand` when delegating to `GlobalWxCommand` for the Open-Meteo weather provider.
- Added a new test fixture for `WxCommand` configured to delegate to `GlobalWxCommand`, ensuring that aliases are preserved and correctly matched in keyword checks.
- Enhanced unit tests to verify the delegation behavior and keyword matching for both standard and aliased commands.
- Added a new function `nws_http_means_no_coverage` to determine if NWS HTTP status codes indicate no coverage for a location.
- Updated `WxCommand` and `WeatherService` classes to utilize this function, improving error handling for NOAA alerts.
- Introduced a lazy loading mechanism for `_nws_alerts_available` to manage alert availability based on NWS coverage status.
- Enhanced logging to provide clearer warnings when NWS alerts are unavailable due to HTTP errors.
agessaman