This commit introduces asynchronous handling for response formatting in the command manager, allowing network-backed filters to resolve off-thread without blocking the event loop. Key changes include:
- Added `check_keywords_async` method to `CommandManager` to handle keyword checks asynchronously.
- Updated `TestCommand` and `PathCommand` to utilize the new async formatting capabilities.
- Enhanced `response_template` with `format_piped_template_async` for improved performance during template rendering.
- Updated documentation to reflect support for async response formatting in both `path`'s `reply_prefix` and the test command's `response_format`.
This change improves responsiveness and reduces potential delays caused by synchronous network calls during message processing.
Follow-up to the PR #254 integration fixes, from an xhigh review of the
merged branch. Five of fourteen findings; the other nine are written up in
.scratch/pr254-deferred-review-findings.md (untracked).
- Re-raise CancelledError from resolve_template_async. gather's
`return_exceptions=True` captures it like any other exception, so the
BaseException branch was swallowing it: a render cancelled during shutdown
logged "shortening failed" and carried on to transmit.
- Warn once per template, not once per render. The unresolved-template
warning runs on the inbound message path, so a shorten_url filter in
Test_Command.response_format wrote a WARNING per matching message,
unthrottled, to a rotating 5 MB log.
- Add a `urlencode` filter. A quoted literal substitutes nested fields
verbatim, which is right for prose and wrong for URLs: `sender` is whatever
a remote node advertises, and an unencoded `&`, `#` or space rewrote the
query, truncated at a fragment, or malformed the link that was then POSTed
to the shortener. `/` is encoded too, since an interpolated field is one
path segment.
- Distinguish the two ways shortening can be unavailable. A template needing
resolution with no config now warns at the resolver instead of returning {}
indistinguishably from "nothing to do"; a pre-pass that ran but could not
shorten logs at debug, since that is transient network trouble rather than
a misconfiguration and must not escalate on the message path.
- Cover the path_command wiring. _format_path_reply_prefix is the only place
the pre-pass is called and had no test, so a dropped `await` or a missing
`shortened=` would have disabled shortening silently.
The warn-once cache is module state, so its test fixture is autouse: without
it a test that renders an unresolved template suppresses the warning in
whichever test runs next, which is an order-dependent failure.
Every template shipped in config.ini.example still renders identically to the
regex parser this replaced.
Integration fixes on top of the merge, in three groups.
Regressions in the shared shortener, which is not opt-in — weather alerts
call it unconditionally whenever an alert carries a link:
- Restore the `response.ok` guard on the v.gd path. Without it a 503 whose
body happens to start with `http` (a captive portal, a CDN error page) was
returned as the short URL. The test that covered this still passed because
it never set `.text`, so `MagicMock.startswith()` answered truthily; it now
supplies a body that would be accepted on a 200, so only the status check
can make it pass.
- Add the same guard to the shlink path, where a rejected API key is a 401
with a problem-details body and was indistinguishable from an empty result.
- Restore debug-level logging for `Timeout`/`ConnectionError`. A mesh node's
uplink drops out routinely and this had become `logger.error`.
- Drop the `shortUrlSlug` fallback. Shlink's create response carries
`shortUrl` and `shortCode`; `shortUrlSlug` is not in its schema, and a bare
slug is not a link — emitting one would have put `abc123` in a message.
The `shorten_url` filter did blocking HTTP inside the synchronous render
path (`process_message` → `check_keywords` → `format_response`), so a 5 s
shortener timeout stalled the radio transport along with everything else.
Making `check_keywords` async would touch 27 call sites, so instead the
network work moves ahead of the render:
- `resolve_template_async()` renders once with the filter in collection mode,
which walks the real chain so a clause already suppressed by `hops_min`
costs no request, then shortens what survived concurrently off-thread.
- The filter itself now only reads that mapping and never calls out. Asked to
render without a pre-pass it warns and drops the clause rather than block.
- On failure the clause is dropped rather than sent unshortened. A v.gd link
is ~19 bytes where the analyzer URL is ~59, against a ~158-byte budget the
prefix is subtracted from, so falling back would have turned one
transmission into two whenever the shortener was unreachable.
Smaller: rename `if_notempty` to `if_nonempty` to match the existing
`prefix_if_nonempty` (nothing ships using it yet, so there is no config to
migrate); `_build_create_shlink_url` no longer takes the long URL and API key
it never used; move the CHANGELOG entry from Fixed to Added and name the
filter it actually added; ungate the render debug log from `config`; document
that `shorten_url` is supported in `path`'s `reply_prefix` only.
Every template shipped in config.ini.example still renders identically to the
regex parser it replaced.
{firstlast_distance|prefix_if_nonempty: | F/L Dist: } prints "| F/L Dist: N/A"
on a direct message: the distance helpers return the literal "N/A" when there
is no path to measure, prefix_if_nonempty only asks whether the value is
non-empty, and "N/A" is. Suppressing that needed a gate, and pathbytes_min was
the only one available—so it was being used as a stand-in for "did this
message take any hops", which is not what it asks. It asks how the path is
encoded, so pathbytes_min:2 also discards a one-byte multi-hop path whose
distance is real and measurable.
hops_min:N asks about the route itself. hops_min:1 drops a clause on a direct
message and nothing else:
hops_min:1 pathbytes_min:2
direct cleared cleared
1-byte 2 hops kept cleared <- the difference
2-byte 2 hops kept kept
unknown cleared cleared
An unknown hop count clears the value: a gate that cannot confirm the route
should suppress rather than guess, matching pathbytes_min.
The hop-count logic moves to utils.message_hop_count so the filter and
BaseCommand.get_hops_display_values share one implementation instead of two.
It returns None for unknown rather than 0, since a gate must not read
"cannot tell" as "direct".
Adds the 16-char MeshCore packet identity hash to the three formatters that
render user-facing templates: [Keywords] responses, the test command's
response_format, and the path command's reply_prefix. It is read only from
routing_info, which is attached solely from an RF packet correlated to the
message, so a hash from an unrelated transmission is never shown. Missing,
empty, and the 0000000000000000 error sentinel all render empty.
get_packet_hash_placeholder returns "" rather than a False sentinel. Empty
gives byte-identical output on every path—prefix_if_nonempty already tests
falsiness and str.format renders "" as nothing—whereas False would print
the literal string "False" into a mesh reply from any consumer of
get_standard_placeholder_fields that formatted it directly.
TestCommand.format_response now builds on get_standard_placeholder_fields
instead of duplicating it. That also honors the extra argument its docstring
already promised but silently ignored, and picks up the path-string hop
fallback, so {hops} reads "0" rather than "?" for a message whose hops and
routing_info are both unset but whose path says "Direct (0 hops)".
Documents the placeholder in config.ini.example, the default config that
core.py writes on a fresh install, the README and docs/path-command-config.md.
The generated config was also missing {elapsed}; added that too.
Total path distance was already a solved problem here: the test command
computes it and exposes {path_distance} through the piped-template
engine, with pipe filters and docs. The path command just had no way to
reach it, since reply_prefix went through plain str.format.
Rather than add a second distance implementation, this reuses what
exists. BaseCommand.get_standard_placeholder_fields now returns the
shared placeholder set so subclasses can extend it, the path command
renders reply_prefix through format_piped_template with an added
{path_distance}, and the sum itself uses utils.calculate_distance over
the nodes the path command has already resolved.
Distance covers sender -> each hop -> bot and is deliberately blank when
the chain cannot be measured end to end (unresolved hop, prefix
collision, missing or 0,0 coordinates, unknown sender, no configured
bot position), so a partial sum is never reported as the real distance.
Because the prefix now supports pipe filters, an empty value takes its
label with it: {path_distance|prefix_if_nonempty:📏 }
Supersedes #198, which added a parallel haversine, a show_path_distance
flag, and a distance_traveled key across 10 locales.
- Implemented caching for multi-byte mesh aggregation, allowing concurrent requests to share computation and reducing CPU load.
- Updated the web viewer to coalesce live edge events and refresh the mesh graph every 30 seconds, improving responsiveness on busy networks.
- Adjusted logging behavior in the web viewer to respect the configured log level, ensuring efficient logging without excessive duplicate entries.
- Added configuration options for mesh graph caching duration, enhancing user control over performance settings.
- Implemented chunked deletion for data retention, allowing for smoother cleanup processes without monopolizing SQLite's writer lock.
- Configured retention settings to delete in batches with pauses, improving performance on SD-card installations.
- Updated Linux service installers to allocate 1GB of memory and 200% CPU, providing better resource management for Raspberry Pi workloads.
- Added detailed configuration examples for Raspberry Pi in documentation to guide users on optimal settings.
- Updated mesh graph path splitting and aggregation to run in SQLite, improving performance by avoiding Python materialization.
- Defaulted graph persistence to batched writes for new installations, reducing WAL churn and SD-card writes.
- Enhanced data retention to execute shortly after startup, independent of the nightly maintenance schedule.
- Added a table-specific index for `mesh_connections` to support window and retention queries, ensuring efficient data access.
- Updated configuration examples to include new `{hops}` and `{hops_label}` placeholders for path command replies, enhancing response detail.
- Modified `BaseCommand` to support hop count retrieval and formatting, ensuring accurate display of hop information in responses.
- Enhanced unit tests to verify correct formatting of hop-related placeholders in response messages, improving overall command functionality.
- Enhanced the command reference with detailed descriptions for `cmd`, `version`, `weather`, and `path` commands, including usage examples and configuration options.
- Added new `RandomLine` command documentation for configurable triggers.
- Updated configuration documentation to reflect the deprecation of the global `[Aliases]` section, encouraging per-command alias definitions.
- Clarified the `[Rate_Limits]` and `[Webhook]` sections in the configuration guide.
- Improved the web viewer documentation, emphasizing security practices and configuration options.
- Introduced `reply_prefix` to prepend a customizable string to path command replies, supporting placeholders for dynamic content.
- Added `minimum_path_bytes` setting to control the resolution of repeater names based on the number of bytes per hop, enhancing path command behavior.
- Updated relevant documentation and translations to reflect these new configuration options.
- Implemented unit tests to ensure correct functionality of the new features.
- Updated `meshcore` dependency version to `2.2.14` in both `pyproject.toml` and `requirements.txt`.
- Added multi-byte path support in the `PathCommand`, allowing for 1-, 2-, and 3-byte-per-hop paths.
- Enhanced `MessageHandler` to utilize `routing_info` for accurate path extraction and validation.
- Improved path extraction methods in `MultitestCommand` and `TestCommand` to prefer `routing_info` for node IDs.
- Refactored path handling logic across various commands to ensure consistent multi-byte path processing.
- Updated `MeshGraph` to support multi-resolution storage of edges, allowing prefixes of 2, 4, or 6 hex chars without truncation.
- Implemented prefix matching logic to ensure distinct links are maintained and accurately retrieved based on prefix queries.
- Refactored methods in `MessageHandler` and `PathCommand` to accommodate variable prefix lengths during graph lookups.
- Enhanced tests to validate prefix match functionality and edge management in the mesh graph.
- Added a new section in the FAQ detailing how to run meshcore-bot on a Raspberry Pi Zero 2 W, including specific configuration settings to manage memory usage effectively.
- Updated the path command configuration documentation to clarify the default values and usage of `graph_startup_load_days` and `graph_capture_enabled`, improving user understanding of these settings.
- Renamed various documentation files to use lowercase for improved consistency, including `WEB_VIEWER.md`, `DOCKER.md`, and `PATH_COMMAND_CONFIG.md`.
- Updated references in `config.ini.example`, `docker-setup.sh`, `README.md`, and multiple documentation files to reflect the new file names.
- Removed obsolete documentation files related to the Discord Bridge, Map Uploader, Packet Capture, and Weather Service, streamlining the documentation structure.
Adam Gessaman