Add a password type to settings_schema so plugin secrets render as masked inputs and never reach the browser. Stored values stay in config.ini; an empty box on save leaves an existing secret unchanged.
[Test_Command] distance_unit (auto/km/mi) controls {path_distance} and {firstlast_distance}. Auto follows the reply language (miles for en/en-US, kilometres otherwise, including en-GB). Elapsed times of 1s and above render as seconds to keep the ack short.
The published packet payload mixed clocks: "timestamp" was UTC ISO 8601
with a Z suffix, but the "time" and "date" fields beside it were rendered
from a naive datetime.now(), i.e. the host's local wall clock. A consumer
reading those two off a bot in a non-UTC zone sees a skew of exactly that
zone's UTC offset and flags the observer's clock as wrong.
Local time was never the intended reading. The original script took time
and date off the firmware log line, and mctomqtt sets the device clock
from calendar.timegm(), so upstream both fields are already UTC.
All three now render one aware UTC instant, so they cannot disagree with
each other or straddle a second boundary. _utc_iso_timestamp() takes an
optional instant to make that possible; its no-arg behavior is unchanged.
The local translation path defaulted to a bare `local/translations/`, resolved
against the process cwd and unrelated to `[Bot] local_dir_path`. Since
`local_dir_path` already selects where an operator's commands, service plugins
and config overlay live, the catalog belongs in that same tree. It now defaults
to `<local_dir_path>/translations`, resolved absolute against the bot root, so
relocating `local_dir_path` moves the catalog with it and the lookup no longer
depends on the working directory. An explicit `local_translation_path` still
wins.
Only the constructor at startup was passing the local path. `get_translator()`
builds and caches a translator per detected language, and `reload_config()`
builds a fresh one, and both were still constructing `Translator` with the
distributed path alone. So local overrides silently vanished from any reply that
used the sender's language, and did not survive a config reload. Both now pass
it, `reload_config()` republishes it alongside `translation_path`, and it is
snapshotted and restored on rollback. The DummyTranslator fallback sets it too,
since `get_translator()` would otherwise raise AttributeError there.
Collapsed `_deep_merge_translations` into the existing `_merge_translations`,
which already merged deeply with the primary winning. Rewrote that one to stop
mutating its input: it shallow-copied `fallback` and then recursed into the
shared sub-dicts, so merging a base language over English corrupted the cached
English catalog at every level below the first. Also replaced the two `print()`
calls on the error paths with logger calls.
Registered `local_translation_path` in the config schema next to
`translation_path`, and reverted a trailing-whitespace reflow that touched 84
lines of config.ini.example for a one-key addition.
Tests: local overlay (single-string override, added keys, local-only catalog,
missing directory), the merge no longer mutating its fallback, the default
following `local_dir_path`, an explicit setting overriding it, the overlay
reaching `bot.translator` end to end, and a reload picking up a moved path.
Addresses the review on #243. The original change interpolated the last
traceback frame into an `err_desc` string used in two places, and the second
one was the problem: `errors.execution_error` is sent back over the mesh, so
an absolute install path went out over RF and spent airtime on the error
path, where retries are most likely.
logger.exception() puts the full traceback in the log instead of just one
frame, which is strictly more than the original wanted, and the RF reply goes
back to str(e). That also removes the `list(tb.stack)[-1]` IndexError on an
empty stack, since the frame handling is gone entirely.
Tests cover both halves: the failure is logged via exception(), and the reply
carries only the exception text with no filesystem path.
CHANGELOG entry moves under the existing `## [Unreleased]` / `### Fixed`.
Add shlink support and rework response templates onto a state machine.
Merged via this branch rather than the PR head: the fork is org-owned, so
"allow edits by maintainers" does not grant push access and the review fixes
could not be pushed to #254 directly. Roger Fedor's four commits are included
with authorship intact, followed by two commits addressing the review.
Differentially tested the current parser against the pre-PR one over every template
shape shipped in config.ini.example and docs (31 shapes x 3 field sets x 4 message
states, 372 comparisons) plus 32 adversarial inputs. That found one regression I had
introduced: reading a quote immediately after ':' as the start of a quoted argument
voided `prefix_if_nonempty:"`, whose unterminated string rejected the whole
placeholder and emitted raw template text. An unterminated quote now falls back to
greedy parsing, so a literal quote prefix behaves as it always has. Both cases are
pinned by tests.
Also covered the shorten_url alias in feed formats, which had none: accepted as a
function, chains like shorten, falls back to the original on failure, is seen through
by shorten_feed_urls so a link is not shortened twice, and does not swallow an
unrelated name that merely starts with it.
Reset the warn-once flag in the async no-warning test. It asserted warning was never
called while a preceding test could already have tripped the flag, so it would have
passed vacuously.
Security: a shlink deployment with short_url_website unset POSTed the operator's
API key to v.gd. _normalize_base falls back to the public default and the shlink
branch guarded only on a missing key, never on a missing base. shlink now requires
an explicit base and refuses a v.gd/is.gd host outright rather than sending an
X-Api-Key there.
Correctness: _shorten_url_with_gd lost its response.ok guard when the backends were
split out, so a 502 whose body starts with http was returned as the short URL and
transmitted over RF. Restored, with a warning. The regression test that should have
caught this passed only because it left mock_resp.text as a MagicMock; it now uses
a realistic proxy maintenance body.
_shorten_url_with_shlink never checked the status either. Shlink reports failures as
RFC 7807 problem details, which parse as JSON and simply lack shortUrl, so a bad API
key was indistinguishable from an unshortenable URL at DEBUG. It now warns on a
non-OK status, a non-JSON body, and a missing shortUrl. Dropped the shortUrlSlug
fallback: that is a request field, not a response field, and returning it puts a
bare slug where a link belongs.
Timeouts and connection errors are back on their own DEBUG handler. They had fallen
through to the broad handler, whose level rose to ERROR in the same diff, so a
routine intermittent uplink logged "Unexpected error shortening URL" on every reply.
Parser: _GREEDY_ARG_FILTERS was tested before the quoted-argument branch, so
prefix_if_nonempty, the one filter already in shipped configs, could not use the
quoted syntax this PR adds. {path_distance|prefix_if_nonempty:"Dist {sender}: "}
rendered raw template text. A quote immediately after the ':' now selects the quoted
grammar; anything else stays greedy, so config.ini.example's
`prefix_if_nonempty: | Path Dist: ` keeps its pipe and its whitespace.
Blocking HTTP on the event loop: the path command rendered its reply prefix inline
from async code, so a slow shortener stalled radio RX, MQTT and every other handler
for the full 5s timeout. Added format_piped_template_async and switched the path
command to it. The test command still renders synchronously through the sync
check_keywords dispatcher; making that async is a separate change, so the filter now
warns once when it runs on the loop.
Naming: one operation should not have two names in an operator-facing DSL. shorten
and shorten_url are aliases in both response templates and feed formats, and
if_nonempty is canonical with if_notempty as an alias, so a filter chain copied
between a feed format and a response_format works either way.
Also: reduced _build_create_shlink_url to the one parameter it uses and dropped its
dead query/startswith lines, renamed the shlink POST callable from `get`, documented
the config argument on format_piped_template, stopped gating the render trace on an
unrelated parameter and logging field values (sender IDs, user phrases) with it, and
moved the changelog entry from Fixed to Added and Changed.
- Extracted `modules/alert_format.py` as the single NWS alert formatter, replacing four copies of the event-type abbreviation table and two of the time compactor. `!wx alerts` and the proactive `WeatherService` broadcasts now localize from one code path, so a Russian bot no longer answers `!wx alerts` in English while its proactive alerts are Russian.
- Stopped leaking translation key paths into mesh broadcasts. `Translator.translate` returns the dotted key when a lookup misses in both the locale and the English fallback, which is right for development but reached the air in production: an unclassifiable NWS title rendered as `⚪Hazardous services.weather_service.event_types.Unknown`, an unmapped WMO code as `services.weather_service.weather_descriptions.4`, and an oddly-cased `wind_speed_unit` as `services.weather_service.wind_speed_units.KMH`. `alert_format.translate_or()` carries an English default at each site, and `WeatherService` now normalizes and validates its three `[Weather]` unit settings the way `GlobalWxCommand` already did.
- Fixed alert expiry rendering in every locale. The formatter rendered a timestamp to a string and re-parsed its own output with `(\d+)(AM|PM)` against a hardcoded English month list, so translated months took the wrong branch and truncated mid-string. Times now carry parsed parts and render through a per-locale `common.alerts.time_12h` template — the space before AM/PM was correct (Russian writes "6 дня", not "6дня"); the downstream regex was the bug.
- Restored month abbreviation. `_compact_time` iterated over abbreviations and replaced them in the string instead of mapping full names, so English stopped shortening "June 28" and Russian replaced the "Jun" inside "June", leaving a stray Latin "e" (`июнe 28`). Reuses the existing `common.date_time.month_abbreviations` rather than the duplicate `services.weather_service.months` block.
- Made `!gwx` display units follow `[Weather]` config instead of the response language. Visibility switched on `base_language != 'en'`, so `language = ru` with the default `temperature_unit = fahrenheit` printed Fahrenheit beside kilometers, and `en-GB` was forced to miles. Pressure is a locale convention rather than a metric/imperial split, so each catalog names its own via `commands.gwx.pressure_unit` — previously every non-English locale inherited mmHg from the English catalog, whose `pressure_mmhg` string contained Russian text, giving German and French users Cyrillic pressure units.
- Let localized `H`/`L` labels reach a standard install. `config.ini.example` shipped the three `temperature_*_format` keys uncommented with literal `H:`/`L:`, and a config value always beats the new locale-aware default, so a Russian bot built from the documented example still rendered `H:47°C L:33°C`. The example now uses the `{high_label}`/`{low_label}` placeholders, which were documented in the docstring but not in the file.
- Routed high/low labels through the reply's translator. `_format_high_low` passed `bot.translator`, so with `auto_detect_language` on, an English-default bot answering a Russian sender localized the rest of the line but not `H:`/`L:`. Added `BaseCommand.response_translator` for this, replacing `wx_international`'s reach into the private `_response_translator` ContextVar.
- Fixed a byte-budget overrun in `!gwx`. The guard on the extra conditions block compared a character count against a byte-derived budget while the rest of the function used `_count_display_width`; Cyrillic is two bytes per character, so the block was appended after the budget was spent.
- Reverted nine `commands.gwx` English rewordings that were not localization work, including the configuration hint in `mqtt_weather_no_subscriber` — dropping it left a mis-configured operator with no pointer to the two keys they need.
- Fixed the Russian `visibility` string, which said "км" on the miles key — the same locale/config conflation as the code bug, in the data. Shortened the Russian event-type abbreviations, which were full words consuming a quarter of the 130-byte budget at two bytes per character.
- Added `commands.wx.hourly_not_available`, missing from every catalog so `!wx hourly` printed the raw key path. Predates this branch; found while auditing every translation key the weather modules reference.
- Moved alert strings to `common.alerts.*` and wind directions to `common.wind_directions.*`, since a command and a service both read them.
- Addresses #267
- Added `original_content` attribute to `MeshMessage` to store the on-air body of messages, ensuring it remains unchanged during command processing.
- Updated command matching methods across various commands to utilize `_cleaned_content_matches`, which restores original content when necessary, preventing mention stripping from altering the message context.
- Implemented tests to verify that original content is preserved and correctly utilized in command execution and message handling.
Add the `local_tranlation_path` to the `Localization` section of
the configuration file to allow one to specify a local translation
file outside the distributed translation files. This allows
translation files to be built for local commands without the
need of altering the distributed ones.
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
Render GWX pressure in mm Hg (rounded hPa->mmHg conversion) for
non-English locales instead of hPa, matching the metric handling used for
visibility. Also fix format_temperature_high_low so custom [Weather]
templates can use the documented {high_label}/{low_label} placeholders,
and add tests for the translator param.
Carto now requires an API key for basemaps.cartocdn.com and is retiring
raster tiles, so the mesh map's dark theme rendered an "API KEY REQUIRED"
watermark. OSM hosts no dark tiles of its own -- its Standard layer is
light only -- so switch the dark basemap to OpenFreeMap's vector 'dark'
style, rendered through MapLibre GL via maplibre-gl-leaflet. Leaflet and
the light OSM raster layer are unchanged, and the bridge renders into
tilePane so marker and overlay z-order is untouched.
Fall back to inverted OSM raster tiles when WebGL 2 or the bridge is
unavailable, so a failure degrades to a filtered map rather than a blank
one. The filter targets .leaflet-tile rather than .leaflet-tile-container:
the container is a 0x0 element that its absolutely-positioned tiles
overflow, so a filter there has an empty reference box and paints nothing.
CSP: drop cartocdn, allow tiles.openfreemap.org, and permit blob: workers.
MapLibre spawns its renderer in a worker from a blob: URL; without
worker-src it falls back to default-src 'self' and never starts.
Verifying a channel message against the RF cache only ever checked the row
strategy 4 returned — the most recent packet heard. That assumes the RF log row
and the decoded CHAN event for one reception arrive back to back with nothing in
between. On a dense mesh they do not: a repeater's echo of the very same packet
is routinely logged in the gap.
The reporter's message was heard directly (SNR 13.25, 0 hops) and again via
repeater f0 185 ms later (SNR 12.0, 1 hop). Both rows carry packet hash
392926C85DCB87D0, but the check saw only the echo, disagreed on path length and
SNR, and left the route unresolved — so the bot withheld a path it had decoded
correctly and answered "No path information available in current message". The
reporter's own observation that MultiTest still reported paths is the tell:
multitest reads recent_rf_data directly and never consults the match tag, so the
radio data was there the whole time.
The cache is now searched for the row the payload matches rather than testing
just the newest one. #80's guarantee is unchanged — a route is still only ever
attributed on a positive match, never on recency — so this widens where the
check looks, not what it accepts. When more than one row matches they must
resolve to a single non-empty packet hash, which is only true of receptions of
one packet; two unrelated packets that happen to agree on all three fields stay
a fallback. A debug line now names the case where the newest row was not the
message's packet, so this failure mode is visible in logs rather than silent.
Side effect worth knowing: SNR and RSSI now come from the message's own
reception too. The reporter's message was logged at SNR 12.0 / RSSI -10, the
echo's figures, when its actual reception was 13.25 / -32.
Five tests cover it, including a reproduction built from the issue's log. Three
fail on the current code; the other two pin behaviour the fix must not break —
newest-wins among receptions of one packet, and scope_eligible_only still
filtering the search so the scope correlation cannot be handed an ineligible row.
This commit addresses several issues related to MQTT connections. It prevents MQTT brokers from entering a reconnect storm by ensuring that the packet-capture watchdog only triggers a reconnect when necessary, avoiding duplicate CONNACKs and spurious disconnects. Additionally, it ensures that renewed MQTT auth tokens are properly enforced by implementing a clean reconnect process.
New configuration options are introduced: `mqttN_keepalive` for setting the PINGREQ interval per broker, and `mqttN_jwt_reconnect_on_renew` to control reconnect behavior after token renewal. Documentation has been updated to reflect these changes and to clarify the importance of unique client IDs for brokers in the same cluster.
- Add optional translator param to format_temperature_high_low()
- Translate temperature labels (H/L) via common.temp_high_label/low_label
- Translate wind directions via services.weather_service.wind_directions
- Fix wind format spacing (direction and speed were concatenated)
- Pass translator from Weather_Service, !wx, and !gwx callers
- Add _translate() helper to WeatherService using self.bot.translator
- Replace all hardcoded English strings with services.weather_service.* keys
- Add services.weather_service section to en.json (fallback for all locales)
- Add ru.json with full Russian translation (~450 keys)
format_keyword_response_with_placeholders kept the third copy of the hop-count
logic, and it was the weakest: it consulted message.hops and then went straight
to parsing the path display string, never looking at routing_info. So a keyword
reply and a command reply could describe the same packet differently.
Three cases disagreed, all now resolved:
case keyword command
routing path_length ? -> 2
routing path_nodes ? -> 3
routing beats path text 1 -> 2
The last is the one that was actually wrong rather than merely unhelpful: with
both present it took the display string over the packet's own path_length.
routing_info is the decoded packet, so it wins, which is what BaseCommand
already did.
All three formatters now call utils.message_hop_count. Behaviour is unchanged
wherever routing_info is absent, so a message carrying only a path string still
parses the same way and "?" still means the count cannot be determined at all.
{firstlast_distance|prefix_if_nonempty: | F/L Dist: } prints "| F/L Dist: N/A"
on a direct message: the distance helpers return the literal "N/A" when there
is no path to measure, prefix_if_nonempty only asks whether the value is
non-empty, and "N/A" is. Suppressing that needed a gate, and pathbytes_min was
the only one available—so it was being used as a stand-in for "did this
message take any hops", which is not what it asks. It asks how the path is
encoded, so pathbytes_min:2 also discards a one-byte multi-hop path whose
distance is real and measurable.
hops_min:N asks about the route itself. hops_min:1 drops a clause on a direct
message and nothing else:
hops_min:1 pathbytes_min:2
direct cleared cleared
1-byte 2 hops kept cleared <- the difference
2-byte 2 hops kept kept
unknown cleared cleared
An unknown hop count clears the value: a gate that cannot confirm the route
should suppress rather than guess, matching pathbytes_min.
The hop-count logic moves to utils.message_hop_count so the filter and
BaseCommand.get_hops_display_values share one implementation instead of two.
It returns None for unknown rather than 0, since a gate must not read
"cannot tell" as "direct".
{path_distance|pathbytes_min:2|prefix_if_nonempty: | Path Dist: } printed
"| Path Dist: N/A" on a direct message. bytes_per_hop_from_routing_and_nodes
returned routing_info's bytes_per_hop before it ever looked at whether there
were any hops, so a 0-hop packet whose format happens to use 2-byte hops
reported 2, the gate passed, and prefix_if_nonempty found "N/A" waiting for
it and dutifully attached the label.
bytes_per_hop describes how a path is encoded. A direct packet has no path
for it to describe, so the format field is not evidence of a wide path and
must not stand in for one. Both docstrings already promised 1 for the direct
case—"Returns 1 when no nodes (direct / unknown)"—so this is the code
catching up to its stated contract rather than a change of policy.
Latent until now: channel messages never had routing_info attached, so the
field was absent and the fallback returned 1 by accident. Direct DMs have
been hitting it all along.
{packet_hash} came back empty on a channel, and so did {path} and
{connection_info}. message.routing_info is only attached when the RF row is
correlated, and a channel message can never correlate: MeshCore's CHAN event
carries neither raw_hex nor a pubkey prefix, so every prefix strategy in
find_recent_rf_data is skipped and the message lands on the most-recent-packet
fallback, which #80 rightly refuses to take a route from.
The fallback is in fact almost always the right packet—the firmware emits the
RF log row and the decoded CHAN event for one reception back to back—but
"almost always" is a guess, and #80 is what guessing costs. The CHAN payload
does restate three things the RF row records independently: payload type,
path length and SNR. Requiring all three to agree, on a row no older than the
correlation window, turns the fallback into a checked hypothesis and earns a
real match kind rather than a fallback tag.
SNR carries the weight: it is one reception's measured value, quantised to
0.25 dB, so an unrelated packet agreeing on all three is improbable rather
than merely unlikely. Across the 55 channel messages in my logs the
immediately preceding RF row agreed on all three every time, always within a
second, while a window-wide search on the same fields was ambiguous for half
of them—so this verifies the ordering the firmware already gives us instead
of searching for a match.
Because these messages now correlate, they also satisfy the '*' gate by the
primary route, resolve their scope properly when flood_scopes is regional,
and contribute their path to the mesh graph, which they never did before.
Adds the 16-char MeshCore packet identity hash to the three formatters that
render user-facing templates: [Keywords] responses, the test command's
response_format, and the path command's reply_prefix. It is read only from
routing_info, which is attached solely from an RF packet correlated to the
message, so a hash from an unrelated transmission is never shown. Missing,
empty, and the 0000000000000000 error sentinel all render empty.
get_packet_hash_placeholder returns "" rather than a False sentinel. Empty
gives byte-identical output on every path—prefix_if_nonempty already tests
falsiness and str.format renders "" as nothing—whereas False would print
the literal string "False" into a mesh reply from any consumer of
get_standard_placeholder_fields that formatted it directly.
TestCommand.format_response now builds on get_standard_placeholder_fields
instead of duplicating it. That also honors the extra argument its docstring
already promised but silently ignored, and picks up the path-string hop
fallback, so {hops} reads "0" rather than "?" for a message whose hops and
routing_info are both unset but whose path says "Direct (0 hops)".
Documents the placeholder in config.ini.example, the default config that
core.py writes on a fresh install, the README and docs/path-command-config.md.
The generated config was also missing {elapsed}; added that too.
MeshCore's CHAN payload carries neither raw_hex nor a pubkey prefix, so a
channel message has no correlation key and find_recent_rf_data always lands
on the most-recent-packet fallback. rf_data_is_correlated() is therefore
never true for one, and _is_confirmed_global_flood could never pass: across
three log files every one of 45 received channel messages was rejected.
I raised this when the gate went in and accepted the counter that the
handler already has the general RF correlation and decoded packet info for
this message's own packet. That holds for DMs and is false for channel
messages.
'*' asks one question—was this a scoped regional flood?—and there is a
second way to answer it that the handler already computes and then discards.
A scoped message travels as TRANSPORT_FLOOD GRP_TXT, so if no such packet is
anywhere in the RF window, the message cannot have been scoped. That is a
window-wide fact, so unlike a route type read off a fallback row it does not
depend on having picked the right cached packet.
Correlated rows stay authoritative: a correlated TRANSPORT_FLOOD is still
blocked even when the scope window is empty, so an empty window cannot
launder a message the radio positively identified as scoped. Replaying the
real packets from the log, the 43 plain-FLOOD messages now get a reply and
the 2 that had a TC_FLOOD GRP_TXT alongside them still fail closed.
Keep the disabled-command fallback in one place. A disabled command still
matches; can_execute is what holds it back, and the dispatcher continues past it
so another command's alias can claim the trigger. The per-command enabled guards
in matches_keyword were a second mechanism for a problem already solved once, and
one every command would have to repeat, so drop them along with PathCommand's
matches_keyword override entirely.
Restore control-character tolerance in the test command. Matching runs against
clean_content-normalized text again, so a garbled "test" still matches. A test is
exactly the message someone sends when their link is marginal. The cleaned text
only sticks when the test command claims the message, since matching runs early
in the command scan and collapsing whitespace on another command's free-form
argument is not its business.
Make enable_p_shortcut = false actually work. Appending "p" to the inherited
class list left it on PathCommand.keywords for every instance built afterwards,
so a reload with the shortcut off still answered "p".
Have split_trigger_and_args honour the configured command prefix instead of
stripping a leading "!" unconditionally, and match word by word rather than
slicing the lowered copy by keyword length, which cuts the args in the wrong
place when str.lower() changes length for a non-ASCII alias.
Mention gating now applies to the test command, which the bespoke matcher had
skipped: "test @[Bob]" in a channel no longer gets an ack.
Added a new method `split_trigger_and_args` to the `BaseCommand` class, which splits message content into a matched keyword and arguments. This enhancement allows for better handling of command triggers and arguments across various command classes, ensuring that multi-word triggers are prioritized and leading characters are stripped appropriately. Updated the `ChannelsCommand`, `DiceCommand`, `HackerCommand`, `MultitestCommand`, `RollCommand`, and `TraceCommand` classes to utilize this new method for cleaner and more consistent command execution logic.
Removed redundant logging for command execution and added detailed debug logging for cases where a command cannot execute due to soft rejections. This allows for better tracking of command flow and ensures that keyword matching continues for subsequent commands. Updated the test_command to include config aliases in keyword matching, enhancing its flexibility.
PUT verified original_schedule outside the lock, so a concurrent delete
between the check and the write resurrected the entry as a new one, and
two concurrent renames of the same original left both results present.
The existence check now runs inside _save_scheduled_message_locked
against the same snapshot the duplicate check uses, so create, update and
delete are each a single critical section.
Added a concurrency test: two simultaneous creates of one schedule now
produce exactly one 200 and one 409 with a single entry on disk.
A flood_scopes of "*" alone leaves scope_keys empty while setting
flood_scope_allow_global, and the loader already logs that as an active
allowlist. The handler gated on scope_keys alone, so that configuration
skipped authorisation entirely and admitted absent, uncorrelated and
TRANSPORT_FLOOD traffic. The gate now fires when either is set, so "*"
means global-only rather than everything.
The scheduled-message duplicate check ran outside the write lock inside
update_ini_values, so two concurrent creates for the same schedule could
both pass and the second silently replace the first instead of getting
the 409. Check and write are now one critical section, and delete is too.
Repaired the delete path's error handling while moving it, so an OSError
during the write is still a 500 rather than escaping.
Accepting the reviewer's rejection of my narrower version. I had argued
that requiring positive proof would silence legitimate global replies
whenever correlation is unavailable, but that objection does not hold:
the handler already has the general RF correlation and decoded packet
info for this message's own packet, so the normal global case can be
proven rather than assumed. Only genuinely uncorrelated traffic is
affected, and for an allowlist that should fail closed.
_is_confirmed_global_flood() now requires RF data correlated to this
message showing RouteType.FLOOD. Absent, uncorrelated, or
TRANSPORT_FLOOD data means the scope is unknown, and '*' no longer
admits it.
The channel-message tests never set flood_scope_keys, so it was a Mock
and read as truthy, meaning they were unintentionally exercising the
allowlist and only passed because `not allow_global` on a Mock is False.
Set explicitly to the unconfigured default they meant to test.
Path distance was still leaking between concurrent requests three ways,
all of which Codex reproduced:
- The formatter fell back to shared instance state whenever the request's
own value was None, so a request with no measurable distance rendered
another request's. It now consults instance state only when there is no
request to read from.
- One extraction branch still called _decode_path without the request, so
that route stored its distance on shared state alone.
- _get_sender_location read the shared _current_message, so an
interleaved command could measure from the wrong sender. It takes the
request's message, falling back to the shared value only for direct
calls that pass nothing.
Scope authorisation: '*' permits unscoped global traffic, not traffic of
unknown scope. When a scope-eligible packet was heard but could not be
tied to this message, its scope is unknown and '*' no longer admits it.
Deliberately narrower than the reviewer suggested: when no scope-eligible
packet was heard at all, '*' still applies, because nothing scoped being
heard is what a genuinely global message looks like. Requiring positive
proof of FLOOD in that case would silence legitimate global replies
whenever correlation is unavailable, which is too much availability to
trade for the residual risk.
Two fixes from the previous round were incomplete:
- The scheduled-message chunk budget used the schedule's explicit scope,
but send_channel_message resolves an unset scope from
flood_scope.<channel> and then outgoing_flood_scope_override. A
schedule with an implicit regional scope was therefore sized for a
global send and every chunk could overshoot once the sender added the
regional header. The budget now resolves the effective scope, and
assumes regional if that resolution fails, since guessing regional only
ever makes chunks smaller.
- Resetting _last_path_distance_km per request fixed sequential reuse but
not concurrency: the decode awaits a database lookup, and the
dispatcher runs handlers as independent tasks, so two path commands can
interleave and render each other's distance. The distance now rides on
the request's own message, with the instance attribute kept only as a
fallback for direct calls.
Two new findings:
- rf_data_is_correlated() treated pubkey and partial-prefix matches as
packet-unique, but a sender prefix identifies a sender, not one
transmission. With several cached packets from the same sender, the
first (usually oldest) was returned and allowed to supply a route.
Those strategies now take the newest match and are authoritative only
when the match is unambiguous; otherwise the entry is marked fallback,
so it still provides SNR/RSSI but never a route.
- The flood_scopes allowlist accepted a scope resolved from an
uncorrelated fallback packet. The HMAC proves the cached packet is in
an allowed scope, not that this message is, so a recent allowed-scope
packet could admit an unrelated message. Scope authorisation now
requires packet-bound correlation and logs plainly when it does not
have it.
That last one is a deliberate fail-closed change to an authorisation
path that predates tonight. Channel messages normally carry raw_hex and
correlate exactly, so the fallback is the exception rather than the rule,
but a deployment using flood_scopes will now stay quiet in cases where it
previously replied on an assumed scope.
Correctness:
- {path_distance} was always blank in production. The resolution code that
builds repeater_info dropped latitude/longitude in every branch, so the
calculator could never find a coordinate. My tests passed hand-built
dicts straight to the calculator and never exercised the builder, which
is why they stayed green. Coordinates are now carried through all four
construction sites, and the new tests drive _lookup_repeater_names via
its lookup_func hook so the real builder runs.
- The #80 route guard was defeated two ways in the channel handler. When
the RF data was an uncorrelated fallback, control fell through to the
raw-hex and routing_info fallbacks below, which took the route from the
unrelated packet anyway; the guard had actually made that path
reachable. message.routing_info was also assigned unconditionally, and
the path command reads it. "Not attributable" is now a terminal branch
and the routing_info hand-off checks provenance.
- Same fix was incomplete for DMs: routing_info was captured and turned
into path_info before the provenance check ran, so the later check only
declined to overwrite an already-wrong value. Guarded at the source.
- Rendering could transmit for real. Capture only intercepts
send_response, but advert calls send_advert() directly and
send_response_chunked never checked capture_sink. Chunked sends are now
captured, and rendering is opt-in via BaseCommand.render_safe (default
False) instead of a denylist that cannot be complete. This also closes
the DM-only leak: schedule is not marked safe, so {cmd:schedule} can no
longer broadcast configuration to a channel.
- Multi-part rendered output was rejoined into one oversized send.
Scheduled messages are now split to the RF body budget and sent through
send_channel_messages_chunked, on character boundaries so multi-byte
text is not corrupted.
- _last_path_distance_km is instance state that was only set on success,
so an invalid path request could show the previous request's distance.
Reset at the start of every execute().
- Stale-contact retries were only counted on non-OK results, so timeouts
and exceptions left a contact eligible forever and could recreate the
storm. All failed attempts count now.
Web viewer:
- A failed config reload was reported as a successful save. The API and
UI now distinguish "saved and active" from "saved, restart needed".
- Preview count was unbounded; clamped to 1-20.
- update_ini_values is a read-modify-replace with no locking, so two
concurrent viewer saves could lose one. Serialised behind a lock.
The bar had grown to twelve items at full expansion, over half of them
configuration surfaces. Radio, Scheduled Messages, Greeter, Feeds,
Plugins and Configuration now sit behind one gear, so the top level is
about what the mesh is doing (Dashboard, Real-time, Contacts, Mesh
Graph, Multibyte, Logs) and the gear is about how the bot is set up.
Logs stays top level: it is what you reach for while watching behaviour,
not while configuring.
Also added active-page highlighting, which the bar never had. A settings
page lights up both its dropdown entry and the gear, so the current
location is still obvious once a page is one level down.
Conditional entries keep their guards, so Greeter and Feeds appear in
the menu only when enabled.
Adds a Schedule page that lists every [Scheduled_Messages] entry with its
next run time and supports add, edit and delete. Writes go to config.ini
and queue a config reload, so schedules change without restarting the
bot, which was the actual request.
It edits the config section the bot already reads rather than
introducing a database table. reload_config() already re-runs
setup_scheduled_messages() with rollback, so there is nothing to keep in
sync and the schedule command lists exactly what the page shows.
Validation runs through the same parsers the scheduler uses, so the UI
cannot accept a schedule the bot would later reject. The builder
composes cron from plain-language options and previews the next five
runs; entries the bot cannot run are listed as "Not scheduled" with the
reason instead of being hidden, since a typo that silences a message is
what an operator most needs to see. The 15-minute floor for {cmd:...}
messages is enforced at save time too.
Also relabels the radio Disconnect button to "Stop Bot" behind a
confirmation (#240). It was never a radio-only disconnect: the main loop
runs while self.connected is true, so disconnecting exits the process.
That surprised an operator running under tmux with nothing to restart
it. disconnect_radio()'s docstring now says so as well.
find_recent_rf_data has four strategies, and the fourth returns the most
recent packet in the cache when the first three fail to correlate. That
fallback exists for SNR/RSSI timing issues, but callers were also taking
its route. So when correlation missed, a message was recorded with some
other transmission's path and hop count — the reporter's four-hop message
stored as a single direct hop via 79, which is the last hop of an
unrelated packet. Rare, because it only fires when correlation fails.
Results are now tagged with how they were matched (exact, pubkey, partial
or fallback) and rf_data_is_correlated() gates the route. The tag rides on
a shallow copy so the cache entry is never marked, and it fails closed: an
untagged dict is treated as uncorrelated.
Guarded in both handlers. The DM path was worse than the channel one — it
falls back to find_recent_rf_data() with no correlation key at all, which
can only ever return a fallback, and then overwrote message.path and
message.routing_info from it. routing_info is what the path command reads,
so a wrong route reached the user directly.
The route is now left unresolved rather than fabricated, which also stops
a bogus edge being written to the mesh graph from path_nodes that belong
to a different packet. SNR and RSSI still use the fallback as before;
mis-correlated signal figures are approximate rather than structurally
wrong, and changing them is a separate call.
Existing tests asserted the returned dict was the cache entry itself.
Identity was incidental, so they now compare contents and additionally
assert the provenance tag.
The seed check added in c1cbdf9 only guarded stale-contact cleanup, but
three other paths read the same raw last_advert and drew the same wrong
conclusion from it:
_get_repeaters_for_purging sorts by apparent age, oldest first
_get_companions_for_purging scores by days_inactive, most inactive first
purge_old_repeaters removes anything older than a cutoff
In all three an unset clock reads as maximum age, so a node that had
never been time-synced was the first candidate for eviction regardless
of whether it was active. Unlike the stale-contact path, these do not
just log — they remove contacts.
Same rule everywhere now: an unset device clock means staleness is
unknown, and unknown staleness is not grounds for removal.
Corrects the timestamp guard shipped in 5b07ee2, which assumed the
reporter's "722 days ago" contacts were genuine mid-2024 observations.
They were not.
MeshCore seeds an unset clock with a hardcoded time rather than zero:
1715770351 (15 May 2024) in VolatileRTCClock, and RTC_TIME_MIN
1772323200 (1 Mar 2026) on the NRF52 and ESP32 RTC paths. Measured from
the date in the issue, 1715770351 is exactly 722 days — every affected
contact reported the same figure because they were all sitting on the
same seed.
So those contacts were never stale. Their clocks were unset, and the bot
was reading that as extreme age: they sorted to the top of the staleness
list, consumed the whole per-sweep removal budget, and the bot kept
trying to evict nodes that may well have been active. The previous
2020-then-2024-01-01 floor sat below both seeds and never fired.
Now matched against the seeds themselves, plus anything at or below the
earliest (which still covers a raw 0 decoding to 1970). A device running
unsynced for a while reports seed + uptime and remains undetectable;
noted in the docstring rather than guessed at.
A failed remove_contact leaves the contact on the device, so the next
cleanup sweep selected it again and logged the same warning again. With
the contact list stuck near its limit the sweeps kept coming, which is
the reported flood of hundreds of "Failed to remove stale contact"
warnings that only a restart cleared — and the restart only helped
because it reset the in-memory contact list, not because anything was
fixed.
Refusals are now counted per public key. After 3 consecutive failures
the contact is excluded from future sweeps, with one summary warning
saying the list may stay near its limit and that the contact needs
removing from the companion app. A successful removal clears the count,
so a transient failure costs nothing.
Also stop treating unset and future last_seen values as staleness. An
unset timestamp parses as 1970, and since candidates are sorted by
staleness descending, those entries sorted to the top and consumed the
whole max_remove budget every sweep — starving the contacts that could
actually have been removed. A genuinely old contact (the reporter's 722
days) is still selected; that is a real observation, not a bad
timestamp.
Two guards on command placeholders in scheduled messages, neither
configurable, because both exist to protect a shared medium:
A 15-minute floor. A schedule containing {cmd:...} that fires more often
is rejected at startup with an error rather than quietly running slower.
The interval is sampled and measured by the tightest gap between
firings, so "0,1 * * * *" is correctly treated as every 60 seconds and
not as hourly. Schedules without a command placeholder are unaffected.
The command's own cooldown_seconds now applies to a render. Scheduling
is not a way around the rate a command was configured to run at. The
execution is recorded before the command runs, matching execute_commands,
so a slow or failing render cannot be retried straight past the cooldown.
Also fills documentation gaps from the preceding commits:
- --install-extras was only in the script's own --help; documented in
service-installation.md (including alongside --update-venv) and
upgrade.md.
- weather-service.md documented weather_alarm's once-a-day scheduling
with no route to more than one forecast a day, which is exactly what
people go there looking for. It now points at {cmd:wx ...} and notes
that sunrise/sunset still belong to weather_alarm.
Gerard Hickey