A fifth review found two gaps. A scheduled trial that returned False was re-armed at once, but False also follows exhausted no_event_received retries, so the radio could be tried again and again without another answered probe. And a reply's success settlement ran in a worker thread that a cancelled caller could cancel while it was still queued, stranding the trial in flight.
_end_offline_trial ends a trial that went out without succeeding (False or an exception, from a reply or a scheduled send) under the state lock alone; only paths where nothing reached the send still re-arm. The success settlement is shielded, so it completes even if the caller is cancelled. The cancellation test queues the settlement behind a busy one-worker executor and fails without the shield.
Only scheduled messages and interval adverts could claim the trial armed by an answered health probe, so a bot with neither stayed offline however healthy the radio became. send_dm and send_channel_message now claim an armed trial when offline and run the send through _send_as_offline_trial, under the trial's identity; its result settles the trial (True clears the outage, False or an exception fails it until the next answered probe), so a dead radio still gets at most one attempt per probe. Settlement runs in a worker thread because it can write bot_metadata. Tests drive the real CommandManager guard, and the alert-thread counting in the recovery tests no longer stops asyncio.to_thread's workers.
A third review found four more gaps. A trial send that hit the 60 s wait kept running and could transmit later, during a newer trial; _record_send_success cleared any outage without checking whose success it was, so an old send finishing late could clear a newer one; the wrappers claimed the trial before logging and time lookup, outside the code that settles it; and the startup advert's timeout path recorded its failure on the event loop, where tripping the outage writes bot_metadata.
_record_send_success now takes the trial id and clears only when that trial is still in flight, deciding and resetting under the state lock (_reset_offline_state_locked); success of an ordinary send while offline leaves the outage alone. A trial whose wait fails is cancelled. Each wrapper settles the trial if anything after admission raises. The startup advert records its failure through asyncio.to_thread.
A second review found five gaps. A loop that closed between is_running() and submission stranded the trial in flight; fallback sends (no running main loop) settled every trial as inconclusive; a timed-out send that kept running passed the guard during a later trial, because the context flag was a bare True; the viewer clear checked the outage generation, released the lock, then cleared, so it could clear a newer outage; and the health loop ran the metadata sync, with its database retries, on the event loop.
Trials now carry an id: _admit_measured_send returns it, _as_offline_trial sets it in the context, and is_radio_offline and the settle methods only honor the trial currently in flight. The scheduler's two wrappers share _run_measured_send, which submits inside its error handling (closing the coroutine if submission fails), settles a fallback trial by its real outcome, and settles any trial left open in a finally. _clear_radio_offline_state takes an expected generation and checks it under the state lock. The health loop checks the 30 s throttle on the loop and runs the sync itself in a worker thread. The delayed-write test now waits for the newer outage to trip before releasing the stalled writer, so it fails without publish serialization.
Review of the circuit breaker found four problems. The trial was a shared flag that let every sender through, interactive sends never settled it, and it could stay open forever. The scheduler threw away send_channel_message's result, so a send the radio rejected counted as a success and cleared the outage. Metadata writes happened outside the state lock, so a stalled clear could overwrite a newer outage's 'true' and the viewer check would then cancel it. And one failed read-back of the persisted 'true' disabled the viewer clear for the rest of the outage.
Now a probe arms one trial, which the scheduler's next measured send (scheduled message or interval advert) claims atomically through _admit_measured_send; that send runs under a context flag so only it passes the send-path offline guards. Its real outcome settles it: success clears the outage, a timeout ends the trial until the next answered probe, and a False result or nothing sent re-arms it. _send_scheduled_message_async now returns whether the message went out. Metadata is published under its own lock from the current state, confirmed per outage generation, and retried from the health loop until confirmed. The in-process viewer clear calls the full clear. Tests go through the real scheduler on a live loop, race admission across threads, stall a clear write while a new outage trips, and count alerts from the first trip.
The threshold test timed one request, so a single GC pause or a busy CI runner could fail it. On the Python 3.10 runner it failed four times in a row at 0.22 to 0.26 s on a branch that does not touch the stats path, while the same request takes about 22 ms locally on both that branch and dev, with or without coverage. It now times five requests and asserts on the fastest, which an algorithmic regression still slows, and the failure message lists every timing.
Moving the geocode decision ahead of the await meant every advert that arrived while another was still geocoding read the same uncommitted row and geocoded again; six adverts for one contact made nine or more Nominatim requests instead of two, at the 1 req/s budget. track_contact_advertisement now holds a per-contact asyncio lock across the read, geocode and write, which restores the old one-at-a-time behavior for a contact while different contacts still proceed independently. The lock entry is dropped when its last holder leaves.
Also: geocoding_cache reads in the worker-thread helpers use .get() so the loop-thread expiry pass cannot raise KeyError between the check and the read, the write-block duplicate guard calls _is_duplicate_advert_packet, and the guard test loses leftover scaffolding.
Review follow-up. A get_time probe is a host-to-companion frame with no RF transmit, so it answers in exactly the failure mode the offline state exists for; clearing on it would flap offline and online every probe interval and send an alert email each time. A probe success now only opens a trial: sends are let through, and the next send the scheduler measures either clears the state or closes it again, without another alert. The alert goes out once per outage.
The reconnect clear is gone (a connect can still fail after the socket is up, and the probe covers recovery after it). The viewer-clear check runs only after this process has read back its own 'true', so a read that races the trip cannot cancel it; it is throttled to every 30 s and runs before the transport check, so it works while disconnected. State changes take a lock, since the scheduler thread and the event loop both write them.
track_contact_advertisement, _background_geocoding, scan_and_catalog_repeaters and populate_missing_geolocation_data are async but called rate-limited Nominatim synchronously (with the rate limiter's time.sleep), so the whole bot stalled for each lookup. The async-path guard test only scanned commands and service plugins, so it never saw this file.
The calls now go through asyncio.to_thread. track_contact_advertisement decides whether to geocode, and does it, before its write block, which stays free of awaits so concurrent adverts cannot interleave between its reads and writes; the duplicate-packet guard runs both before geocoding and again inside the write block. The guard test now scans repeater_manager.py and knows the state/country and full-location helpers; a new test shows advert geocoding runs off the loop thread and fails without the change.
After radio_offline_threshold consecutive send timeouts the bot enters radio-offline state and every send path refuses to transmit. The only thing that cleared it was _record_send_success, which runs after a send goes out, so once offline the bot stayed offline until restarted. The web viewer's Clear Offline Flag could not help either: the viewer runs in its own process, so it only wrote bot_metadata, which the bot never read back.
Three paths now clear the state: a health probe the radio answers, a successful (re)connect, and the health loop noticing that bot_metadata says the flag was cleared. The new tests for the probe and viewer paths fail without the change.
send_channel_message compared the set_flood_scope reply's type to the string "ERROR", but it is an EventType whose value is "command_error", so the comparison never matched. A rejected regional scope sent the message at whatever scope the radio held, and a rejected restore to global flood left the radio pinned to the region for every later send, both without a log line. Compare against EventType.ERROR.
The existing tests for this path built their error reply with the same "ERROR" string, so they passed against the broken check; they now use EventType.ERROR and fail without the fix.
Review follow-up. The firmware's direct-path estimate includes hop count and about 750 ms of fixed allowance, not airtime alone, so the comments and changelog now say that. Values above 60 seconds are capped, since a value meant as milliseconds would otherwise hold one DM for hours, and the changelog notes that an unacknowledged DM now takes at least 24 seconds to report failure. The late-ACK test has a wider margin so it cannot flake on a loaded machine.
send_msg_with_retry was called with timeout=0, so each attempt waited 1.2 times the radio's suggested_timeout. The companion firmware derives that from airtime alone (500 + (6 x airtime + 250) x (hops + 1) ms direct, 500 + 16 x airtime flood), but the round trip also carries fixed costs that do not shrink with airtime: the recipient's ACK delay, repeater retransmit and rx delays, the radio's TX queue and the host link. On a fast preset a zero-hop attempt gave up after about 1.3 seconds, the ACK of a delivered message arrived after the last attempt, and the send reported failure. That is the residual #290 symptom: webhook DMs erroring within a couple of seconds for messages the recipient had.
New [Bot] dm_min_ack_timeout (default 8 seconds) is passed as min_timeout. An ACK still ends the wait at once, so successful sends are no slower, and since the radio lock is per-frame the longer wait on a failing DM does not hold up other sends. 0 restores the old behavior. The new test drives the real meshcore retry loop with an ACK slower than the radio's estimate and fails without the floor.
Builds on #303. Channel text is capped at 155 bytes of "<name>: <text>" instead of 124. The firmware encrypts a 4-byte timestamp and a 1-byte text type ahead of the text in 16-byte AES blocks, so 155 fills the 160-byte block exactly (163-byte payload), and the last 5 bytes up to the firmware's 160 would spill into another block (179-byte payload).
124 was set on the premise that repeaters drop 165-byte frames. They don't: allowPacketForward has no size check and MAX_PACKET_PAYLOAD is 184, and 163-byte GRP_TXT payloads are routinely seen at 3+ hops. The frame-budget tests now pin the limit within the firmware's 160 and to a whole cipher block instead of to a 131-byte relay ceiling.
Also corrects the payload model's plaintext header from 4 bytes to 5, and moves a test import back above the helper that had split the import block.
* feat(web_viewer): split anonymous read access from an admin session
When web_viewer_password is set, Dashboard, Contacts, and Mesh Graph are readable without logging in, along with the read-only APIs they need. Radio, Config, Logs, every mutating API, /api/channels, and the live Socket.IO streams require an admin session, since those can expose channel keys and decrypted traffic. The Realtime page still renders publicly but shows "Admin login required" instead of connecting. Anything not on the public allowlist fails closed.
Login now compares HMAC digests with hmac.compare_digest and clears the session before setting authenticated_admin. Logout clears the whole session and returns to the dashboard.
Co-authored-by: Gerard Hickey <hickey@kinetic-compute.com>
* fix(web_viewer): stop Contacts redeclaring IS_ADMIN and hide banner admin buttons
contacts.html declared its own top-level const IS_ADMIN, but base.html already declares one in an earlier script on the same page. Classic scripts share one global lexical scope, so the redeclaration threw a SyntaxError and the whole Contacts script never ran, leaving the table on its loading placeholder in every mode. Contacts now uses the global from base.html.
The zombie-recover and clear-offline banner buttons were still rendered for anonymous viewers even though their /api/admin/* endpoints now return 401. They are gated on the same auth_enabled/is_admin check as the other admin controls.
* fix(web_viewer): drop sockets on logout, block off-site login redirects, allow HEAD on public paths
Logging out cleared the HTTP session but left that login's Socket.IO connections open, and a socket keeps the session it connected with, so a /realtime tab kept streaming packets, decrypted messages, and logs after logout. Each login now gets an admin_login_id, sockets record it on connect, and logout disconnects the sockets opened under that login. Other admin logins stay connected.
The login next check passed ///host and /\host because urlparse sees no netloc in either, but browsers resolve both as another origin. Next targets are now rejected if they start with //, contain a backslash, or contain a tab or newline. This check predates the admin split.
Public pages and GET APIs only accepted literal GET, so HEAD (and OPTIONS) on them returned 401 and made uptime checks report the viewer as down. They now accept HEAD and OPTIONS too; admin paths are unchanged.
* docs(changelog): note the login redirect fix
* fix(web_viewer): pass IS_ADMIN through a body data attribute so ESLint can parse base.html
The frontend lint job failed with "Parsing error: Unexpected token {" because base.html's inline script assigned IS_ADMIN from a raw Jinja expression, which isn't valid JS until rendered. The flag now lives in data-is-admin on <body> and the script reads it from document.body.dataset. I also dropped the unused AUTH_ENABLED constant and declared IS_ADMIN as a readonly ESLint global, since contacts.html reads it from base.html.
---------
Co-authored-by: Gerard Hickey <hickey@kinetic-compute.com>
Decoded packet_stream rows had no way back to the original packet, so alternate routes could not be joined. The hash is already on routing_info; keep it in the existing JSON blob.
An oversized webhook or channel body was handed to the firmware whole. The device dropped it without an ack, retries stalled the transport, and the radio looked dead. Split to the frame budget, keep links intact, and tag multi-part sends so they can be reassembled.
Reserve the wx location prefix from that same budget, and let NOAA alert polling be turned off without disabling the rest of the weather service.
Commands are bare by default; `command_prefix` is empty unless configured, so
`!wx` misrepresents a stock install. Shipped release sections are left alone.
Fold the Unreleased section into 1.1.0 and date it. The 1.1.0 heading was
written on 2026-09-13 but never tagged, so everything merged since belongs in
it; the section also carried two separate "### Added" headings from earlier
merges, now consolidated. All 79 entries preserved.
Extend the v1.0.0 -> v1.1.0 upgrade guide with the meshcore 2.3.14 requirement,
migration 24, and the configuration and behavior changes that landed after the
heading was first written.
The report used /n and got two literal characters. The escape works the same in
every keyword path — I checked the test command's template engine too — so the
defect is that the instructions never said "backslash" and used `test` as their
example, which is the one keyword a built-in command owns. That command renders
through a different engine and silently drops the mesh-info placeholders listed
in the same block.
Also documents the configparser continuation form, which needs no escape, and
corrects two packet-capture examples that still showed a naive local timestamp
where the bot has published UTC with a Z suffix since v1.0.0 (#276).
waev.app rejects a token whose exp is more than an hour past its iat, so the
project-wide 24-hour default could never authenticate there — the operator saw
only a bare auth failure with no hint that the lifetime was the problem, while
the same public key worked from meshcoretomqtt.
waev.app hosts now resolve to a 3600s TTL with renewal at 3500s when no value
was configured for them, per broker or globally; a configured value still wins,
and the substitution is logged. Matching is the apex and its subdomains only, so
a lookalike host like waev.app.example.com does not qualify.
Classify every channel message's flood scope, tally it per channel per day,
and optionally warn senders whose messages carry no region code. Warnings are
off by default and start in dry run; they fire only on positive RF evidence of
an unscoped FLOOD, and are fenced by a per-sender cooldown, a mesh-wide
cooldown and a daily cap read from the database.
Adds Region Scopes and Default Region Scope cards to the Radio page so the
bot's own scopes and the radio's firmware default can be set from the web
viewer.
# Conflicts:
# CHANGELOG.md
* feat(website): Include local commands when generating website
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
* feat(website): Disabled commands are excluded from generated HTML
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
* Updated CHANGELOG.md
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
* fix(website): resolve enablement through the bot's own config helpers
The generated page is meant to list what the bot actually answers, so it
has to read the config the same way the bot does. Three places where it
did not:
The local `config.ini` overlay was never read. `read_config` parsed only
the base file, while the bot overlays `<local_dir_path>/config.ini` on
top of it, and that overlay is exactly where the web viewer saves a local
plugin's settings. A local command disabled from the settings page still
showed up on the site.
`is_command_enabled` derived the section name and the legacy `enabled`
spellings by hand, duplicating `BaseCommand._derive_config_section_name`
and guessing that a legacy key lives in the command's own section. Half
of them do not: `[Jokes] joke_enabled` disables `Joke_Command`. It now
calls `command_section_name` and `read_enabled`, which share the alias
table with the runtime and the settings UI.
That table was missing the same-section `[Joke_Command] joke_enabled` and
`[DadJoke_Command] dadjoke_enabled` spellings, which both commands accept
through their own fallback. The settings page read only the `[Jokes]`
form, so a bot disabled the same-section way displayed as enabled.
Also fold the duplicated local-commands-dir resolution into
`resolve_local_commands_dir`, take `bot_root` as a `MinimalBot` argument
instead of assigning it post-construction twice, and restore the `hidden`
attribute check that `generate_samples` lost when it moved to
`filter_commands`.
---------
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
Co-authored-by: agessaman <adam@gessaman.com>
Adds a `[Hello_Command] include_sender` setting (default off) that names the user the hello reply is answering, so a busy channel can tell whose greeting the bot picked up.
The mention takes the place of the random human descriptor, keeping the translated sentence intact, and is dropped when it would push the reply past the channel body budget. DMs are unaffected, and channel sender names are sanitized before being echoed.
* feat(command): Add contact command
The contact command allows the user to ask the bot to provide its
public key as a clickable contact. This allows the user to then DM the
bot outside of a channel.
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
* fix(contact): harden matching and self_info handling, add tests and docs
Review follow-ups on the contact command:
- Match through _cleaned_content_matches so a non-matching message keeps its
original content. The raw cleanup_message_for_matching call rewrote
message.content in place during the keyword scan, which is the #267
regression every other command already avoids.
- Match against self.keywords so a configured alias works.
- Read self_info defensively (dict or object, may be absent) and validate the
public key instead of sending a literal "<None:1:None>" over the air.
- Drop the hardcoded '!' prefix strip in execute; matching already normalizes
the content, and prefixes are configurable.
- Forward skip_channel_check to the base can_execute.
- Remove copy-paste docstring leftovers from the roll command.
- Move [Contact_Command] into the alphabetical run in config.ini.example and
fix the "chanels" typo.
- Add tests, a docs/command-reference.md entry, and flesh out the changelog.
---------
Signed-off-by: Gerard Hickey <hickey@kinetic-compute.com>
Co-authored-by: agessaman <adam@gessaman.com>
The ACK fix I wrote for the radio-lock work is now upstream and released
(meshcore_py#108, v2.3.13). Before it, send_msg_with_retry reported ACKed DMs as
failures: the ACK subscription was registered only after send_msg returned, so
an ACK queued right behind MSG_SENT was dispatched with no listener, and each
attempt accepted only its own ACK code, so a late ACK answering an earlier
attempt was ignored. The bot logged "no ACK received after retries" and skipped
the delivery bookkeeping for messages the recipient had in fact received. Pin
2.3.14, which also fixes send_cmd's destination-type handling.
Two fixes from re-reading the merged radio-lock change:
A region-scoped channel message now restores global flood even when
set_flood_scope raises. The restore only ran in the finally around the send, so
a set that raised left the device pinned to that region and every later send
went out under it.
Drop the __setattr__ left behind when _SerializedCommands became
_serialize_command_frames; it sat after the function's return, unreachable, and
still referenced the proxy's _commands slot.
Radio commands were serialized per method call, so a composite command held the lock for everything it awaited. A DM's send_msg_with_retry kept every other command waiting through all of its ACK timeouts (up to ~36 s with three attempts), stalling channel replies, other DMs, scheduled sends, and automatic message fetching. req_regions_sync did the same for each neighbor scope reply.
Serialize at the frame instead: wrap CommandHandler.send() on the handler instance, which is where every meshcore command writes its frame and waits for the radio's immediate reply. There is still at most one in-flight companion frame, paced as before, and library methods that call self.send are covered without a proxy. Waits for ACKs and remote responses now run outside the lock.
Add MeshCoreBot.radio_session() for short sequences that must not interleave, and use it for region-scoped channel sends so set scope, send, and restore stay together. The per-call lock never guaranteed that: a DM retry loop queued between set_flood_scope and send_chan_msg sent its flood attempts under the channel's scope.
_channel_secrets() assigned dict_items and enumerate to the same
variable, and mypy inferred Any | None for the channel_idx .get() with
a mixed Any | int default. Declare items as Iterable[tuple[int, Any]]
and hold the raw index as Any. Runtime behavior is unchanged.
Also cover the list layout meshcore actually uses, where entries
without channel_idx fall back to their position.
generate_website.py can layer custom CSS on top of the built-in style chosen with --style. --embed-css appends a local file to the page's <style> block so the page stays a single file, and --link-css adds a stylesheet link after it, so both override built-in rules of equal specificity. The flags can be combined (the linked stylesheet loads last), apply to --sample pages, and an unreadable --embed-css file stops generation with an error. The website docs gain a custom CSS guide and a reference for the generated page's structure, classes, and theme variables.
agessaman