diff --git a/AGENTS.md b/AGENTS.md index 94f832c4..0c09b6f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,7 @@ A web interface for MeshCore mesh radio networks. The backend connects to a Mesh - `frontend/AGENTS.md` - Frontend (React, state management, WebSocket, components) Ancillary AGENTS.md files which should generally not be reviewed unless specific work is being performed on those features include: -- `app/AGENTS_MQTT.md` - MQTT architecture (private broker, community analytics, JWT auth, packet format protocol) +- `app/fanout/AGENTS_fanout.md` - Fanout bus architecture (MQTT, bots, webhooks, Apprise) - `frontend/src/components/AGENTS_packet_visualizer.md` - Packet visualizer (force-directed graph, advert-path identity, layout engine) ## Architecture Overview @@ -75,7 +75,7 @@ Ancillary AGENTS.md files which should generally not be reviewed unless specific - Raw packet feed — a debug/observation tool ("radio aquarium"); interesting to watch or copy packets from, but not critical infrastructure - Map view — visual display of node locations from advertisements - Network visualizer — force-directed graph of mesh topology -- Bot system — automated message responses +- Fanout integrations (MQTT, bots, webhooks, Apprise) — see `app/fanout/AGENTS_fanout.md` - Read state tracking / mark-all-read — convenience feature for unread badges; no need for transactional atomicity or race-condition hardening ## Error Handling Philosophy @@ -97,7 +97,7 @@ The following are **deliberate design choices**, not bugs. They are documented i 1. **No CORS restrictions**: The backend allows all origins (`allow_origins=["*"]`). This lets users access their radio from any device/origin on their network without configuration hassle. 2. **No authentication or authorization**: There is no login, no API keys, no session management. The app is designed for trusted networks (home LAN, VPN). The README warns users not to expose it to untrusted networks. -3. **Arbitrary bot code execution**: The bot system (`app/bot.py`) executes user-provided Python via `exec()` with full `__builtins__`. This is intentional — bots are a power-user feature for automation. The README explicitly warns that anyone on the network can execute arbitrary code through this. Operators can set `MESHCORE_DISABLE_BOTS=true` to completely disable the bot system at startup — this skips all bot execution, returns 403 on bot settings updates, and shows a disabled message in the frontend. +3. **Arbitrary bot code execution**: The bot system (`app/fanout/bot_exec.py`) executes user-provided Python via `exec()` with full `__builtins__`. This is intentional — bots are a power-user feature for automation. The README explicitly warns that anyone on the network can execute arbitrary code through this. Operators can set `MESHCORE_DISABLE_BOTS=true` to completely disable the bot system at startup — this skips all bot execution, returns 403 on bot settings updates, and shows a disabled message in the frontend. ## Intentional Packet Handling Decision @@ -147,17 +147,14 @@ This message-layer echo/path handling is independent of raw-packet storage dedup . ├── app/ # FastAPI backend │ ├── AGENTS.md # Backend documentation -│ ├── bot.py # Bot execution and outbound bot sends │ ├── main.py # App entry, lifespan │ ├── routers/ # API endpoints │ ├── packet_processor.py # Raw packet pipeline, dedup, path handling -│ ├── repository/ # Database CRUD (contacts, channels, messages, raw_packets, settings) +│ ├── repository/ # Database CRUD (contacts, channels, messages, raw_packets, settings, fanout) │ ├── event_handlers.py # Radio events │ ├── decoder.py # Packet decryption │ ├── websocket.py # Real-time broadcasts -│ ├── mqtt_base.py # Shared MQTT publisher base class (lifecycle, reconnect, backoff) -│ ├── mqtt.py # Private MQTT publisher -│ └── community_mqtt.py # Community MQTT publisher (raw packet sharing) +│ └── fanout/ # Fanout bus: MQTT, bots, webhooks, Apprise (see fanout/AGENTS_fanout.md) ├── frontend/ # React frontend │ ├── AGENTS.md # Frontend documentation │ ├── src/ @@ -262,7 +259,7 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | Method | Endpoint | Description | |--------|----------|-------------| -| GET | `/api/health` | Connection status | +| GET | `/api/health` | Connection status, fanout statuses, bots_disabled flag | | GET | `/api/radio/config` | Radio configuration | | PATCH | `/api/radio/config` | Update name, location, radio params | | PUT | `/api/radio/private-key` | Import private key to radio | @@ -315,6 +312,10 @@ All endpoints are prefixed with `/api` (e.g., `/api/health`). | POST | `/api/settings/blocked-keys/toggle` | Toggle blocked key | | POST | `/api/settings/blocked-names/toggle` | Toggle blocked name | | POST | `/api/settings/migrate` | One-time migration from frontend localStorage | +| GET | `/api/fanout` | List all fanout configs | +| POST | `/api/fanout` | Create new fanout config | +| PATCH | `/api/fanout/{id}` | Update fanout config (triggers module reload) | +| DELETE | `/api/fanout/{id}` | Delete fanout config (stops module) | | GET | `/api/statistics` | Aggregated mesh network statistics | | WS | `/api/ws` | Real-time updates | @@ -360,33 +361,11 @@ Read state (`last_read_at`) is tracked **server-side** for consistency across de **Note:** These are NOT the same as `Message.conversation_key` (the database field). -### MQTT Publishing +### Fanout Bus (MQTT, Bots, Webhooks, Apprise) -Optional MQTT integration forwards mesh events to an external broker for home automation, logging, or alerting. All MQTT config is stored in the database (`app_settings`), not env vars — configured from the Settings pane, no server restart needed. +All external integrations are managed through the fanout bus (`app/fanout/`). Each integration is a `FanoutModule` with scope-based event filtering, stored in the `fanout_configs` table and managed via `GET/POST/PATCH/DELETE /api/fanout`. -**Two independent toggles**: publish decrypted messages, publish raw packets. - -**Topic structure** (default prefix `meshcore`): -- `meshcore/dm:` — decrypted DM -- `meshcore/gm:` — decrypted channel message -- `meshcore/raw/dm:` — raw packet attributed to a DM contact -- `meshcore/raw/gm:` — raw packet attributed to a channel -- `meshcore/raw/unrouted` — raw packets that couldn't be attributed - -**Architecture**: `broadcast_event()` in `websocket.py` calls `mqtt_broadcast()` — a single hook covering all message and raw_packet broadcasts. The `MqttPublisher` in `app/mqtt.py` manages a background connection loop with auto-reconnect and backoff. Publishes are fire-and-forget (silent drop if disconnected). Connection state changes trigger toasts via `broadcast_error`/`broadcast_success`. The health endpoint includes `mqtt_status` (`disabled` when no broker host is set, or when both publish toggles are off). - -**Security**: MQTT password stored in plaintext in SQLite, consistent with the project's trusted-network design. - -### Community MQTT Sharing - -Separate from private MQTT, the community publisher (`app/community_mqtt.py`) shares raw packets with the MeshCore community aggregator for coverage mapping and analysis. Only raw packets are shared — never decrypted messages. - -- Connects to community broker (default `mqtt-us-v1.letsmesh.net:443`) via WebSockets over TLS. -- Authentication via Ed25519 JWT signed with the radio's private key. Tokens auto-renew before 24h expiry. -- Broker address: separate `community_mqtt_broker_host` and `community_mqtt_broker_port` fields; defaults to `mqtt-us-v1.letsmesh.net:443`. -- Topic: `meshcore/{IATA}/{pubkey}/packets` — IATA is a 3-letter region code. -- JWT `email` claim enables node claiming on the community aggregator. -- Config: `community_mqtt_enabled`, `community_mqtt_iata`, `community_mqtt_broker_host`, `community_mqtt_broker_port`, `community_mqtt_email` in `app_settings`. +`broadcast_event()` in `websocket.py` dispatches `message` and `raw_packet` events to the fanout manager. See `app/fanout/AGENTS_fanout.md` for full architecture details. ### Server-Side Decryption @@ -430,7 +409,7 @@ mc.subscribe(EventType.ACK, handler) | `MESHCORE_DATABASE_PATH` | `data/meshcore.db` | SQLite database location | | `MESHCORE_DISABLE_BOTS` | `false` | Disable bot system entirely (blocks execution and config) | -**Note:** Runtime app settings are stored in the database (`app_settings` table), not environment variables. These include `max_radio_contacts`, `auto_decrypt_dm_on_advert`, `sidebar_sort_order`, `advert_interval`, `last_advert_time`, `favorites`, `last_message_times`, `bots`, all MQTT configuration (`mqtt_broker_host`, `mqtt_broker_port`, `mqtt_username`, `mqtt_password`, `mqtt_use_tls`, `mqtt_tls_insecure`, `mqtt_topic_prefix`, `mqtt_publish_messages`, `mqtt_publish_raw_packets`), community MQTT configuration (`community_mqtt_enabled`, `community_mqtt_iata`, `community_mqtt_broker_host`, `community_mqtt_broker_port`, `community_mqtt_email`), `flood_scope`, `blocked_keys`, and `blocked_names`. They are configured via `GET/PATCH /api/settings` (and related settings endpoints). +**Note:** Runtime app settings are stored in the database (`app_settings` table), not environment variables. These include `max_radio_contacts`, `auto_decrypt_dm_on_advert`, `sidebar_sort_order`, `advert_interval`, `last_advert_time`, `favorites`, `last_message_times`, `flood_scope`, `blocked_keys`, and `blocked_names`. They are configured via `GET/PATCH /api/settings`. MQTT, bot, webhook, and Apprise configs are stored in the `fanout_configs` table, managed via `/api/fanout`. Byte-perfect channel retries are user-triggered via `POST /api/messages/channel/{message_id}/resend` and are allowed for 30 seconds after the original send. diff --git a/LICENSES.md b/LICENSES.md index 12419d56..10971fae 100644 --- a/LICENSES.md +++ b/LICENSES.md @@ -56,6 +56,41 @@ SOFTWARE. +### apprise (1.9.7) — BSD-2-Clause + +
+Full license text + +``` +BSD 2-Clause License + +Copyright (c) 2025, Chris Caron +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` + +
+ ### fastapi (0.128.0) — MIT
@@ -87,6 +122,28 @@ THE SOFTWARE.
+### httpx (0.28.1) — BSD License + +
+Full license text + +``` +Copyright © 2019, [Encode OSS Ltd](https://www.encode.io/). +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` + +
+ ### meshcore (2.2.5) — MIT
diff --git a/app/AGENTS.md b/app/AGENTS.md index 9820ae10..50d083c1 100644 --- a/app/AGENTS.md +++ b/app/AGENTS.md @@ -27,10 +27,7 @@ app/ ├── packet_processor.py # Raw packet pipeline, dedup, path handling ├── event_handlers.py # MeshCore event subscriptions and ACK tracking ├── websocket.py # WS manager + broadcast helpers -├── mqtt_base.py # Shared MQTT publisher base class (lifecycle, reconnect, backoff) -├── mqtt.py # Private MQTT publisher (fire-and-forget forwarding) -├── community_mqtt.py # Community MQTT publisher (raw packet sharing) -├── bot.py # Bot execution and outbound bot sends +├── fanout/ # Fanout bus: MQTT, bots, webhooks, Apprise (see fanout/AGENTS_fanout.md) ├── dependencies.py # Shared FastAPI dependency providers ├── keystore.py # Ephemeral private/public key storage for DM decryption ├── frontend_static.py # Mount/serve built frontend (production) @@ -43,6 +40,7 @@ app/ ├── packets.py ├── read_state.py ├── settings.py + ├── fanout.py ├── repeaters.py ├── statistics.py └── ws.py @@ -103,33 +101,13 @@ app/ - `0` means disabled. - Last send time tracked in `app_settings.last_advert_time`. -### MQTT publishing +### Fanout bus -- Optional forwarding of mesh events to an external MQTT broker. -- All config in `app_settings` (not env vars): `mqtt_broker_host`, `mqtt_broker_port`, `mqtt_username`, `mqtt_password`, `mqtt_use_tls`, `mqtt_tls_insecure`, `mqtt_topic_prefix`, `mqtt_publish_messages`, `mqtt_publish_raw_packets`. -- Disabled when `mqtt_broker_host` is empty, or when both publish toggles are off (`mqtt_publish_messages=false` and `mqtt_publish_raw_packets=false`). -- `broadcast_event()` in `websocket.py` calls `mqtt_broadcast()` — single hook covers all message and raw_packet events. -- `MqttPublisher` (`app/mqtt.py`) runs a background connection loop with auto-reconnect and exponential backoff (5s → 30s). -- Publishes are fire-and-forget; individual publish failures logged but not surfaced to users. -- Connection state changes surface via `broadcast_error`/`broadcast_success` toasts. -- Health endpoint includes `mqtt_status` field (`connected`, `disconnected`, `disabled`), where `disabled` covers both "no broker host configured" and "nothing enabled to publish". -- Settings changes trigger `mqtt_publisher.restart()` — no server restart needed. -- Topics: `{prefix}/dm:{key}`, `{prefix}/gm:{key}`, `{prefix}/raw/dm:{key}`, `{prefix}/raw/gm:{key}`, `{prefix}/raw/unrouted`. - -### Community MQTT - -- Separate publisher (`app/community_mqtt.py`) for sharing raw packets with the MeshCore community aggregator. -- Implementation intent: keep functional parity with the reference implementation at `https://github.com/agessaman/meshcore-packet-capture` unless this repository explicitly documents a deliberate deviation. -- Independent from the private `MqttPublisher` — different broker, authentication, and topic structure. -- Connects to the community broker (default `mqtt-us-v1.letsmesh.net:443`) via WebSockets over TLS. -- Authentication: Ed25519 JWT tokens signed with the radio's expanded "orlp" private key. Tokens expire after 24 hours; proactive renewal at 23 hours. -- Broker address: separate `community_mqtt_broker_host` and `community_mqtt_broker_port` fields; defaults to `mqtt-us-v1.letsmesh.net:443`. -- JWT claims include `publicKey`, `owner` (radio pubkey), `client` (app identifier), and optional `email` (for node claiming on the community aggregator). -- Topic: `meshcore/{IATA}/{pubkey}/packets` — IATA is a 3-letter region code (required to enable; no default). -- Only raw packets are published — never decrypted messages. -- Publishes are fire-and-forget. The connection loop detects publish failures via `connected` flag and reconnects within 60 seconds. -- Health endpoint includes `community_mqtt_status` field. -- Settings: `community_mqtt_enabled`, `community_mqtt_iata`, `community_mqtt_broker_host`, `community_mqtt_broker_port`, `community_mqtt_email`. +- All external integrations (MQTT, bots, webhooks, Apprise) are managed through the fanout bus (`app/fanout/`). +- Configs stored in `fanout_configs` table, managed via `GET/POST/PATCH/DELETE /api/fanout`. +- `broadcast_event()` in `websocket.py` dispatches to the fanout manager for `message` and `raw_packet` events. +- Each integration is a `FanoutModule` with scope-based filtering. +- See `app/fanout/AGENTS_fanout.md` for full architecture details. ## API Surface (all under `/api`) @@ -201,6 +179,12 @@ app/ - `POST /settings/blocked-names/toggle` - `POST /settings/migrate` +### Fanout +- `GET /fanout` — list all fanout configs +- `POST /fanout` — create new fanout config +- `PATCH /fanout/{id}` — update fanout config (triggers module reload) +- `DELETE /fanout/{id}` — delete fanout config (stops module) + ### Statistics - `GET /statistics` — aggregated mesh network stats (entity counts, message/packet splits, activity windows, busiest channels) @@ -242,13 +226,11 @@ Main tables: - `preferences_migrated` - `advert_interval` - `last_advert_time` -- `bots` -- `mqtt_broker_host`, `mqtt_broker_port`, `mqtt_username`, `mqtt_password` -- `mqtt_use_tls`, `mqtt_tls_insecure`, `mqtt_topic_prefix`, `mqtt_publish_messages`, `mqtt_publish_raw_packets` -- `community_mqtt_enabled`, `community_mqtt_iata`, `community_mqtt_broker_host`, `community_mqtt_broker_port`, `community_mqtt_email` - `flood_scope` - `blocked_keys`, `blocked_names` +Note: MQTT, community MQTT, and bot configs were migrated to the `fanout_configs` table (migrations 36-38). + ## Security Posture (intentional) - No authn/authz. @@ -279,6 +261,8 @@ tests/ ├── test_decoder.py # Packet parsing/decryption ├── test_disable_bots.py # MESHCORE_DISABLE_BOTS=true feature ├── test_echo_dedup.py # Echo/repeat deduplication (incl. concurrent) +├── test_fanout.py # Fanout bus CRUD, scope matching, manager dispatch +├── test_fanout_integration.py # Fanout integration tests ├── test_event_handlers.py # ACK tracking, event registration, cleanup ├── test_frontend_static.py # Frontend static file serving ├── test_health_mqtt_status.py # Health endpoint MQTT status field diff --git a/app/AGENTS_MQTT.md b/app/AGENTS_MQTT.md deleted file mode 100644 index b2299933..00000000 --- a/app/AGENTS_MQTT.md +++ /dev/null @@ -1,377 +0,0 @@ -# MQTT Architecture - -RemoteTerm implements two independent MQTT publishing systems that share a common base class: - -1. **Private MQTT** — forwards mesh events to a user-configured broker (home automation, logging, alerting) -2. **Community MQTT** — shares raw RF packets with the MeshCore community aggregator for coverage mapping - -Both are optional, configured entirely through the Settings UI, and require no server restart. - -## File Map - -``` -app/ -├── mqtt_base.py # BaseMqttPublisher — shared lifecycle, connection loop, reconnect -├── mqtt.py # MqttPublisher — private broker forwarding -├── community_mqtt.py # CommunityMqttPublisher — community aggregator integration -├── keystore.py # In-memory Ed25519 key storage (community auth) -├── models.py # AppSettings — all MQTT fields (14 total) -├── repository/settings.py # Database CRUD for MQTT settings -├── routers/settings.py # PATCH /api/settings — validates + restarts publishers -├── routers/health.py # GET /api/health — mqtt_status, community_mqtt_status -├── websocket.py # broadcast_event() — fans out to WS + both MQTT publishers -└── migrations.py # Migration 031 (private fields), 032 (community fields) - -frontend/src/ -├── components/settings/SettingsMqttSection.tsx # Dual collapsible settings UI -└── types.ts # AppSettings, AppSettingsUpdate, HealthStatus - -tests/ -├── test_mqtt.py # Topic routing, lifecycle -├── test_community_mqtt.py # JWT generation, packet format, hash, broadcast -└── test_health_mqtt_status.py # Health endpoint status reporting -``` - -## Base Publisher (`app/mqtt_base.py`) - -`BaseMqttPublisher` is an abstract class that manages the full MQTT client lifecycle for both publishers. Subclasses implement hooks; the base class owns the connection loop. - -### Connection Loop - -The `_connection_loop()` runs as a background `asyncio.Task` and never exits unless cancelled: - -``` -loop: - ├─ _is_configured()? No → call _on_not_configured(), wait for settings change, loop - ├─ _pre_connect()? False → wait and retry - ├─ Build client via _build_client_kwargs() - ├─ Connect with aiomqtt.Client - ├─ Set connected=True, broadcast success toast via _on_connected() - ├─ Wait in 60s intervals: - │ ├─ _on_periodic_wake(elapsed) → subclass hook (e.g., periodic status republish) - │ ├─ Settings version changed? → break, reconnect with new settings - │ ├─ _should_break_wait()? → break (e.g., JWT expiry) - │ └─ Otherwise keep waiting (paho-mqtt handles keepalive internally) - ├─ On error: set connected=False, broadcast error toast, exponential backoff - └─ On cancel: cleanup and exit -``` - -### Abstract Hooks - -| Hook | Returns | Purpose | -|------|---------|---------| -| `_is_configured()` | `bool` | Should the publisher attempt to connect? | -| `_build_client_kwargs(settings)` | `dict` | Arguments for `aiomqtt.Client(...)` | -| `_on_connected(settings)` | `(title, detail)` | Success toast content | -| `_on_error()` | `(title, detail)` | Error toast content | - -### Optional Hooks - -| Hook | Default | Purpose | -|------|---------|---------| -| `_pre_connect(settings)` | `return True` | Async setup before connect; return `False` to retry | -| `_should_break_wait(elapsed)` | `return False` | Force reconnect while connected (e.g., token renewal) | -| `_on_not_configured()` | no-op | Called repeatedly while waiting for configuration | -| `_on_periodic_wake(elapsed)` | no-op | Called every ~60s while connected (e.g., periodic status republish) | - -### Lifecycle Methods - -- `start(settings)` — stores settings, starts the background loop task -- `stop()` — cancels the task, disconnects the client -- `restart(settings)` — `stop()` then `start()` (called when settings change) -- `publish(topic, payload)` — JSON-serializes and publishes; silently drops if disconnected - -### Backoff - -Reconnect delay: 5 seconds minimum, exponential growth, capped at `_backoff_max` (30s for private, 60s for community). Resets on successful connect. - -### QoS - -All publishing uses QoS 0 (at-most-once delivery), the aiomqtt default. - -## Private MQTT (`app/mqtt.py`) - -### When It Connects - -`_is_configured()` returns `True` when all of: -- `mqtt_broker_host` is non-empty -- At least one of `mqtt_publish_messages` or `mqtt_publish_raw_packets` is enabled - -If the user unchecks both publish toggles and saves, the publisher disconnects and the health status shows "Disabled". - -### Client Configuration - -```python -hostname: settings.mqtt_broker_host -port: settings.mqtt_broker_port (default 1883) -username: settings.mqtt_username or None -password: settings.mqtt_password or None -tls_context: ssl.create_default_context() if mqtt_use_tls, else None - # mqtt_tls_insecure=True disables hostname check + cert verification -``` - -TLS is opt-in. When enabled with `mqtt_tls_insecure`, both `check_hostname` and `verify_mode` are relaxed for self-signed certificates. - -### Topic Structure - -Default prefix: `meshcore` (configurable via `mqtt_topic_prefix`). - -**Decrypted messages** (when `mqtt_publish_messages` is on): -- `{prefix}/dm:{contact_key}` — private DM -- `{prefix}/gm:{channel_key}` — channel message -- `{prefix}/message:{conversation_key}` — fallback for unknown type - -**Raw packets** (when `mqtt_publish_raw_packets` is on): -- `{prefix}/raw/dm:{contact_key}` — attributed to a DM contact -- `{prefix}/raw/gm:{channel_key}` — attributed to a channel -- `{prefix}/raw/unrouted` — unattributed - -Topic routing uses `decrypted_info.contact_key` and `decrypted_info.channel_key` from the raw packet data. - -### Fire-and-Forget Pattern - -`mqtt_broadcast(event_type, data)` is called synchronously from `broadcast_event()` in `websocket.py`. It filters to only `"message"` and `"raw_packet"` events, then creates an `asyncio.Task` for the actual publish. No awaiting — failures are logged at WARNING level and silently dropped. - -## Community MQTT (`app/community_mqtt.py`) - -Implements the [meshcore-packet-capture](https://github.com/agessaman/meshcore-packet-capture) protocol for sharing raw RF packets with the MeshCore community aggregator. - -### When It Connects - -`_is_configured()` returns `True` when all of: -- `community_mqtt_enabled` is `True` -- The radio's private key is available in the keystore (`has_private_key()`) - -The private key is exported from the radio firmware on startup via `export_and_store_private_key()` in `app/keystore.py`. This requires `ENABLE_PRIVATE_KEY_EXPORT` to be enabled in the radio firmware. If unavailable, the publisher broadcasts a warning and waits. - -### Client Configuration - -```python -hostname: community_mqtt_broker_host or "mqtt-us-v1.letsmesh.net" -port: community_mqtt_broker_port or 443 -transport: "websockets" -tls_context: ssl.create_default_context() # always enforced, not user-configurable -websocket_path: "/" -username: "v1_{pubkey_hex}" -password: {jwt_token} -``` - -TLS is always on — the community connection uses WebSocket Secure (WSS) with full certificate verification. There is no option to disable it. - -### JWT Authentication - -The community broker authenticates via Ed25519-signed JWT tokens. - -**Token format:** `header_b64url.payload_b64url.signature_hex` - -**Header:** -```json -{"alg": "Ed25519", "typ": "JWT"} -``` - -**Payload:** -```json -{ - "publicKey": "{PUBKEY_HEX_UPPER}", - "iat": 1234567890, - "exp": 1234654290, - "aud": "{broker_host}", - "owner": "{PUBKEY_HEX_UPPER}", - "client": "RemoteTerm (github.com/jkingsman/Remote-Terminal-for-MeshCore)", - "email": "user@example.com" // optional, only if configured -} -``` - -**Signing:** MeshCore uses an "expanded" 64-byte Ed25519 key format (`scalar[32] || prefix[32]`, the "orlp" format). Standard Ed25519 libraries expect seed format and would re-hash the key. The `_ed25519_sign_expanded()` function performs signing manually using `nacl.bindings.crypto_scalarmult_ed25519_base_noclamp()` — a direct port of meshcore-packet-capture's `ed25519_sign_with_expanded_key()`. - -**Token lifetime:** 24 hours. The `_should_break_wait()` hook forces a reconnect at the 23-hour mark to renew before expiry. - -### Status Messages - -On connect and every 5 minutes thereafter, the community publisher sends a retained status message to `meshcore/{IATA}/{PUBKEY}/status` with device info and radio telemetry: - -```json -{ - "status": "online", - "timestamp": "2024-01-15T10:30:00.000000", - "origin": "NodeName", - "origin_id": "PUBKEY_HEX_UPPER", - "model": "T-Deck", - "firmware_version": "v2.2.2 (Build: 2025-01-15)", - "radio": "915.0,250.0,10,8", - "client_version": "RemoteTerm 2.4.0", - "stats": { - "battery_mv": 4200, - "uptime_secs": 3600, - "errors": 0, - "queue_len": 0, - "noise_floor": -120, - "last_rssi": -85, - "last_snr": 10.5, - "tx_air_secs": 42, - "rx_air_secs": 150 - } -} -``` - -- `model` and `firmware_version` are fetched once per connection via `send_device_query()` (requires firmware version >= 3) -- `radio` is comma-separated raw values from `self_info` (freq, BW, SF, CR) matching the reference format -- `client_version` is read from Python package metadata (`remoteterm-meshcore`) -- `stats` is fetched from `get_stats_core()` + `get_stats_radio()` every 5 minutes; omitted if firmware doesn't support stats commands -- All radio queries use `blocking=False` — if the radio is busy, cached values are used. No user-facing operations are ever blocked. -- LWT (Last Will and Testament) publishes `{"status": "offline", ...}` on the same topic with retain - -### Packet Formatting - -`_format_raw_packet()` converts raw packet broadcast data into the meshcore-packet-capture JSON format: - -```json -{ - "origin": "NodeName", - "origin_id": "PUBKEY_HEX_UPPER", - "timestamp": "2024-01-15T10:30:00.000000", - "type": "PACKET", - "direction": "rx", - "time": "10:30:00", - "date": "15/01/2024", - "len": "42", - "packet_type": "5", - "route": "F", - "payload_len": "30", - "raw": "AABBCCDD...", - "SNR": "10.5", - "RSSI": "-85", - "hash": "A1B2C3D4E5F6G7H8", - "path": "ab,cd,ef" -} -``` - -- `origin` is the radio's device name from `meshcore.self_info` -- `route` is derived from the header's bottom 2 bits: `0,1→"F"` (Flood), `2→"D"` (Direct), `3→"T"` (Trace) -- `path` is only present when `route=="D"` -- `hash` matches MeshCore's C++ `Packet::calculatePacketHash()`: SHA-256 of `payload_type[1 byte] + [path_len as uint16 LE, TRACE only] + payload_data`, truncated to first 16 hex characters - -### Topic Structure - -``` -meshcore/{IATA}/{PUBKEY_HEX}/packets -``` - -IATA must be exactly 3 uppercase letters (e.g., `DEN`, `LAX`). Validated both client-side (input maxLength + uppercase conversion) and server-side (regex `^[A-Z]{3}$`, returns HTTP 400 on failure). - -### Only Raw Packets - -The community publisher only handles `"raw_packet"` events. Decrypted messages are never shared with the community — `community_mqtt_broadcast()` explicitly filters `event_type != "raw_packet"`. - -## Event Flow - -``` -Radio RF event - ↓ -meshcore_py library callback - ↓ -app/event_handlers.py (on_contact_message, on_rx_log_data, etc.) - ↓ -Store to SQLite database - ↓ -broadcast_event(event_type, data) ← app/websocket.py - ├─ WebSocket → browser clients - ├─ mqtt_broadcast() ← app/mqtt.py (messages + raw packets) - │ └─ asyncio.create_task(_mqtt_maybe_publish()) - └─ community_mqtt_broadcast() ← app/community_mqtt.py (raw packets only) - └─ asyncio.create_task(_community_maybe_publish()) -``` - -## Settings & Persistence - -### Database Fields (`app_settings` table) - -**Private MQTT** (Migration 031): - -| Column | Type | Default | -|--------|------|---------| -| `mqtt_broker_host` | TEXT | `''` | -| `mqtt_broker_port` | INTEGER | `1883` | -| `mqtt_username` | TEXT | `''` | -| `mqtt_password` | TEXT | `''` | -| `mqtt_use_tls` | INTEGER | `0` | -| `mqtt_tls_insecure` | INTEGER | `0` | -| `mqtt_topic_prefix` | TEXT | `'meshcore'` | -| `mqtt_publish_messages` | INTEGER | `0` | -| `mqtt_publish_raw_packets` | INTEGER | `0` | - -**Community MQTT** (Migration 032): - -| Column | Type | Default | -|--------|------|---------| -| `community_mqtt_enabled` | INTEGER | `0` | -| `community_mqtt_iata` | TEXT | `''` | -| `community_mqtt_broker_host` | TEXT | `'mqtt-us-v1.letsmesh.net'` | -| `community_mqtt_broker_port` | INTEGER | `443` | -| `community_mqtt_email` | TEXT | `''` | - -### Settings API - -`PATCH /api/settings` accepts any subset of MQTT fields. The router tracks whether private or community fields changed independently: - -- If any private MQTT field changed → `await mqtt_publisher.restart(result)` -- If any community MQTT field changed → `await community_publisher.restart(result)` - -This means toggling a publish checkbox triggers a full disconnect/reconnect cycle. - -### Health API - -`GET /api/health` reports both statuses: - -```json -{ - "mqtt_status": "connected | disconnected | disabled", - "community_mqtt_status": "connected | disconnected | disabled" -} -``` - -Status logic for each publisher: -- `_is_configured()` returns `True` → report `"connected"` or `"disconnected"` based on `publisher.connected` -- `_is_configured()` returns `False` → report `"disabled"` - -## App Lifecycle - -**Startup** (in `app/main.py` lifespan): -1. Database connects, radio connects -2. `export_and_store_private_key()` — export Ed25519 key from radio (needed for community auth) -3. Load `AppSettings` from database -4. `mqtt_publisher.start(settings)` — spawns background connection loop -5. `community_publisher.start(settings)` — spawns background connection loop - -**Shutdown:** -1. `community_publisher.stop()` -2. `mqtt_publisher.stop()` -3. Radio and database cleanup - -## Frontend (`SettingsMqttSection.tsx`) - -The MQTT settings UI is a single React component with two collapsible sections (both collapsed by default): - -### Private MQTT Broker Section -- Header shows connection status indicator (green/red/gray dot + label) -- Always visible when expanded: Publish Messages and Publish Raw Packets checkboxes -- Broker configuration (host, port, username, password, TLS, topic prefix) only revealed when at least one publish checkbox is checked -- Responsive grid layout (`grid-cols-1 sm:grid-cols-2`) for host+port and username+password pairs - -### Community Analytics Section -- Header shows connection status indicator -- Enable Community Analytics checkbox -- When enabled: broker host/port, IATA code input (3 chars, auto-uppercase), owner email -- Broker host shows "MQTT over TLS (WebSocket Secure) only" note - -### Shared -- Beta warning banner at the top (links to GitHub issues) -- Single "Save MQTT Settings" button outside both collapsibles -- Save constructs an `AppSettingsUpdate` and calls `PATCH /api/settings` -- Success/error feedback via toast notifications - -## Security Notes - -- **Private MQTT password** is stored in plaintext in SQLite, consistent with the project's trusted-network design. -- **Community MQTT** always uses TLS with full certificate verification. The Ed25519 private key is held in memory only (never persisted to disk) and is used solely for JWT signing. -- **Community data** is limited to raw RF packets — decrypted message content is never shared. diff --git a/app/config.py b/app/config.py index 36c80e98..20617ec6 100644 --- a/app/config.py +++ b/app/config.py @@ -48,6 +48,40 @@ class Settings(BaseSettings): settings = Settings() +class _RepeatSquelch(logging.Filter): + """Suppress rapid-fire identical messages and emit a summary instead. + + Attached to the ``meshcore`` library logger to catch its repeated + "Serial Connection started" lines that flood the log when another + process holds the serial port. + """ + + def __init__(self, threshold: int = 3) -> None: + super().__init__() + self._last_msg: str | None = None + self._repeat_count: int = 0 + self._threshold = threshold + + def filter(self, record: logging.LogRecord) -> bool: + msg = record.getMessage() + if msg == self._last_msg: + self._repeat_count += 1 + if self._repeat_count == self._threshold: + record.msg = ( + "%s (repeated %d times — possible serial port contention from another process)" + ) + record.args = (msg, self._repeat_count) + record.levelno = logging.WARNING + record.levelname = "WARNING" + return True + # Suppress further repeats beyond the threshold + return self._repeat_count < self._threshold + else: + self._last_msg = msg + self._repeat_count = 1 + return True + + def setup_logging() -> None: """Configure logging for the application.""" logging.basicConfig( @@ -55,3 +89,6 @@ def setup_logging() -> None: format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", ) + # Squelch repeated messages from the meshcore library (e.g. rapid-fire + # "Serial Connection started" when the port is contended). + logging.getLogger("meshcore").addFilter(_RepeatSquelch()) diff --git a/app/event_handlers.py b/app/event_handlers.py index 84dceb39..e0cd6e6d 100644 --- a/app/event_handlers.py +++ b/app/event_handlers.py @@ -1,4 +1,3 @@ -import asyncio import logging import time from typing import TYPE_CHECKING @@ -155,23 +154,6 @@ async def on_contact_message(event: "Event") -> None: if contact: await ContactRepository.update_last_contacted(sender_pubkey, received_at) - # Run bot if enabled - from app.bot import run_bot_for_message - - asyncio.create_task( - run_bot_for_message( - sender_name=contact.name if contact else None, - sender_key=sender_pubkey, - message_text=payload.get("text", ""), - is_dm=True, - channel_key=None, - channel_name=None, - sender_timestamp=payload.get("sender_timestamp"), - path=payload.get("path"), - is_outgoing=False, - ) - ) - async def on_rx_log_data(event: "Event") -> None: """Store raw RF packet data and process via centralized packet processor. diff --git a/app/fanout/AGENTS_fanout.md b/app/fanout/AGENTS_fanout.md new file mode 100644 index 00000000..b816f1ca --- /dev/null +++ b/app/fanout/AGENTS_fanout.md @@ -0,0 +1,284 @@ +# Fanout Bus Architecture + +The fanout bus is a unified system for dispatching mesh radio events (decoded messages and raw packets) to external integrations. It replaces the previous scattered singleton MQTT publishers with a modular, configurable framework. + +## Core Concepts + +### FanoutModule (base.py) +Base class that all integration modules extend: +- `__init__(config_id, config, *, name="")` — constructor; receives the config UUID, the type-specific config dict, and the user-assigned name +- `start()` / `stop()` — async lifecycle (e.g. open/close connections) +- `on_message(data)` — receive decoded messages (DM/channel) +- `on_raw(data)` — receive raw RF packets +- `status` property (**must override**) — return `"connected"`, `"disconnected"`, or `"error"` + +### FanoutManager (manager.py) +Singleton that owns all active modules and dispatches events: +- `load_from_db()` — startup: load enabled configs, instantiate modules +- `reload_config(id)` — CRUD: stop old, start new +- `remove_config(id)` — delete: stop and remove +- `broadcast_message(data)` — scope-check + dispatch `on_message` +- `broadcast_raw(data)` — scope-check + dispatch `on_raw` +- `stop_all()` — shutdown +- `get_statuses()` — health endpoint data + +All modules are constructed uniformly: `cls(config_id, config_blob, name=cfg.get("name", ""))`. + +### Scope Matching +Each config has a `scope` JSON blob controlling what events reach it: +```json +{"messages": "all", "raw_packets": "all"} +{"messages": "none", "raw_packets": "all"} +{"messages": {"channels": ["key1"], "contacts": "all"}, "raw_packets": "none"} +``` +Community MQTT always enforces `{"messages": "none", "raw_packets": "all"}`. + +## Event Flow + +``` +Radio Event -> packet_processor / event_handler + -> broadcast_event("message"|"raw_packet", data, realtime=True) + -> WebSocket broadcast (always) + -> FanoutManager.broadcast_message/raw (only if realtime=True) + -> scope check per module + -> module.on_message / on_raw +``` + +Setting `realtime=False` (used during historical decryption) skips fanout dispatch entirely. + +## Current Module Types + +### mqtt_private (mqtt_private.py) +Wraps `MqttPublisher` from `app/fanout/mqtt.py`. Config blob: +- `broker_host`, `broker_port`, `username`, `password` +- `use_tls`, `tls_insecure`, `topic_prefix` + +### mqtt_community (mqtt_community.py) +Wraps `CommunityMqttPublisher` from `app/fanout/community_mqtt.py`. Config blob: +- `broker_host`, `broker_port`, `iata`, `email` +- Only publishes raw packets (on_message is a no-op) + +### bot (bot.py) +Wraps bot code execution via `app/fanout/bot_exec.py`. Config blob: +- `code` — Python bot function source code +- Executes in a thread pool with timeout and semaphore concurrency control +- Rate-limits outgoing messages for repeater compatibility + +### webhook (webhook.py) +HTTP webhook delivery. Config blob: +- `url`, `method` (POST/PUT/PATCH) +- `hmac_secret` (optional) — when set, each request includes an HMAC-SHA256 signature of the JSON body +- `hmac_header` (optional, default `X-Webhook-Signature`) — header name for the signature (value format: `sha256=`) +- `headers` — arbitrary extra headers (JSON object) + +### apprise (apprise_mod.py) +Push notifications via Apprise library. Config blob: +- `urls` — newline-separated Apprise notification service URLs +- `preserve_identity` — suppress Discord webhook name/avatar override +- `include_path` — include routing path in notification body + +## Adding a New Integration Type + +### Step-by-step checklist + +#### 1. Backend module (`app/fanout/my_type.py`) + +Create a class extending `FanoutModule`: + +```python +from app.fanout.base import FanoutModule + +class MyTypeModule(FanoutModule): + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + super().__init__(config_id, config, name=name) + # Initialize module-specific state + + async def start(self) -> None: + """Open connections, create clients, etc.""" + + async def stop(self) -> None: + """Close connections, clean up resources.""" + + async def on_message(self, data: dict) -> None: + """Handle decoded messages. Omit if not needed.""" + + async def on_raw(self, data: dict) -> None: + """Handle raw packets. Omit if not needed.""" + + @property + def status(self) -> str: + """Required. Return 'connected', 'disconnected', or 'error'.""" + ... +``` + +Constructor requirements: +- Must accept `config_id: str, config: dict, *, name: str = ""` +- Must forward `name` to super: `super().__init__(config_id, config, name=name)` + +#### 2. Register in manager (`app/fanout/manager.py`) + +Add import and mapping in `_register_module_types()`: + +```python +from app.fanout.my_type import MyTypeModule +_MODULE_TYPES["my_type"] = MyTypeModule +``` + +#### 3. Router changes (`app/routers/fanout.py`) + +Three changes needed: + +**a)** Add to `_VALID_TYPES` set: +```python +_VALID_TYPES = {"mqtt_private", "mqtt_community", "bot", "webhook", "apprise", "my_type"} +``` + +**b)** Add a validation function: +```python +def _validate_my_type_config(config: dict) -> None: + """Validate my_type config blob.""" + if not config.get("some_required_field"): + raise HTTPException(status_code=400, detail="some_required_field is required") +``` + +**c)** Wire validation into both `create_fanout_config` and `update_fanout_config` — add an `elif` to the validation block in each: +```python +elif body.type == "my_type": + _validate_my_type_config(body.config) +``` +Note: validation only runs when the config will be enabled (disabled configs are treated as drafts). + +**d)** Add scope enforcement in `_enforce_scope()` if the type has fixed scope constraints (e.g. raw_packets always none). Otherwise it falls through to the `mqtt_private` default which allows both messages and raw_packets to be configurable. + +#### 4. Frontend editor component (`SettingsFanoutSection.tsx`) + +Four changes needed in this single file: + +**a)** Add to `TYPE_LABELS` and `TYPE_OPTIONS` at the top: +```tsx +const TYPE_LABELS: Record = { + // ... existing entries ... + my_type: 'My Type', +}; + +const TYPE_OPTIONS = [ + // ... existing entries ... + { value: 'my_type', label: 'My Type' }, +]; +``` + +**b)** Create an editor component (follows the same pattern as existing editors): +```tsx +function MyTypeConfigEditor({ + config, + scope, + onChange, + onScopeChange, +}: { + config: Record; + scope: Record; + onChange: (config: Record) => void; + onScopeChange: (scope: Record) => void; +}) { + return ( +
+ {/* Type-specific config fields */} + + +
+ ); +} +``` + +If your type does NOT have user-configurable scope (like bot or community MQTT), omit the `scope`/`onScopeChange` props and the `ScopeSelector`. + +The `ScopeSelector` component is defined within the same file. It accepts an optional `showRawPackets` prop: +- **Without `showRawPackets`** (webhook, apprise): shows message scope only (all/only/except — no "none" option since that would make the integration a no-op). A warning appears when the effective selection matches nothing. +- **With `showRawPackets`** (private MQTT): adds a "Forward raw packets" toggle and includes the "No messages" option (valid when raw packets are enabled). The warning appears only when both raw packets and messages are effectively disabled. + +**c)** Add default config and scope in `handleAddCreate`: +```tsx +const defaults: Record> = { + // ... existing entries ... + my_type: { some_field: '', other_field: true }, +}; +const defaultScopes: Record> = { + // ... existing entries ... + my_type: { messages: 'all', raw_packets: 'none' }, +}; +``` + +**d)** Wire the editor into the detail view's conditional render block: +```tsx +{editingConfig.type === 'my_type' && ( + +)} +``` + +#### 5. Tests + +**Backend integration tests** (`tests/test_fanout_integration.py`): +- Test that a configured + enabled module receives messages via `FanoutManager.broadcast_message` +- Test scope filtering (all, none, selective) +- Test that a disabled module does not receive messages + +**Backend unit tests** (`tests/test_fanout_hitlist.py` or a dedicated file): +- Test config validation (required fields, bad values) +- Test module-specific logic in isolation + +**Frontend tests** (`frontend/src/test/fanoutSection.test.tsx`): +- The existing suite covers the list/edit/create flow generically. If your editor has special behavior, add specific test cases. + +#### Summary of files to touch + +| File | Change | +|------|--------| +| `app/fanout/my_type.py` | New module class | +| `app/fanout/manager.py` | Import + register in `_register_module_types()` | +| `app/routers/fanout.py` | `_VALID_TYPES` + validator function + scope enforcement | +| `frontend/.../SettingsFanoutSection.tsx` | `TYPE_LABELS` + `TYPE_OPTIONS` + editor component + defaults + detail view wiring | +| `tests/test_fanout_integration.py` | Integration tests | + +## REST API + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/api/fanout` | List all fanout configs | +| POST | `/api/fanout` | Create new config | +| PATCH | `/api/fanout/{id}` | Update config (triggers module reload) | +| DELETE | `/api/fanout/{id}` | Delete config (stops module) | + +## Database + +`fanout_configs` table: +- `id` TEXT PRIMARY KEY +- `type`, `name`, `enabled`, `config` (JSON), `scope` (JSON) +- `sort_order`, `created_at` + +Migrations: +- **36**: Creates `fanout_configs` table, migrates existing MQTT settings from `app_settings` +- **37**: Migrates bot configs from `app_settings.bots` JSON column into fanout rows +- **38**: Drops legacy `mqtt_*`, `community_mqtt_*`, and `bots` columns from `app_settings` + +## Key Files + +- `app/fanout/base.py` — FanoutModule base class +- `app/fanout/manager.py` — FanoutManager singleton +- `app/fanout/mqtt_base.py` — BaseMqttPublisher ABC (shared MQTT connection loop) +- `app/fanout/mqtt.py` — MqttPublisher (private MQTT publishing) +- `app/fanout/community_mqtt.py` — CommunityMqttPublisher (community MQTT with JWT auth) +- `app/fanout/mqtt_private.py` — Private MQTT fanout module +- `app/fanout/mqtt_community.py` — Community MQTT fanout module +- `app/fanout/bot.py` — Bot fanout module +- `app/fanout/bot_exec.py` — Bot code execution, response processing, rate limiting +- `app/fanout/webhook.py` — Webhook fanout module +- `app/fanout/apprise_mod.py` — Apprise fanout module +- `app/repository/fanout.py` — Database CRUD +- `app/routers/fanout.py` — REST API +- `app/websocket.py` — `broadcast_event()` dispatches to fanout +- `frontend/src/components/settings/SettingsFanoutSection.tsx` — UI diff --git a/app/fanout/__init__.py b/app/fanout/__init__.py new file mode 100644 index 00000000..885a25bf --- /dev/null +++ b/app/fanout/__init__.py @@ -0,0 +1,8 @@ +from app.fanout.base import FanoutModule +from app.fanout.manager import FanoutManager, fanout_manager + +__all__ = [ + "FanoutManager", + "FanoutModule", + "fanout_manager", +] diff --git a/app/fanout/apprise_mod.py b/app/fanout/apprise_mod.py new file mode 100644 index 00000000..0cbf31d8 --- /dev/null +++ b/app/fanout/apprise_mod.py @@ -0,0 +1,125 @@ +"""Fanout module for Apprise push notifications.""" + +from __future__ import annotations + +import asyncio +import logging +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit + +from app.fanout.base import FanoutModule + +logger = logging.getLogger(__name__) + + +def _parse_urls(raw: str) -> list[str]: + """Split multi-line URL string into individual URLs.""" + return [line.strip() for line in raw.splitlines() if line.strip()] + + +def _normalize_discord_url(url: str) -> str: + """Add avatar=no to Discord URLs to suppress identity override.""" + parts = urlsplit(url) + scheme = parts.scheme.lower() + host = parts.netloc.lower() + + is_discord = scheme in ("discord", "discords") or ( + scheme in ("http", "https") + and host in ("discord.com", "discordapp.com") + and parts.path.lower().startswith("/api/webhooks/") + ) + if not is_discord: + return url + + query = dict(parse_qsl(parts.query, keep_blank_values=True)) + query["avatar"] = "no" + return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment)) + + +def _format_body(data: dict, *, include_path: bool) -> str: + """Build a human-readable notification body from message data.""" + msg_type = data.get("type", "") + text = data.get("text", "") + sender_name = data.get("sender_name") or "Unknown" + + via = "" + if include_path: + paths = data.get("paths") + if paths and isinstance(paths, list) and len(paths) > 0: + path_str = paths[0].get("path", "") if isinstance(paths[0], dict) else "" + else: + path_str = None + + if msg_type == "PRIV" and path_str is None: + via = " **via:** [`direct`]" + elif path_str is not None: + path_str = path_str.strip().lower() + if path_str == "": + via = " **via:** [`direct`]" + else: + hops = [path_str[i : i + 2] for i in range(0, len(path_str), 2)] + if hops: + hop_list = ", ".join(f"`{h}`" for h in hops) + via = f" **via:** [{hop_list}]" + + if msg_type == "PRIV": + return f"**DM:** {sender_name}: {text}{via}" + + channel_name = data.get("channel_name") or data.get("conversation_key", "channel") + return f"**{channel_name}:** {sender_name}: {text}{via}" + + +def _send_sync(urls_raw: str, body: str, *, preserve_identity: bool) -> bool: + """Send notification synchronously via Apprise. Returns True on success.""" + import apprise as apprise_lib + + urls = _parse_urls(urls_raw) + if not urls: + return False + + notifier = apprise_lib.Apprise() + for url in urls: + if preserve_identity: + url = _normalize_discord_url(url) + notifier.add(url) + + return bool(notifier.notify(title="", body=body)) + + +class AppriseModule(FanoutModule): + """Sends push notifications via Apprise for incoming messages.""" + + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + super().__init__(config_id, config, name=name) + self._last_error: str | None = None + + async def on_message(self, data: dict) -> None: + # Skip outgoing messages — only notify on incoming + if data.get("outgoing"): + return + + urls = self.config.get("urls", "") + if not urls or not urls.strip(): + return + + preserve_identity = self.config.get("preserve_identity", True) + include_path = self.config.get("include_path", True) + body = _format_body(data, include_path=include_path) + + try: + success = await asyncio.to_thread( + _send_sync, urls, body, preserve_identity=preserve_identity + ) + self._last_error = None if success else "Apprise notify returned failure" + if not success: + logger.warning("Apprise notification failed for module %s", self.config_id) + except Exception as exc: + self._last_error = str(exc) + logger.exception("Apprise send error for module %s", self.config_id) + + @property + def status(self) -> str: + if not self.config.get("urls", "").strip(): + return "disconnected" + if self._last_error: + return "error" + return "connected" diff --git a/app/fanout/base.py b/app/fanout/base.py new file mode 100644 index 00000000..f0af94c0 --- /dev/null +++ b/app/fanout/base.py @@ -0,0 +1,35 @@ +"""Base class for fanout integration modules.""" + +from __future__ import annotations + + +class FanoutModule: + """Base class for all fanout integrations. + + Each module wraps a specific integration (MQTT, webhook, etc.) and + receives dispatched messages/packets from the FanoutManager. + + Subclasses must override the ``status`` property. + """ + + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + self.config_id = config_id + self.config = config + self.name = name + + async def start(self) -> None: + """Start the module (e.g. connect to broker). Override for persistent connections.""" + + async def stop(self) -> None: + """Stop the module (e.g. disconnect from broker).""" + + async def on_message(self, data: dict) -> None: + """Called for decoded messages (DM/channel). Override if needed.""" + + async def on_raw(self, data: dict) -> None: + """Called for raw RF packets. Override if needed.""" + + @property + def status(self) -> str: + """Return 'connected', 'disconnected', or 'error'.""" + raise NotImplementedError diff --git a/app/fanout/bot.py b/app/fanout/bot.py new file mode 100644 index 00000000..40a1193f --- /dev/null +++ b/app/fanout/bot.py @@ -0,0 +1,142 @@ +"""Fanout module wrapping bot execution logic.""" + +from __future__ import annotations + +import asyncio +import logging + +from app.fanout.base import FanoutModule + +logger = logging.getLogger(__name__) + + +class BotModule(FanoutModule): + """Wraps a single bot's code execution and response routing. + + Each BotModule represents one bot configuration. It receives decoded + messages via ``on_message``, executes the bot's Python code in a + background task (after a 2-second settle delay), and sends any response + back through the radio. + """ + + def __init__(self, config_id: str, config: dict, *, name: str = "Bot") -> None: + super().__init__(config_id, config, name=name) + self._tasks: set[asyncio.Task] = set() + self._active = True + + async def stop(self) -> None: + self._active = False + for task in self._tasks: + task.cancel() + # Wait briefly for tasks to acknowledge cancellation + if self._tasks: + await asyncio.gather(*self._tasks, return_exceptions=True) + self._tasks.clear() + + async def on_message(self, data: dict) -> None: + """Kick off bot execution in a background task so we don't block dispatch.""" + task = asyncio.create_task(self._run_for_message(data)) + self._tasks.add(task) + task.add_done_callback(self._tasks.discard) + + async def _run_for_message(self, data: dict) -> None: + from app.fanout.bot_exec import ( + BOT_EXECUTION_TIMEOUT, + execute_bot_code, + process_bot_response, + ) + + code = self.config.get("code", "") + if not code or not code.strip(): + return + + msg_type = data.get("type", "") + is_dm = msg_type == "PRIV" + + # Extract bot parameters from broadcast data + if is_dm: + conversation_key = data.get("conversation_key", "") + sender_key = data.get("sender_key") or conversation_key + is_outgoing = data.get("outgoing", False) + message_text = data.get("text", "") + channel_key = None + channel_name = None + + # Outgoing DMs: sender is us, not the contact + if is_outgoing: + sender_name = None + else: + sender_name = data.get("sender_name") + if sender_name is None: + from app.repository import ContactRepository + + contact = await ContactRepository.get_by_key(conversation_key) + sender_name = contact.name if contact else None + else: + conversation_key = data.get("conversation_key", "") + sender_key = None + is_outgoing = bool(data.get("outgoing", False)) + sender_name = data.get("sender_name") + channel_key = conversation_key + + channel_name = data.get("channel_name") + if channel_name is None: + from app.repository import ChannelRepository + + channel = await ChannelRepository.get_by_key(conversation_key) + channel_name = channel.name if channel else None + + # Strip "sender: " prefix from channel message text + text = data.get("text", "") + if sender_name and text.startswith(f"{sender_name}: "): + message_text = text[len(f"{sender_name}: ") :] + else: + message_text = text + + sender_timestamp = data.get("sender_timestamp") + path_value = data.get("path") + # Message model serializes paths as list of dicts; extract first path string + if path_value is None: + paths = data.get("paths") + if paths and isinstance(paths, list) and len(paths) > 0: + path_value = paths[0].get("path") if isinstance(paths[0], dict) else None + + # Wait for message to settle (allows retransmissions to be deduped) + await asyncio.sleep(2) + + # Execute bot code in thread pool with timeout + from app.fanout.bot_exec import _bot_executor, _bot_semaphore + + async with _bot_semaphore: + loop = asyncio.get_running_loop() + try: + response = await asyncio.wait_for( + loop.run_in_executor( + _bot_executor, + execute_bot_code, + code, + sender_name, + sender_key, + message_text, + is_dm, + channel_key, + channel_name, + sender_timestamp, + path_value, + is_outgoing, + ), + timeout=BOT_EXECUTION_TIMEOUT, + ) + except asyncio.TimeoutError: + logger.warning("Bot '%s' execution timed out", self.name) + return + except Exception as e: + logger.warning("Bot '%s' execution error: %s", self.name, e) + return + + if response and self._active: + await process_bot_response(response, is_dm, sender_key or "", channel_key) + + @property + def status(self) -> str: + return "connected" diff --git a/app/bot.py b/app/fanout/bot_exec.py similarity index 72% rename from app/bot.py rename to app/fanout/bot_exec.py index a1fcbfea..e50e2c34 100644 --- a/app/bot.py +++ b/app/fanout/bot_exec.py @@ -19,8 +19,6 @@ from typing import Any from fastapi import HTTPException -from app.config import settings as server_settings - logger = logging.getLogger(__name__) # Limit concurrent bot executions to prevent resource exhaustion @@ -259,97 +257,3 @@ async def _send_single_bot_message( # Update last send time after successful send _last_bot_send_time = time.monotonic() - - -async def run_bot_for_message( - sender_name: str | None, - sender_key: str | None, - message_text: str, - is_dm: bool, - channel_key: str | None, - channel_name: str | None = None, - sender_timestamp: int | None = None, - path: str | None = None, - is_outgoing: bool = False, -) -> None: - """ - Run all enabled bots for a message (incoming or outgoing). - - This is the main entry point called by message handlers after - a message is successfully decrypted and stored. Bots run serially, - and errors in one bot don't prevent others from running. - - Args: - sender_name: Display name of the sender - sender_key: 64-char hex public key of sender (DMs only, None for channels) - message_text: The message content - is_dm: True for direct messages, False for channel messages - channel_key: Channel key for channel messages - channel_name: Channel name (e.g. "#general"), None for DMs - sender_timestamp: Sender's timestamp from the message - path: Hex-encoded routing path - is_outgoing: Whether this is our own outgoing message - """ - if server_settings.disable_bots: - return - - # Early check if any bots are enabled (will re-check after sleep) - from app.repository import AppSettingsRepository - - settings = await AppSettingsRepository.get() - enabled_bots = [b for b in settings.bots if b.enabled and b.code.strip()] - if not enabled_bots: - return - - async with _bot_semaphore: - logger.debug( - "Running %d bot(s) for message from %s (is_dm=%s)", - len(enabled_bots), - sender_name or (sender_key[:12] if sender_key else "unknown"), - is_dm, - ) - - # Wait for the initiating message's retransmissions to propagate through the mesh - await asyncio.sleep(2) - - # Re-check settings after sleep (user may have changed bot config) - settings = await AppSettingsRepository.get() - enabled_bots = [b for b in settings.bots if b.enabled and b.code.strip()] - if not enabled_bots: - logger.debug("All bots disabled during wait, skipping") - return - - # Run each enabled bot serially - loop = asyncio.get_event_loop() - for bot in enabled_bots: - logger.debug("Executing bot '%s'", bot.name) - try: - response = await asyncio.wait_for( - loop.run_in_executor( - _bot_executor, - execute_bot_code, - bot.code, - sender_name, - sender_key, - message_text, - is_dm, - channel_key, - channel_name, - sender_timestamp, - path, - is_outgoing, - ), - timeout=BOT_EXECUTION_TIMEOUT, - ) - except asyncio.TimeoutError: - logger.warning( - "Bot '%s' execution timed out after %ds", bot.name, BOT_EXECUTION_TIMEOUT - ) - continue # Continue to next bot - except Exception as e: - logger.warning("Bot '%s' execution error: %s", bot.name, e) - continue # Continue to next bot - - # Send response if any - if response: - await process_bot_response(response, is_dm, sender_key or "", channel_key) diff --git a/app/community_mqtt.py b/app/fanout/community_mqtt.py similarity index 86% rename from app/community_mqtt.py rename to app/fanout/community_mqtt.py index 72fe6a9b..22524bc5 100644 --- a/app/community_mqtt.py +++ b/app/fanout/community_mqtt.py @@ -15,17 +15,15 @@ import hashlib import importlib.metadata import json import logging -import re import ssl import time from datetime import datetime -from typing import Any +from typing import Any, Protocol import aiomqtt import nacl.bindings -from app.models import AppSettings -from app.mqtt_base import BaseMqttPublisher +from app.fanout.mqtt_base import BaseMqttPublisher logger = logging.getLogger(__name__) @@ -43,12 +41,21 @@ _STATS_MIN_CACHE_SECS = 60 # Don't re-fetch stats within 60s # Ed25519 group order _L = 2**252 + 27742317777372353535851937790883648493 -_IATA_RE = re.compile(r"^[A-Z]{3}$") # Route type mapping: bottom 2 bits of first byte _ROUTE_MAP = {0: "F", 1: "F", 2: "D", 3: "T"} +class CommunityMqttSettings(Protocol): + """Attributes expected on the settings object for the community MQTT publisher.""" + + community_mqtt_enabled: bool + community_mqtt_broker_host: str + community_mqtt_broker_port: int + community_mqtt_iata: str + community_mqtt_email: str + + def _base64url_encode(data: bytes) -> str: """Base64url encode without padding.""" return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii") @@ -258,7 +265,7 @@ def _format_raw_packet(data: dict[str, Any], device_name: str, public_key_hex: s return packet -def _build_status_topic(settings: AppSettings, pubkey_hex: str) -> str: +def _build_status_topic(settings: CommunityMqttSettings, pubkey_hex: str) -> str: """Build the ``meshcore/{IATA}/{PUBKEY}/status`` topic string.""" iata = settings.community_mqtt_iata.upper().strip() return f"meshcore/{iata}/{pubkey_hex}/status" @@ -310,7 +317,7 @@ class CommunityMqttPublisher(BaseMqttPublisher): self._last_stats_fetch: float = 0.0 self._last_status_publish: float = 0.0 - async def start(self, settings: AppSettings) -> None: + async def start(self, settings: object) -> None: self._key_unavailable_warned = False self._cached_device_info = None self._cached_stats = None @@ -323,12 +330,8 @@ class CommunityMqttPublisher(BaseMqttPublisher): from app.keystore import has_private_key from app.websocket import broadcast_error - if ( - self._settings - and self._settings.community_mqtt_enabled - and not has_private_key() - and not self._key_unavailable_warned - ): + s: CommunityMqttSettings | None = self._settings + if s and not has_private_key() and not self._key_unavailable_warned: broadcast_error( "Community MQTT unavailable", "Radio firmware does not support private key export.", @@ -339,9 +342,11 @@ class CommunityMqttPublisher(BaseMqttPublisher): """Check if community MQTT is enabled and keys are available.""" from app.keystore import has_private_key - return bool(self._settings and self._settings.community_mqtt_enabled and has_private_key()) + s: CommunityMqttSettings | None = self._settings + return bool(s and s.community_mqtt_enabled and has_private_key()) - def _build_client_kwargs(self, settings: AppSettings) -> dict[str, Any]: + def _build_client_kwargs(self, settings: object) -> dict[str, Any]: + s: CommunityMqttSettings = settings # type: ignore[assignment] from app.keystore import get_private_key, get_public_key from app.radio import radio_manager @@ -350,13 +355,13 @@ class CommunityMqttPublisher(BaseMqttPublisher): assert private_key is not None and public_key is not None # guaranteed by _pre_connect pubkey_hex = public_key.hex().upper() - broker_host = settings.community_mqtt_broker_host or _DEFAULT_BROKER - broker_port = settings.community_mqtt_broker_port or _DEFAULT_PORT + broker_host = s.community_mqtt_broker_host or _DEFAULT_BROKER + broker_port = s.community_mqtt_broker_port or _DEFAULT_PORT jwt_token = _generate_jwt_token( private_key, public_key, audience=broker_host, - email=settings.community_mqtt_email or "", + email=s.community_mqtt_email or "", ) tls_context = ssl.create_default_context() @@ -365,7 +370,7 @@ class CommunityMqttPublisher(BaseMqttPublisher): if radio_manager.meshcore and radio_manager.meshcore.self_info: device_name = radio_manager.meshcore.self_info.get("name", "") - status_topic = _build_status_topic(settings, pubkey_hex) + status_topic = _build_status_topic(s, pubkey_hex) offline_payload = json.dumps( { "status": "offline", @@ -386,9 +391,10 @@ class CommunityMqttPublisher(BaseMqttPublisher): "will": aiomqtt.Will(status_topic, offline_payload, retain=True), } - def _on_connected(self, settings: AppSettings) -> tuple[str, str]: - broker_host = settings.community_mqtt_broker_host or _DEFAULT_BROKER - broker_port = settings.community_mqtt_broker_port or _DEFAULT_PORT + def _on_connected(self, settings: object) -> tuple[str, str]: + s: CommunityMqttSettings = settings # type: ignore[assignment] + broker_host = s.community_mqtt_broker_host or _DEFAULT_BROKER + broker_port = s.community_mqtt_broker_port or _DEFAULT_PORT return ("Community MQTT connected", f"{broker_host}:{broker_port}") async def _fetch_device_info(self) -> dict[str, str]: @@ -479,7 +485,9 @@ class CommunityMqttPublisher(BaseMqttPublisher): return self._cached_stats - async def _publish_status(self, settings: AppSettings, *, refresh_stats: bool = True) -> None: + async def _publish_status( + self, settings: CommunityMqttSettings, *, refresh_stats: bool = True + ) -> None: """Build and publish the enriched retained status message.""" from app.keystore import get_public_key from app.radio import radio_manager @@ -514,9 +522,9 @@ class CommunityMqttPublisher(BaseMqttPublisher): await self.publish(status_topic, payload, retain=True) self._last_status_publish = time.monotonic() - async def _on_connected_async(self, settings: AppSettings) -> None: + async def _on_connected_async(self, settings: object) -> None: """Publish a retained online status message after connecting.""" - await self._publish_status(settings) + await self._publish_status(settings) # type: ignore[arg-type] async def _on_periodic_wake(self, elapsed: float) -> None: if not self._settings: @@ -540,7 +548,7 @@ class CommunityMqttPublisher(BaseMqttPublisher): return True return False - async def _pre_connect(self, settings: AppSettings) -> bool: + async def _pre_connect(self, settings: object) -> bool: from app.keystore import get_private_key, get_public_key private_key = get_private_key() @@ -555,50 +563,3 @@ class CommunityMqttPublisher(BaseMqttPublisher): pass return False return True - - -# Module-level singleton -community_publisher = CommunityMqttPublisher() - - -def community_mqtt_broadcast(event_type: str, data: dict[str, Any]) -> None: - """Fire-and-forget community MQTT publish for raw packets only.""" - if event_type != "raw_packet": - return - if not community_publisher.connected or community_publisher._settings is None: - return - asyncio.create_task(_community_maybe_publish(data)) - - -async def _community_maybe_publish(data: dict[str, Any]) -> None: - """Format and publish a raw packet to the community broker.""" - settings = community_publisher._settings - if settings is None or not settings.community_mqtt_enabled: - return - - try: - from app.keystore import get_public_key - from app.radio import radio_manager - - public_key = get_public_key() - if public_key is None: - return - - pubkey_hex = public_key.hex().upper() - - # Get device name from radio - device_name = "" - if radio_manager.meshcore and radio_manager.meshcore.self_info: - device_name = radio_manager.meshcore.self_info.get("name", "") - - packet = _format_raw_packet(data, device_name, pubkey_hex) - iata = settings.community_mqtt_iata.upper().strip() - if not _IATA_RE.fullmatch(iata): - logger.debug("Community MQTT: skipping publish — no valid IATA code configured") - return - topic = f"meshcore/{iata}/{pubkey_hex}/packets" - - await community_publisher.publish(topic, packet) - - except Exception as e: - logger.warning("Community MQTT broadcast error: %s", e) diff --git a/app/fanout/manager.py b/app/fanout/manager.py new file mode 100644 index 00000000..665179a3 --- /dev/null +++ b/app/fanout/manager.py @@ -0,0 +1,243 @@ +"""FanoutManager: owns all active fanout modules and dispatches events.""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any + +from app.fanout.base import FanoutModule + +logger = logging.getLogger(__name__) +_DISPATCH_TIMEOUT_SECONDS = 30.0 + +# Type string -> module class mapping +_MODULE_TYPES: dict[str, type] = {} + + +def _register_module_types() -> None: + """Lazily populate the type registry to avoid circular imports.""" + if _MODULE_TYPES: + return + from app.fanout.apprise_mod import AppriseModule + from app.fanout.bot import BotModule + from app.fanout.mqtt_community import MqttCommunityModule + from app.fanout.mqtt_private import MqttPrivateModule + from app.fanout.webhook import WebhookModule + + _MODULE_TYPES["mqtt_private"] = MqttPrivateModule + _MODULE_TYPES["mqtt_community"] = MqttCommunityModule + _MODULE_TYPES["bot"] = BotModule + _MODULE_TYPES["webhook"] = WebhookModule + _MODULE_TYPES["apprise"] = AppriseModule + + +def _matches_filter(filter_value: Any, key: str) -> bool: + """Check a single filter value (channels or contacts) against a key. + + Supported shapes: + "all" -> True + "none" -> False + ["key1", "key2"] -> key in list (only listed) + {"except": ["key1", "key2"]} -> key not in list (all except listed) + """ + if filter_value == "all": + return True + if filter_value == "none": + return False + if isinstance(filter_value, list): + return key in filter_value + if isinstance(filter_value, dict) and "except" in filter_value: + return key not in filter_value["except"] + return False + + +def _scope_matches_message(scope: dict, data: dict) -> bool: + """Check whether a message event matches the given scope.""" + messages = scope.get("messages", "none") + if messages == "all": + return True + if messages == "none": + return False + if isinstance(messages, dict): + msg_type = data.get("type", "") + conversation_key = data.get("conversation_key", "") + if msg_type == "CHAN": + return _matches_filter(messages.get("channels", "none"), conversation_key) + elif msg_type == "PRIV": + return _matches_filter(messages.get("contacts", "none"), conversation_key) + return False + + +def _scope_matches_raw(scope: dict, _data: dict) -> bool: + """Check whether a raw packet event matches the given scope.""" + return scope.get("raw_packets", "none") == "all" + + +class FanoutManager: + """Owns all active fanout modules and dispatches events.""" + + def __init__(self) -> None: + self._modules: dict[str, tuple[FanoutModule, dict]] = {} # id -> (module, scope) + self._restart_locks: dict[str, asyncio.Lock] = {} + + async def load_from_db(self) -> None: + """Read enabled fanout_configs and instantiate modules.""" + _register_module_types() + from app.repository.fanout import FanoutConfigRepository + + configs = await FanoutConfigRepository.get_enabled() + for cfg in configs: + await self._start_module(cfg) + + async def _start_module(self, cfg: dict[str, Any]) -> None: + """Instantiate and start a single module from a config dict.""" + config_id = cfg["id"] + config_type = cfg["type"] + config_blob = cfg["config"] + scope = cfg["scope"] + + # Skip bot modules when bots are disabled server-wide + if config_type == "bot": + from app.config import settings as server_settings + + if server_settings.disable_bots: + logger.info("Skipping bot module %s (bots disabled by server config)", config_id) + return + + cls = _MODULE_TYPES.get(config_type) + if cls is None: + logger.warning("Unknown fanout type %r for config %s, skipping", config_type, config_id) + return + + try: + module = cls(config_id, config_blob, name=cfg.get("name", "")) + await module.start() + self._modules[config_id] = (module, scope) + logger.info( + "Started fanout module %s (type=%s)", cfg.get("name", config_id), config_type + ) + except Exception: + logger.exception("Failed to start fanout module %s", config_id) + + async def reload_config(self, config_id: str) -> None: + """Stop old module (if any) and start updated config.""" + lock = self._restart_locks.setdefault(config_id, asyncio.Lock()) + async with lock: + await self.remove_config(config_id) + + from app.repository.fanout import FanoutConfigRepository + + cfg = await FanoutConfigRepository.get(config_id) + if cfg is None or not cfg["enabled"]: + return + await self._start_module(cfg) + + async def remove_config(self, config_id: str) -> None: + """Stop and remove a module.""" + entry = self._modules.pop(config_id, None) + if entry is not None: + module, _ = entry + try: + await module.stop() + except Exception: + logger.exception("Error stopping fanout module %s", config_id) + + async def _dispatch_matching( + self, + data: dict, + *, + matcher: Any, + handler_name: str, + log_label: str, + ) -> None: + """Dispatch to all matching modules concurrently.""" + tasks = [] + for config_id, (module, scope) in list(self._modules.items()): + if matcher(scope, data): + tasks.append(self._run_handler(config_id, module, handler_name, data, log_label)) + if tasks: + await asyncio.gather(*tasks) + + async def _run_handler( + self, + config_id: str, + module: FanoutModule, + handler_name: str, + data: dict, + log_label: str, + ) -> None: + """Run one module handler with per-module exception isolation.""" + try: + handler = getattr(module, handler_name) + await asyncio.wait_for(handler(data), timeout=_DISPATCH_TIMEOUT_SECONDS) + except asyncio.TimeoutError: + logger.error( + "Fanout %s %s timed out after %.1fs; restarting module", + config_id, + log_label, + _DISPATCH_TIMEOUT_SECONDS, + ) + await self._restart_module(config_id, module) + except Exception: + logger.exception("Fanout %s %s error", config_id, log_label) + + async def _restart_module(self, config_id: str, module: FanoutModule) -> None: + """Restart a timed-out module if it is still the active instance.""" + lock = self._restart_locks.setdefault(config_id, asyncio.Lock()) + async with lock: + entry = self._modules.get(config_id) + if entry is None or entry[0] is not module: + return + try: + await module.stop() + await module.start() + except Exception: + logger.exception("Failed to restart timed-out fanout module %s", config_id) + self._modules.pop(config_id, None) + + async def broadcast_message(self, data: dict) -> None: + """Dispatch a decoded message to modules whose scope matches.""" + await self._dispatch_matching( + data, + matcher=_scope_matches_message, + handler_name="on_message", + log_label="on_message", + ) + + async def broadcast_raw(self, data: dict) -> None: + """Dispatch a raw packet to modules whose scope matches.""" + await self._dispatch_matching( + data, + matcher=_scope_matches_raw, + handler_name="on_raw", + log_label="on_raw", + ) + + async def stop_all(self) -> None: + """Shutdown all modules.""" + for config_id, (module, _) in list(self._modules.items()): + try: + await module.stop() + except Exception: + logger.exception("Error stopping fanout module %s", config_id) + self._modules.clear() + self._restart_locks.clear() + + def get_statuses(self) -> dict[str, dict[str, str]]: + """Return status info for each active module.""" + from app.repository.fanout import _configs_cache + + result: dict[str, dict[str, str]] = {} + for config_id, (module, _) in self._modules.items(): + info = _configs_cache.get(config_id, {}) + result[config_id] = { + "name": info.get("name", config_id), + "type": info.get("type", "unknown"), + "status": module.status, + } + return result + + +# Module-level singleton +fanout_manager = FanoutManager() diff --git a/app/fanout/mqtt.py b/app/fanout/mqtt.py new file mode 100644 index 00000000..c2965a4a --- /dev/null +++ b/app/fanout/mqtt.py @@ -0,0 +1,91 @@ +"""MQTT publisher for forwarding mesh network events to an MQTT broker.""" + +from __future__ import annotations + +import logging +import ssl +from typing import Any, Protocol + +from app.fanout.mqtt_base import BaseMqttPublisher + +logger = logging.getLogger(__name__) + + +class PrivateMqttSettings(Protocol): + """Attributes expected on the settings object for the private MQTT publisher.""" + + mqtt_broker_host: str + mqtt_broker_port: int + mqtt_username: str + mqtt_password: str + mqtt_use_tls: bool + mqtt_tls_insecure: bool + mqtt_publish_messages: bool + mqtt_publish_raw_packets: bool + + +class MqttPublisher(BaseMqttPublisher): + """Manages an MQTT connection and publishes mesh network events.""" + + _backoff_max = 30 + _log_prefix = "MQTT" + + def _is_configured(self) -> bool: + """Check if MQTT is configured and has something to publish.""" + s: PrivateMqttSettings | None = self._settings + return bool( + s and s.mqtt_broker_host and (s.mqtt_publish_messages or s.mqtt_publish_raw_packets) + ) + + def _build_client_kwargs(self, settings: object) -> dict[str, Any]: + s: PrivateMqttSettings = settings # type: ignore[assignment] + return { + "hostname": s.mqtt_broker_host, + "port": s.mqtt_broker_port, + "username": s.mqtt_username or None, + "password": s.mqtt_password or None, + "tls_context": self._build_tls_context(s), + } + + def _on_connected(self, settings: object) -> tuple[str, str]: + s: PrivateMqttSettings = settings # type: ignore[assignment] + return ("MQTT connected", f"{s.mqtt_broker_host}:{s.mqtt_broker_port}") + + def _on_error(self) -> tuple[str, str]: + return ("MQTT connection failure", "Please correct the settings or disable.") + + @staticmethod + def _build_tls_context(settings: PrivateMqttSettings) -> ssl.SSLContext | None: + """Build TLS context from settings, or None if TLS is disabled.""" + if not settings.mqtt_use_tls: + return None + ctx = ssl.create_default_context() + if settings.mqtt_tls_insecure: + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +def _build_message_topic(prefix: str, data: dict[str, Any]) -> str: + """Build MQTT topic for a decrypted message.""" + msg_type = data.get("type", "") + conversation_key = data.get("conversation_key", "unknown") + + if msg_type == "PRIV": + return f"{prefix}/dm:{conversation_key}" + elif msg_type == "CHAN": + return f"{prefix}/gm:{conversation_key}" + return f"{prefix}/message:{conversation_key}" + + +def _build_raw_packet_topic(prefix: str, data: dict[str, Any]) -> str: + """Build MQTT topic for a raw packet.""" + info = data.get("decrypted_info") + if info and isinstance(info, dict): + contact_key = info.get("contact_key") + channel_key = info.get("channel_key") + if contact_key: + return f"{prefix}/raw/dm:{contact_key}" + if channel_key: + return f"{prefix}/raw/gm:{channel_key}" + return f"{prefix}/raw/unrouted" diff --git a/app/mqtt_base.py b/app/fanout/mqtt_base.py similarity index 92% rename from app/mqtt_base.py rename to app/fanout/mqtt_base.py index ead961fc..1427b5df 100644 --- a/app/mqtt_base.py +++ b/app/fanout/mqtt_base.py @@ -18,8 +18,6 @@ from typing import Any import aiomqtt -from app.models import AppSettings - logger = logging.getLogger(__name__) _BACKOFF_MIN = 5 @@ -38,6 +36,11 @@ class BaseMqttPublisher(ABC): Subclasses implement the abstract hooks to control configuration checks, client construction, toast messages, and optional wait-loop behavior. + + The settings type is duck-typed — each subclass defines a Protocol + describing the attributes it expects (e.g. ``PrivateMqttSettings``, + ``CommunityMqttSettings``). Callers pass ``SimpleNamespace`` instances + that satisfy the protocol. """ _backoff_max: int = 30 @@ -47,14 +50,14 @@ class BaseMqttPublisher(ABC): def __init__(self) -> None: self._client: aiomqtt.Client | None = None self._task: asyncio.Task[None] | None = None - self._settings: AppSettings | None = None + self._settings: Any = None self._settings_version: int = 0 self._version_event: asyncio.Event = asyncio.Event() self.connected: bool = False # ── Lifecycle ────────────────────────────────────────────────────── - async def start(self, settings: AppSettings) -> None: + async def start(self, settings: object) -> None: """Start the background connection loop.""" self._settings = settings self._settings_version += 1 @@ -74,7 +77,7 @@ class BaseMqttPublisher(ABC): self._client = None self.connected = False - async def restart(self, settings: AppSettings) -> None: + async def restart(self, settings: object) -> None: """Called when settings change — stop + start.""" await self.stop() await self.start(settings) @@ -99,11 +102,11 @@ class BaseMqttPublisher(ABC): """Return True when this publisher should attempt to connect.""" @abstractmethod - def _build_client_kwargs(self, settings: AppSettings) -> dict[str, Any]: + def _build_client_kwargs(self, settings: object) -> dict[str, Any]: """Return the keyword arguments for ``aiomqtt.Client(...)``.""" @abstractmethod - def _on_connected(self, settings: AppSettings) -> tuple[str, str]: + def _on_connected(self, settings: object) -> tuple[str, str]: """Return ``(title, detail)`` for the success toast on connect.""" @abstractmethod @@ -116,7 +119,7 @@ class BaseMqttPublisher(ABC): """Return True to break the inner wait (e.g. token expiry).""" return False - async def _pre_connect(self, settings: AppSettings) -> bool: + async def _pre_connect(self, settings: object) -> bool: """Called before connecting. Return True to proceed, False to retry.""" return True @@ -124,7 +127,7 @@ class BaseMqttPublisher(ABC): """Called each time the loop finds the publisher not configured.""" return # no-op by default; subclasses may override - async def _on_connected_async(self, settings: AppSettings) -> None: + async def _on_connected_async(self, settings: object) -> None: """Async hook called after connection succeeds (before health broadcast). Subclasses can override to publish messages immediately after connecting. diff --git a/app/fanout/mqtt_community.py b/app/fanout/mqtt_community.py new file mode 100644 index 00000000..982efa61 --- /dev/null +++ b/app/fanout/mqtt_community.py @@ -0,0 +1,89 @@ +"""Fanout module wrapping the community MQTT publisher.""" + +from __future__ import annotations + +import logging +import re +from types import SimpleNamespace +from typing import Any + +from app.fanout.base import FanoutModule +from app.fanout.community_mqtt import CommunityMqttPublisher, _format_raw_packet + +logger = logging.getLogger(__name__) + +_IATA_RE = re.compile(r"^[A-Z]{3}$") + + +def _config_to_settings(config: dict) -> SimpleNamespace: + """Map a fanout config blob to a settings namespace for the CommunityMqttPublisher.""" + return SimpleNamespace( + community_mqtt_enabled=True, + community_mqtt_broker_host=config.get("broker_host", "mqtt-us-v1.letsmesh.net"), + community_mqtt_broker_port=config.get("broker_port", 443), + community_mqtt_iata=config.get("iata", ""), + community_mqtt_email=config.get("email", ""), + ) + + +class MqttCommunityModule(FanoutModule): + """Wraps a CommunityMqttPublisher for community packet sharing.""" + + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + super().__init__(config_id, config, name=name) + self._publisher = CommunityMqttPublisher() + + async def start(self) -> None: + settings = _config_to_settings(self.config) + await self._publisher.start(settings) + + async def stop(self) -> None: + await self._publisher.stop() + + async def on_message(self, data: dict) -> None: + # Community MQTT only publishes raw packets, not decoded messages. + pass + + async def on_raw(self, data: dict) -> None: + if not self._publisher.connected or self._publisher._settings is None: + return + await _publish_community_packet(self._publisher, self.config, data) + + @property + def status(self) -> str: + if self._publisher._is_configured(): + return "connected" if self._publisher.connected else "disconnected" + return "disconnected" + + +async def _publish_community_packet( + publisher: CommunityMqttPublisher, + config: dict, + data: dict[str, Any], +) -> None: + """Format and publish a raw packet to the community broker.""" + try: + from app.keystore import get_public_key + from app.radio import radio_manager + + public_key = get_public_key() + if public_key is None: + return + + pubkey_hex = public_key.hex().upper() + + device_name = "" + if radio_manager.meshcore and radio_manager.meshcore.self_info: + device_name = radio_manager.meshcore.self_info.get("name", "") + + packet = _format_raw_packet(data, device_name, pubkey_hex) + iata = config.get("iata", "").upper().strip() + if not _IATA_RE.fullmatch(iata): + logger.debug("Community MQTT: skipping publish — no valid IATA code configured") + return + topic = f"meshcore/{iata}/{pubkey_hex}/packets" + + await publisher.publish(topic, packet) + + except Exception as e: + logger.warning("Community MQTT broadcast error: %s", e) diff --git a/app/fanout/mqtt_private.py b/app/fanout/mqtt_private.py new file mode 100644 index 00000000..2169589a --- /dev/null +++ b/app/fanout/mqtt_private.py @@ -0,0 +1,61 @@ +"""Fanout module wrapping the private MQTT publisher.""" + +from __future__ import annotations + +import logging +from types import SimpleNamespace + +from app.fanout.base import FanoutModule +from app.fanout.mqtt import MqttPublisher, _build_message_topic, _build_raw_packet_topic + +logger = logging.getLogger(__name__) + + +def _config_to_settings(config: dict) -> SimpleNamespace: + """Map a fanout config blob to a settings namespace for the MqttPublisher.""" + return SimpleNamespace( + mqtt_broker_host=config.get("broker_host", ""), + mqtt_broker_port=config.get("broker_port", 1883), + mqtt_username=config.get("username", ""), + mqtt_password=config.get("password", ""), + mqtt_use_tls=config.get("use_tls", False), + mqtt_tls_insecure=config.get("tls_insecure", False), + mqtt_topic_prefix=config.get("topic_prefix", "meshcore"), + mqtt_publish_messages=True, + mqtt_publish_raw_packets=True, + ) + + +class MqttPrivateModule(FanoutModule): + """Wraps an MqttPublisher instance for private MQTT forwarding.""" + + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + super().__init__(config_id, config, name=name) + self._publisher = MqttPublisher() + + async def start(self) -> None: + settings = _config_to_settings(self.config) + await self._publisher.start(settings) + + async def stop(self) -> None: + await self._publisher.stop() + + async def on_message(self, data: dict) -> None: + if not self._publisher.connected or self._publisher._settings is None: + return + prefix = self.config.get("topic_prefix", "meshcore") + topic = _build_message_topic(prefix, data) + await self._publisher.publish(topic, data) + + async def on_raw(self, data: dict) -> None: + if not self._publisher.connected or self._publisher._settings is None: + return + prefix = self.config.get("topic_prefix", "meshcore") + topic = _build_raw_packet_topic(prefix, data) + await self._publisher.publish(topic, data) + + @property + def status(self) -> str: + if not self.config.get("broker_host"): + return "disconnected" + return "connected" if self._publisher.connected else "disconnected" diff --git a/app/fanout/webhook.py b/app/fanout/webhook.py new file mode 100644 index 00000000..0ec9c28f --- /dev/null +++ b/app/fanout/webhook.py @@ -0,0 +1,84 @@ +"""Fanout module for webhook (HTTP POST) delivery.""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import logging + +import httpx + +from app.fanout.base import FanoutModule + +logger = logging.getLogger(__name__) + + +class WebhookModule(FanoutModule): + """Delivers message data to an HTTP endpoint via POST (or configurable method).""" + + def __init__(self, config_id: str, config: dict, *, name: str = "") -> None: + super().__init__(config_id, config, name=name) + self._client: httpx.AsyncClient | None = None + self._last_error: str | None = None + + async def start(self) -> None: + self._client = httpx.AsyncClient(timeout=httpx.Timeout(10.0)) + self._last_error = None + + async def stop(self) -> None: + if self._client: + await self._client.aclose() + self._client = None + + async def on_message(self, data: dict) -> None: + await self._send(data, event_type="message") + + async def _send(self, data: dict, *, event_type: str) -> None: + if not self._client: + return + + url = self.config.get("url", "") + if not url: + return + + method = self.config.get("method", "POST").upper() + extra_headers = self.config.get("headers", {}) + hmac_secret = self.config.get("hmac_secret", "") + hmac_header = self.config.get("hmac_header", "X-Webhook-Signature") + + headers = { + "Content-Type": "application/json", + "X-Webhook-Event": event_type, + **extra_headers, + } + + body_bytes = json.dumps(data, separators=(",", ":"), sort_keys=True).encode() + + if hmac_secret: + sig = hmac.new(hmac_secret.encode(), body_bytes, hashlib.sha256).hexdigest() + headers[hmac_header or "X-Webhook-Signature"] = f"sha256={sig}" + + try: + resp = await self._client.request(method, url, content=body_bytes, headers=headers) + resp.raise_for_status() + self._last_error = None + except httpx.HTTPStatusError as exc: + self._last_error = f"HTTP {exc.response.status_code}" + logger.warning( + "Webhook %s returned %s for %s", + self.config_id, + exc.response.status_code, + url, + ) + except httpx.RequestError as exc: + self._last_error = str(exc) + logger.warning("Webhook %s request error: %s", self.config_id, exc) + + @property + def status(self) -> str: + if not self.config.get("url"): + return "disconnected" + if self._last_error: + return "error" + return "connected" diff --git a/app/main.py b/app/main.py index 4914e4a2..b1f1e6f6 100644 --- a/app/main.py +++ b/app/main.py @@ -18,6 +18,7 @@ from app.radio_sync import ( from app.routers import ( channels, contacts, + fanout, health, messages, packets, @@ -56,23 +57,18 @@ async def lifespan(app: FastAPI): # Always start connection monitor (even if initial connection failed) await radio_manager.start_connection_monitor() - # Start MQTT publishers if configured - from app.community_mqtt import community_publisher - from app.mqtt import mqtt_publisher - from app.repository import AppSettingsRepository + # Start fanout modules (MQTT, etc.) from database configs + from app.fanout.manager import fanout_manager try: - mqtt_settings = await AppSettingsRepository.get() - await mqtt_publisher.start(mqtt_settings) - await community_publisher.start(mqtt_settings) + await fanout_manager.load_from_db() except Exception as e: - logger.warning("Failed to start MQTT publisher(s): %s", e) + logger.warning("Failed to start fanout modules: %s", e) yield logger.info("Shutting down") - await community_publisher.stop() - await mqtt_publisher.stop() + await fanout_manager.stop_all() await radio_manager.stop_connection_monitor() await stop_message_polling() await stop_periodic_advert() @@ -119,6 +115,7 @@ async def radio_disconnected_handler(request: Request, exc: RadioDisconnectedErr # API routes - all prefixed with /api for production compatibility app.include_router(health.router, prefix="/api") +app.include_router(fanout.router, prefix="/api") app.include_router(radio.router, prefix="/api") app.include_router(contacts.router, prefix="/api") app.include_router(repeaters.router, prefix="/api") diff --git a/app/migrations.py b/app/migrations.py index 2546092d..bc5d869d 100644 --- a/app/migrations.py +++ b/app/migrations.py @@ -282,6 +282,27 @@ async def run_migrations(conn: aiosqlite.Connection) -> int: await set_version(conn, 35) applied += 1 + # Migration 36: Create fanout_configs table and migrate existing MQTT settings + if version < 36: + logger.info("Applying migration 36: create fanout_configs and migrate MQTT settings") + await _migrate_036_create_fanout_configs(conn) + await set_version(conn, 36) + applied += 1 + + # Migration 37: Migrate bots from app_settings to fanout_configs + if version < 37: + logger.info("Applying migration 37: migrate bots to fanout_configs") + await _migrate_037_bots_to_fanout(conn) + await set_version(conn, 37) + applied += 1 + + # Migration 38: Drop legacy MQTT, community MQTT, and bots columns from app_settings + if version < 38: + logger.info("Applying migration 38: drop legacy MQTT/bot columns from app_settings") + await _migrate_038_drop_legacy_columns(conn) + await set_version(conn, 38) + applied += 1 + if applied > 0: logger.info( "Applied %d migration(s), schema now at version %d", applied, await get_version(conn) @@ -2014,3 +2035,248 @@ async def _migrate_035_add_block_lists(conn: aiosqlite.Connection) -> None: raise await conn.commit() + + +async def _migrate_036_create_fanout_configs(conn: aiosqlite.Connection) -> None: + """Create fanout_configs table and migrate existing MQTT settings. + + Reads existing MQTT settings from app_settings and creates corresponding + fanout_configs rows. Old columns are NOT dropped (rollback safety). + """ + import json + import uuid + + # 1. Create fanout_configs table + await conn.execute( + """ + CREATE TABLE IF NOT EXISTS fanout_configs ( + id TEXT PRIMARY KEY, + type TEXT NOT NULL, + name TEXT NOT NULL, + enabled INTEGER DEFAULT 0, + config TEXT NOT NULL DEFAULT '{}', + scope TEXT NOT NULL DEFAULT '{}', + sort_order INTEGER DEFAULT 0, + created_at INTEGER NOT NULL + ) + """ + ) + + # 2. Read existing MQTT settings + try: + cursor = await conn.execute( + """ + SELECT mqtt_broker_host, mqtt_broker_port, mqtt_username, mqtt_password, + mqtt_use_tls, mqtt_tls_insecure, mqtt_topic_prefix, + mqtt_publish_messages, mqtt_publish_raw_packets, + community_mqtt_enabled, community_mqtt_iata, + community_mqtt_broker_host, community_mqtt_broker_port, + community_mqtt_email + FROM app_settings WHERE id = 1 + """ + ) + row = await cursor.fetchone() + except Exception: + row = None + + if row is None: + await conn.commit() + return + + import time + + now = int(time.time()) + sort_order = 0 + + # 3. Migrate private MQTT if configured + broker_host = row["mqtt_broker_host"] or "" + if broker_host: + publish_messages = bool(row["mqtt_publish_messages"]) + publish_raw = bool(row["mqtt_publish_raw_packets"]) + enabled = publish_messages or publish_raw + + config = { + "broker_host": broker_host, + "broker_port": row["mqtt_broker_port"] or 1883, + "username": row["mqtt_username"] or "", + "password": row["mqtt_password"] or "", + "use_tls": bool(row["mqtt_use_tls"]), + "tls_insecure": bool(row["mqtt_tls_insecure"]), + "topic_prefix": row["mqtt_topic_prefix"] or "meshcore", + } + + scope = { + "messages": "all" if publish_messages else "none", + "raw_packets": "all" if publish_raw else "none", + } + + await conn.execute( + """ + INSERT INTO fanout_configs (id, type, name, enabled, config, scope, sort_order, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + str(uuid.uuid4()), + "mqtt_private", + "Private MQTT", + 1 if enabled else 0, + json.dumps(config), + json.dumps(scope), + sort_order, + now, + ), + ) + sort_order += 1 + logger.info("Migrated private MQTT settings to fanout_configs (enabled=%s)", enabled) + + # 4. Migrate community MQTT if enabled OR configured (preserve disabled-but-configured) + community_enabled = bool(row["community_mqtt_enabled"]) + community_iata = row["community_mqtt_iata"] or "" + community_host = row["community_mqtt_broker_host"] or "" + community_email = row["community_mqtt_email"] or "" + community_has_config = bool( + community_iata + or community_email + or (community_host and community_host != "mqtt-us-v1.letsmesh.net") + ) + if community_enabled or community_has_config: + config = { + "broker_host": community_host or "mqtt-us-v1.letsmesh.net", + "broker_port": row["community_mqtt_broker_port"] or 443, + "iata": community_iata, + "email": community_email, + } + + scope = { + "messages": "none", + "raw_packets": "all", + } + + await conn.execute( + """ + INSERT INTO fanout_configs (id, type, name, enabled, config, scope, sort_order, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + str(uuid.uuid4()), + "mqtt_community", + "Community MQTT", + 1 if community_enabled else 0, + json.dumps(config), + json.dumps(scope), + sort_order, + now, + ), + ) + logger.info( + "Migrated community MQTT settings to fanout_configs (enabled=%s)", community_enabled + ) + + await conn.commit() + + +async def _migrate_037_bots_to_fanout(conn: aiosqlite.Connection) -> None: + """Migrate bots from app_settings.bots JSON to fanout_configs rows.""" + import json + import uuid + + try: + cursor = await conn.execute("SELECT bots FROM app_settings WHERE id = 1") + row = await cursor.fetchone() + except Exception: + row = None + + if row is None: + await conn.commit() + return + + bots_json = row["bots"] or "[]" + try: + bots = json.loads(bots_json) + except (json.JSONDecodeError, TypeError): + bots = [] + + if not bots: + await conn.commit() + return + + import time + + now = int(time.time()) + + # Use sort_order starting at 200 to place bots after MQTT configs (0-99) + for i, bot in enumerate(bots): + bot_name = bot.get("name") or f"Bot {i + 1}" + bot_enabled = bool(bot.get("enabled", False)) + bot_code = bot.get("code", "") + + config_blob = json.dumps({"code": bot_code}) + scope = json.dumps({"messages": "all", "raw_packets": "none"}) + + await conn.execute( + """ + INSERT INTO fanout_configs (id, type, name, enabled, config, scope, sort_order, created_at) + VALUES (?, 'bot', ?, ?, ?, ?, ?, ?) + """, + ( + str(uuid.uuid4()), + bot_name, + 1 if bot_enabled else 0, + config_blob, + scope, + 200 + i, + now, + ), + ) + logger.info("Migrated bot '%s' to fanout_configs (enabled=%s)", bot_name, bot_enabled) + + await conn.commit() + + +async def _migrate_038_drop_legacy_columns(conn: aiosqlite.Connection) -> None: + """Drop legacy MQTT, community MQTT, and bots columns from app_settings. + + These columns were migrated to fanout_configs in migrations 36 and 37. + SQLite 3.35.0+ supports ALTER TABLE DROP COLUMN. For older versions, + the columns remain but are harmless (no longer read or written). + """ + # Check if app_settings table exists (some test DBs may not have it) + cursor = await conn.execute( + "SELECT name FROM sqlite_master WHERE type='table' AND name='app_settings'" + ) + if await cursor.fetchone() is None: + await conn.commit() + return + + columns_to_drop = [ + "bots", + "mqtt_broker_host", + "mqtt_broker_port", + "mqtt_username", + "mqtt_password", + "mqtt_use_tls", + "mqtt_tls_insecure", + "mqtt_topic_prefix", + "mqtt_publish_messages", + "mqtt_publish_raw_packets", + "community_mqtt_enabled", + "community_mqtt_iata", + "community_mqtt_broker_host", + "community_mqtt_broker_port", + "community_mqtt_email", + ] + + for column in columns_to_drop: + try: + await conn.execute(f"ALTER TABLE app_settings DROP COLUMN {column}") + logger.debug("Dropped %s from app_settings", column) + except aiosqlite.OperationalError as e: + error_msg = str(e).lower() + if "no such column" in error_msg: + logger.debug("app_settings.%s already dropped, skipping", column) + elif "syntax error" in error_msg or "drop column" in error_msg: + logger.debug("SQLite doesn't support DROP COLUMN, %s column will remain", column) + else: + raise + + await conn.commit() diff --git a/app/models.py b/app/models.py index f2e1755f..2fb62a87 100644 --- a/app/models.py +++ b/app/models.py @@ -196,6 +196,7 @@ class Message(BaseModel): outgoing: bool = False acked: int = 0 sender_name: str | None = None + channel_name: str | None = None class MessagesAroundResponse(BaseModel): @@ -399,15 +400,6 @@ class Favorite(BaseModel): id: str = Field(description="Channel key or contact public key") -class BotConfig(BaseModel): - """Configuration for a single bot.""" - - id: str = Field(description="UUID for stable identity across renames/reorders") - name: str = Field(description="User-editable name") - enabled: bool = Field(default=False, description="Whether this bot is enabled") - code: str = Field(default="", description="Python code for this bot") - - class UnreadCounts(BaseModel): """Aggregated unread counts, mention flags, and last message times for all conversations.""" @@ -459,66 +451,6 @@ class AppSettings(BaseModel): default=0, description="Unix timestamp of last advertisement sent (0 = never)", ) - bots: list[BotConfig] = Field( - default_factory=list, - description="List of bot configurations", - ) - mqtt_broker_host: str = Field( - default="", - description="MQTT broker hostname (empty = disabled)", - ) - mqtt_broker_port: int = Field( - default=1883, - description="MQTT broker port", - ) - mqtt_username: str = Field( - default="", - description="MQTT username (optional)", - ) - mqtt_password: str = Field( - default="", - description="MQTT password (optional)", - ) - mqtt_use_tls: bool = Field( - default=False, - description="Whether to use TLS for MQTT connection", - ) - mqtt_tls_insecure: bool = Field( - default=False, - description="Skip TLS certificate verification (for self-signed certs)", - ) - mqtt_topic_prefix: str = Field( - default="meshcore", - description="MQTT topic prefix", - ) - mqtt_publish_messages: bool = Field( - default=False, - description="Whether to publish decrypted messages to MQTT", - ) - mqtt_publish_raw_packets: bool = Field( - default=False, - description="Whether to publish raw packets to MQTT", - ) - community_mqtt_enabled: bool = Field( - default=False, - description="Whether to publish raw packets to the community MQTT broker (letsmesh.net)", - ) - community_mqtt_iata: str = Field( - default="", - description="IATA region code for community MQTT topic routing (3 alpha chars)", - ) - community_mqtt_broker_host: str = Field( - default="mqtt-us-v1.letsmesh.net", - description="Community MQTT broker hostname", - ) - community_mqtt_broker_port: int = Field( - default=443, - description="Community MQTT broker port", - ) - community_mqtt_email: str = Field( - default="", - description="Email address for node claiming on the community aggregator (optional)", - ) flood_scope: str = Field( default="", description="Outbound flood scope / region name (empty = disabled, no tagging)", @@ -533,6 +465,19 @@ class AppSettings(BaseModel): ) +class FanoutConfig(BaseModel): + """Configuration for a single fanout integration.""" + + id: str + type: str # 'mqtt_private' | 'mqtt_community' | 'bot' | 'webhook' | 'apprise' + name: str + enabled: bool + config: dict + scope: dict + sort_order: int = 0 + created_at: int = 0 + + class BusyChannel(BaseModel): channel_key: str channel_name: str diff --git a/app/mqtt.py b/app/mqtt.py deleted file mode 100644 index 79a6e3f2..00000000 --- a/app/mqtt.py +++ /dev/null @@ -1,111 +0,0 @@ -"""MQTT publisher for forwarding mesh network events to an MQTT broker.""" - -from __future__ import annotations - -import asyncio -import logging -import ssl -from typing import Any - -from app.models import AppSettings -from app.mqtt_base import BaseMqttPublisher - -logger = logging.getLogger(__name__) - - -class MqttPublisher(BaseMqttPublisher): - """Manages an MQTT connection and publishes mesh network events.""" - - _backoff_max = 30 - _log_prefix = "MQTT" - - def _is_configured(self) -> bool: - """Check if MQTT is configured and has something to publish.""" - return bool( - self._settings - and self._settings.mqtt_broker_host - and (self._settings.mqtt_publish_messages or self._settings.mqtt_publish_raw_packets) - ) - - def _build_client_kwargs(self, settings: AppSettings) -> dict[str, Any]: - return { - "hostname": settings.mqtt_broker_host, - "port": settings.mqtt_broker_port, - "username": settings.mqtt_username or None, - "password": settings.mqtt_password or None, - "tls_context": self._build_tls_context(settings), - } - - def _on_connected(self, settings: AppSettings) -> tuple[str, str]: - return ("MQTT connected", f"{settings.mqtt_broker_host}:{settings.mqtt_broker_port}") - - def _on_error(self) -> tuple[str, str]: - return ("MQTT connection failure", "Please correct the settings or disable.") - - @staticmethod - def _build_tls_context(settings: AppSettings) -> ssl.SSLContext | None: - """Build TLS context from settings, or None if TLS is disabled.""" - if not settings.mqtt_use_tls: - return None - ctx = ssl.create_default_context() - if settings.mqtt_tls_insecure: - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - return ctx - - -# Module-level singleton -mqtt_publisher = MqttPublisher() - - -def mqtt_broadcast(event_type: str, data: dict[str, Any]) -> None: - """Fire-and-forget MQTT publish, matching broadcast_event's pattern.""" - if event_type not in ("message", "raw_packet"): - return - if not mqtt_publisher.connected or mqtt_publisher._settings is None: - return - asyncio.create_task(_mqtt_maybe_publish(event_type, data)) - - -async def _mqtt_maybe_publish(event_type: str, data: dict[str, Any]) -> None: - """Check settings and build topic, then publish.""" - settings = mqtt_publisher._settings - if settings is None: - return - - try: - if event_type == "message" and settings.mqtt_publish_messages: - topic = _build_message_topic(settings.mqtt_topic_prefix, data) - await mqtt_publisher.publish(topic, data) - - elif event_type == "raw_packet" and settings.mqtt_publish_raw_packets: - topic = _build_raw_packet_topic(settings.mqtt_topic_prefix, data) - await mqtt_publisher.publish(topic, data) - - except Exception as e: - logger.warning("MQTT broadcast error: %s", e) - - -def _build_message_topic(prefix: str, data: dict[str, Any]) -> str: - """Build MQTT topic for a decrypted message.""" - msg_type = data.get("type", "") - conversation_key = data.get("conversation_key", "unknown") - - if msg_type == "PRIV": - return f"{prefix}/dm:{conversation_key}" - elif msg_type == "CHAN": - return f"{prefix}/gm:{conversation_key}" - return f"{prefix}/message:{conversation_key}" - - -def _build_raw_packet_topic(prefix: str, data: dict[str, Any]) -> str: - """Build MQTT topic for a raw packet.""" - info = data.get("decrypted_info") - if info and isinstance(info, dict): - contact_key = info.get("contact_key") - channel_key = info.get("channel_key") - if contact_key: - return f"{prefix}/raw/dm:{contact_key}" - if channel_key: - return f"{prefix}/raw/gm:{channel_key}" - return f"{prefix}/raw/unrouted" diff --git a/app/packet_processor.py b/app/packet_processor.py index 3e40a796..649bdbe8 100644 --- a/app/packet_processor.py +++ b/app/packet_processor.py @@ -129,7 +129,7 @@ async def create_message_from_decrypted( received_at: int | None = None, path: str | None = None, channel_name: str | None = None, - trigger_bot: bool = True, + realtime: bool = True, ) -> int | None: """Create a message record from decrypted channel packet content. @@ -145,7 +145,7 @@ async def create_message_from_decrypted( timestamp: Sender timestamp from the packet received_at: When the packet was received (defaults to now) path: Hex-encoded routing path - trigger_bot: Whether to trigger bot response (False for historical decryption) + realtime: If False, skip fanout dispatch (used for historical decryption) Returns the message ID if created, None if duplicate. """ @@ -195,7 +195,7 @@ async def create_message_from_decrypted( # Use "is not None" to include empty string (direct/0-hop messages) paths = [MessagePath(path=path or "", received_at=received)] if path is not None else None - # Broadcast new message to connected clients + # Broadcast new message to connected clients (and fanout modules when realtime) broadcast_event( "message", Message( @@ -208,27 +208,11 @@ async def create_message_from_decrypted( paths=paths, sender_name=sender, sender_key=resolved_sender_key, + channel_name=channel_name, ).model_dump(), + realtime=realtime, ) - # Run bot if enabled (for incoming channel messages, not historical decryption) - if trigger_bot: - from app.bot import run_bot_for_message - - asyncio.create_task( - run_bot_for_message( - sender_name=sender, - sender_key=None, # Channel messages don't have a sender public key - message_text=message_text, - is_dm=False, - channel_key=channel_key_normalized, - channel_name=channel_name, - sender_timestamp=timestamp, - path=path, - is_outgoing=False, - ) - ) - return msg_id @@ -240,7 +224,7 @@ async def create_dm_message_from_decrypted( received_at: int | None = None, path: str | None = None, outgoing: bool = False, - trigger_bot: bool = True, + realtime: bool = True, ) -> int | None: """Create a message record from decrypted direct message packet content. @@ -255,7 +239,7 @@ async def create_dm_message_from_decrypted( received_at: When the packet was received (defaults to now) path: Hex-encoded routing path outgoing: Whether this is an outgoing message (we sent it) - trigger_bot: Whether to trigger bot response (False for historical decryption) + realtime: If False, skip fanout dispatch (used for historical decryption) Returns the message ID if created, None if duplicate. """ @@ -317,7 +301,8 @@ async def create_dm_message_from_decrypted( # Build paths array for broadcast paths = [MessagePath(path=path or "", received_at=received)] if path is not None else None - # Broadcast new message to connected clients + # Broadcast new message to connected clients (and fanout modules when realtime) + sender_name = contact.name if contact and not outgoing else None broadcast_event( "message", Message( @@ -332,29 +317,12 @@ async def create_dm_message_from_decrypted( sender_name=sender_name, sender_key=conversation_key if not outgoing else None, ).model_dump(), + realtime=realtime, ) # Update contact's last_contacted timestamp (for sorting) await ContactRepository.update_last_contacted(conversation_key, received) - # Run bot if enabled (for all real-time DMs, including our own outgoing messages) - if trigger_bot: - from app.bot import run_bot_for_message - - asyncio.create_task( - run_bot_for_message( - sender_name=contact.name if contact else None, - sender_key=their_public_key, - message_text=decrypted.message, - is_dm=True, - channel_key=None, - channel_name=None, - sender_timestamp=decrypted.timestamp, - path=path, - is_outgoing=outgoing, - ) - ) - return msg_id @@ -424,7 +392,7 @@ async def run_historical_dm_decryption( received_at=packet_timestamp, path=path_hex, outgoing=outgoing, - trigger_bot=False, # Historical decryption should not trigger bot + realtime=False, # Historical decryption should not trigger fanout ) if msg_id is not None: @@ -903,7 +871,7 @@ async def _process_direct_message( their_public_key=contact.public_key, our_public_key=our_public_key.hex(), received_at=timestamp, - path=packet_info.path.hex() if packet_info.path else None, + path=packet_info.path.hex() if packet_info else None, outgoing=is_outgoing, ) diff --git a/app/radio.py b/app/radio.py index 078dd1e8..95d5dd0a 100644 --- a/app/radio.py +++ b/app/radio.py @@ -470,6 +470,8 @@ class RadioManager: from app.websocket import broadcast_health CHECK_INTERVAL_SECONDS = 5 + UNRESPONSIVE_THRESHOLD = 3 + consecutive_setup_failures = 0 while True: try: @@ -483,6 +485,7 @@ class RadioManager: logger.warning("Radio connection lost, broadcasting status change") broadcast_health(False, self._connection_info) self._last_connected = False + consecutive_setup_failures = 0 if not current_connected: # Attempt reconnection on every loop while disconnected @@ -492,6 +495,7 @@ class RadioManager: await self.post_connect_setup() broadcast_health(True, self._connection_info) self._last_connected = True + consecutive_setup_failures = 0 elif not self._last_connected and current_connected: # Connection restored (might have reconnected automatically). @@ -500,19 +504,34 @@ class RadioManager: await self.post_connect_setup() broadcast_health(True, self._connection_info) self._last_connected = True + consecutive_setup_failures = 0 elif current_connected and not self._setup_complete: # Transport connected but setup incomplete — retry logger.info("Retrying post-connect setup...") await self.post_connect_setup() broadcast_health(True, self._connection_info) + consecutive_setup_failures = 0 except asyncio.CancelledError: # Task is being cancelled, exit cleanly break except Exception as e: - # Log error but continue monitoring - don't let the monitor die - logger.exception("Error in connection monitor, continuing: %s", e) + consecutive_setup_failures += 1 + if consecutive_setup_failures == UNRESPONSIVE_THRESHOLD: + logger.error( + "Post-connect setup has failed %d times in a row. " + "The radio port appears open but the radio is not " + "responding to commands. Common causes: another " + "process has the serial port open (check for other " + "RemoteTerm instances, serial monitors, etc.), the " + "firmware is in repeater mode (not client), or the " + "radio needs a power cycle. Will keep retrying.", + consecutive_setup_failures, + ) + elif consecutive_setup_failures < UNRESPONSIVE_THRESHOLD: + logger.exception("Error in connection monitor, continuing: %s", e) + # After the threshold, silently retry (avoid log spam) self._reconnect_task = asyncio.create_task(monitor_loop()) logger.info("Radio connection monitor started") diff --git a/app/radio_sync.py b/app/radio_sync.py index e4976aa5..9e77c9a2 100644 --- a/app/radio_sync.py +++ b/app/radio_sync.py @@ -117,7 +117,16 @@ async def sync_and_offload_contacts(mc: MeshCore) -> dict: result = await mc.commands.get_contacts() if result is None or result.type == EventType.ERROR: - logger.error("Failed to get contacts from radio: %s", result) + logger.error( + "Failed to get contacts from radio: %s. " + "If you see this repeatedly, the radio may be visible on the " + "serial/TCP/BLE port but not responding to commands. Check for " + "another process with the serial port open (other RemoteTerm " + "instances, serial monitors, etc.), verify the firmware is " + "up-to-date and in client mode (not repeater), or try a " + "power cycle.", + result, + ) return {"synced": 0, "removed": 0, "error": str(result)} contacts = result.payload or {} @@ -316,7 +325,7 @@ async def drain_pending_messages(mc: MeshCore) -> int: except asyncio.TimeoutError: break except Exception as e: - logger.debug("Error draining messages: %s", e) + logger.warning("Error draining messages: %s", e, exc_info=True) break return count @@ -350,7 +359,7 @@ async def poll_for_messages(mc: MeshCore) -> int: except asyncio.TimeoutError: pass except Exception as e: - logger.debug("Message poll exception: %s", e) + logger.warning("Message poll exception: %s", e, exc_info=True) return count @@ -384,7 +393,7 @@ async def _message_poll_loop(): except asyncio.CancelledError: break except Exception as e: - logger.debug("Error in message poll loop: %s", e) + logger.warning("Error in message poll loop: %s", e, exc_info=True) def start_message_polling(): @@ -662,8 +671,19 @@ async def _sync_contacts_to_radio_inner(mc: MeshCore) -> dict: logger.debug("Loaded contact %s to radio", contact.public_key[:12]) else: failed += 1 + reason = result.payload + hint = "" + if reason is None: + hint = ( + " (no response from radio — if this repeats, check for " + "serial port contention from another process or try a " + "power cycle)" + ) logger.warning( - "Failed to load contact %s: %s", contact.public_key[:12], result.payload + "Failed to load contact %s: %s%s", + contact.public_key[:12], + reason, + hint, ) except Exception as e: failed += 1 diff --git a/app/repository/__init__.py b/app/repository/__init__.py index 00589561..cb34f9c4 100644 --- a/app/repository/__init__.py +++ b/app/repository/__init__.py @@ -5,6 +5,7 @@ from app.repository.contacts import ( ContactNameHistoryRepository, ContactRepository, ) +from app.repository.fanout import FanoutConfigRepository from app.repository.messages import MessageRepository from app.repository.raw_packets import RawPacketRepository from app.repository.settings import AppSettingsRepository, StatisticsRepository @@ -16,6 +17,7 @@ __all__ = [ "ContactAdvertPathRepository", "ContactNameHistoryRepository", "ContactRepository", + "FanoutConfigRepository", "MessageRepository", "RawPacketRepository", "StatisticsRepository", diff --git a/app/repository/fanout.py b/app/repository/fanout.py new file mode 100644 index 00000000..76fb31d7 --- /dev/null +++ b/app/repository/fanout.py @@ -0,0 +1,137 @@ +"""Repository for fanout_configs table.""" + +import json +import logging +import time +import uuid +from typing import Any + +from app.database import db + +logger = logging.getLogger(__name__) + +# In-memory cache of config metadata (name, type) for status reporting. +# Populated by get_all/get/create/update and read by FanoutManager.get_statuses(). +_configs_cache: dict[str, dict[str, Any]] = {} + + +def _row_to_dict(row: Any) -> dict[str, Any]: + """Convert a database row to a config dict.""" + result = { + "id": row["id"], + "type": row["type"], + "name": row["name"], + "enabled": bool(row["enabled"]), + "config": json.loads(row["config"]) if row["config"] else {}, + "scope": json.loads(row["scope"]) if row["scope"] else {}, + "sort_order": row["sort_order"] or 0, + "created_at": row["created_at"] or 0, + } + _configs_cache[result["id"]] = result + return result + + +class FanoutConfigRepository: + """CRUD operations for fanout_configs table.""" + + @staticmethod + async def get_all() -> list[dict[str, Any]]: + """Get all fanout configs ordered by sort_order.""" + cursor = await db.conn.execute( + "SELECT * FROM fanout_configs ORDER BY sort_order, created_at" + ) + rows = await cursor.fetchall() + return [_row_to_dict(row) for row in rows] + + @staticmethod + async def get(config_id: str) -> dict[str, Any] | None: + """Get a single fanout config by ID.""" + cursor = await db.conn.execute("SELECT * FROM fanout_configs WHERE id = ?", (config_id,)) + row = await cursor.fetchone() + if row is None: + return None + return _row_to_dict(row) + + @staticmethod + async def create( + config_type: str, + name: str, + config: dict, + scope: dict, + enabled: bool = True, + config_id: str | None = None, + ) -> dict[str, Any]: + """Create a new fanout config.""" + new_id = config_id or str(uuid.uuid4()) + now = int(time.time()) + + # Get next sort_order + cursor = await db.conn.execute( + "SELECT COALESCE(MAX(sort_order), -1) + 1 FROM fanout_configs" + ) + row = await cursor.fetchone() + sort_order = row[0] if row else 0 + + await db.conn.execute( + """ + INSERT INTO fanout_configs (id, type, name, enabled, config, scope, sort_order, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + new_id, + config_type, + name, + 1 if enabled else 0, + json.dumps(config), + json.dumps(scope), + sort_order, + now, + ), + ) + await db.conn.commit() + + result = await FanoutConfigRepository.get(new_id) + assert result is not None + return result + + @staticmethod + async def update(config_id: str, **fields: Any) -> dict[str, Any] | None: + """Update a fanout config. Only provided fields are updated.""" + updates = [] + params: list[Any] = [] + + for field in ("name", "enabled", "config", "scope", "sort_order"): + if field in fields: + value = fields[field] + if field == "enabled": + value = 1 if value else 0 + elif field in ("config", "scope"): + value = json.dumps(value) + updates.append(f"{field} = ?") + params.append(value) + + if not updates: + return await FanoutConfigRepository.get(config_id) + + params.append(config_id) + query = f"UPDATE fanout_configs SET {', '.join(updates)} WHERE id = ?" + await db.conn.execute(query, params) + await db.conn.commit() + + return await FanoutConfigRepository.get(config_id) + + @staticmethod + async def delete(config_id: str) -> None: + """Delete a fanout config.""" + await db.conn.execute("DELETE FROM fanout_configs WHERE id = ?", (config_id,)) + await db.conn.commit() + _configs_cache.pop(config_id, None) + + @staticmethod + async def get_enabled() -> list[dict[str, Any]]: + """Get all enabled fanout configs.""" + cursor = await db.conn.execute( + "SELECT * FROM fanout_configs WHERE enabled = 1 ORDER BY sort_order, created_at" + ) + rows = await cursor.fetchall() + return [_row_to_dict(row) for row in rows] diff --git a/app/repository/settings.py b/app/repository/settings.py index 2914647c..351b910e 100644 --- a/app/repository/settings.py +++ b/app/repository/settings.py @@ -4,7 +4,7 @@ import time from typing import Any, Literal from app.database import db -from app.models import AppSettings, BotConfig, Favorite +from app.models import AppSettings, Favorite logger = logging.getLogger(__name__) @@ -26,13 +26,7 @@ class AppSettingsRepository: """ SELECT max_radio_contacts, favorites, auto_decrypt_dm_on_advert, sidebar_sort_order, last_message_times, preferences_migrated, - advert_interval, last_advert_time, bots, - mqtt_broker_host, mqtt_broker_port, mqtt_username, mqtt_password, - mqtt_use_tls, mqtt_tls_insecure, mqtt_topic_prefix, - mqtt_publish_messages, mqtt_publish_raw_packets, - community_mqtt_enabled, community_mqtt_iata, - community_mqtt_broker_host, community_mqtt_broker_port, - community_mqtt_email, flood_scope, + advert_interval, last_advert_time, flood_scope, blocked_keys, blocked_names FROM app_settings WHERE id = 1 """ @@ -69,20 +63,6 @@ class AppSettingsRepository: ) last_message_times = {} - # Parse bots JSON - bots: list[BotConfig] = [] - if row["bots"]: - try: - bots_data = json.loads(row["bots"]) - bots = [BotConfig(**b) for b in bots_data] - except (json.JSONDecodeError, TypeError, KeyError) as e: - logger.warning( - "Failed to parse bots JSON, using empty list: %s (data=%r)", - e, - row["bots"][:100] if row["bots"] else None, - ) - bots = [] - # Parse blocked_keys JSON blocked_keys: list[str] = [] if row["blocked_keys"]: @@ -113,22 +93,6 @@ class AppSettingsRepository: preferences_migrated=bool(row["preferences_migrated"]), advert_interval=row["advert_interval"] or 0, last_advert_time=row["last_advert_time"] or 0, - bots=bots, - mqtt_broker_host=row["mqtt_broker_host"] or "", - mqtt_broker_port=row["mqtt_broker_port"] or 1883, - mqtt_username=row["mqtt_username"] or "", - mqtt_password=row["mqtt_password"] or "", - mqtt_use_tls=bool(row["mqtt_use_tls"]), - mqtt_tls_insecure=bool(row["mqtt_tls_insecure"]), - mqtt_topic_prefix=row["mqtt_topic_prefix"] or "meshcore", - mqtt_publish_messages=bool(row["mqtt_publish_messages"]), - mqtt_publish_raw_packets=bool(row["mqtt_publish_raw_packets"]), - community_mqtt_enabled=bool(row["community_mqtt_enabled"]), - community_mqtt_iata=row["community_mqtt_iata"] or "", - community_mqtt_broker_host=row["community_mqtt_broker_host"] - or "mqtt-us-v1.letsmesh.net", - community_mqtt_broker_port=row["community_mqtt_broker_port"] or 443, - community_mqtt_email=row["community_mqtt_email"] or "", flood_scope=row["flood_scope"] or "", blocked_keys=blocked_keys, blocked_names=blocked_names, @@ -144,21 +108,6 @@ class AppSettingsRepository: preferences_migrated: bool | None = None, advert_interval: int | None = None, last_advert_time: int | None = None, - bots: list[BotConfig] | None = None, - mqtt_broker_host: str | None = None, - mqtt_broker_port: int | None = None, - mqtt_username: str | None = None, - mqtt_password: str | None = None, - mqtt_use_tls: bool | None = None, - mqtt_tls_insecure: bool | None = None, - mqtt_topic_prefix: str | None = None, - mqtt_publish_messages: bool | None = None, - mqtt_publish_raw_packets: bool | None = None, - community_mqtt_enabled: bool | None = None, - community_mqtt_iata: str | None = None, - community_mqtt_broker_host: str | None = None, - community_mqtt_broker_port: int | None = None, - community_mqtt_email: str | None = None, flood_scope: str | None = None, blocked_keys: list[str] | None = None, blocked_names: list[str] | None = None, @@ -200,67 +149,6 @@ class AppSettingsRepository: updates.append("last_advert_time = ?") params.append(last_advert_time) - if bots is not None: - updates.append("bots = ?") - bots_json = json.dumps([b.model_dump() for b in bots]) - params.append(bots_json) - - if mqtt_broker_host is not None: - updates.append("mqtt_broker_host = ?") - params.append(mqtt_broker_host) - - if mqtt_broker_port is not None: - updates.append("mqtt_broker_port = ?") - params.append(mqtt_broker_port) - - if mqtt_username is not None: - updates.append("mqtt_username = ?") - params.append(mqtt_username) - - if mqtt_password is not None: - updates.append("mqtt_password = ?") - params.append(mqtt_password) - - if mqtt_use_tls is not None: - updates.append("mqtt_use_tls = ?") - params.append(1 if mqtt_use_tls else 0) - - if mqtt_tls_insecure is not None: - updates.append("mqtt_tls_insecure = ?") - params.append(1 if mqtt_tls_insecure else 0) - - if mqtt_topic_prefix is not None: - updates.append("mqtt_topic_prefix = ?") - params.append(mqtt_topic_prefix) - - if mqtt_publish_messages is not None: - updates.append("mqtt_publish_messages = ?") - params.append(1 if mqtt_publish_messages else 0) - - if mqtt_publish_raw_packets is not None: - updates.append("mqtt_publish_raw_packets = ?") - params.append(1 if mqtt_publish_raw_packets else 0) - - if community_mqtt_enabled is not None: - updates.append("community_mqtt_enabled = ?") - params.append(1 if community_mqtt_enabled else 0) - - if community_mqtt_iata is not None: - updates.append("community_mqtt_iata = ?") - params.append(community_mqtt_iata) - - if community_mqtt_broker_host is not None: - updates.append("community_mqtt_broker_host = ?") - params.append(community_mqtt_broker_host) - - if community_mqtt_broker_port is not None: - updates.append("community_mqtt_broker_port = ?") - params.append(community_mqtt_broker_port) - - if community_mqtt_email is not None: - updates.append("community_mqtt_email = ?") - params.append(community_mqtt_email) - if flood_scope is not None: updates.append("flood_scope = ?") params.append(flood_scope) diff --git a/app/routers/fanout.py b/app/routers/fanout.py new file mode 100644 index 00000000..476c956a --- /dev/null +++ b/app/routers/fanout.py @@ -0,0 +1,235 @@ +"""REST API for fanout config CRUD.""" + +import logging +import re + +from fastapi import APIRouter, HTTPException +from pydantic import BaseModel, Field + +from app.config import settings as server_settings +from app.repository.fanout import FanoutConfigRepository + +logger = logging.getLogger(__name__) +router = APIRouter(prefix="/fanout", tags=["fanout"]) + +_VALID_TYPES = {"mqtt_private", "mqtt_community", "bot", "webhook", "apprise"} + +_IATA_RE = re.compile(r"^[A-Z]{3}$") + + +class FanoutConfigCreate(BaseModel): + type: str = Field(description="Integration type: 'mqtt_private' or 'mqtt_community'") + name: str = Field(min_length=1, description="User-assigned label") + config: dict = Field(default_factory=dict, description="Type-specific config blob") + scope: dict = Field(default_factory=dict, description="Scope controls") + enabled: bool = Field(default=True, description="Whether enabled on creation") + + +class FanoutConfigUpdate(BaseModel): + name: str | None = Field(default=None, min_length=1, description="Updated label") + config: dict | None = Field(default=None, description="Updated config blob") + scope: dict | None = Field(default=None, description="Updated scope controls") + enabled: bool | None = Field(default=None, description="Enable/disable toggle") + + +def _validate_mqtt_private_config(config: dict) -> None: + """Validate mqtt_private config blob.""" + if not config.get("broker_host"): + raise HTTPException(status_code=400, detail="broker_host is required for mqtt_private") + port = config.get("broker_port", 1883) + if not isinstance(port, int) or port < 1 or port > 65535: + raise HTTPException(status_code=400, detail="broker_port must be between 1 and 65535") + + +def _validate_mqtt_community_config(config: dict) -> None: + """Validate mqtt_community config blob. Normalizes IATA to uppercase.""" + iata = config.get("iata", "").upper().strip() + if not iata or not _IATA_RE.fullmatch(iata): + raise HTTPException( + status_code=400, + detail="IATA code is required and must be exactly 3 uppercase alphabetic characters", + ) + config["iata"] = iata + + +def _validate_bot_config(config: dict) -> None: + """Validate bot config blob (syntax-check the code).""" + code = config.get("code", "") + if not code or not code.strip(): + raise HTTPException(status_code=400, detail="Bot code cannot be empty") + try: + compile(code, "", "exec") + except SyntaxError as e: + raise HTTPException( + status_code=400, + detail=f"Bot code has syntax error at line {e.lineno}: {e.msg}", + ) from None + + +def _validate_apprise_config(config: dict) -> None: + """Validate apprise config blob.""" + urls = config.get("urls", "") + if not urls or not urls.strip(): + raise HTTPException(status_code=400, detail="At least one Apprise URL is required") + + +def _validate_webhook_config(config: dict) -> None: + """Validate webhook config blob.""" + url = config.get("url", "") + if not url: + raise HTTPException(status_code=400, detail="url is required for webhook") + if not url.startswith(("http://", "https://")): + raise HTTPException(status_code=400, detail="url must start with http:// or https://") + method = config.get("method", "POST").upper() + if method not in ("POST", "PUT", "PATCH"): + raise HTTPException(status_code=400, detail="method must be POST, PUT, or PATCH") + headers = config.get("headers", {}) + if not isinstance(headers, dict): + raise HTTPException(status_code=400, detail="headers must be a JSON object") + + +def _enforce_scope(config_type: str, scope: dict) -> dict: + """Enforce type-specific scope constraints. Returns normalized scope.""" + if config_type == "mqtt_community": + return {"messages": "none", "raw_packets": "all"} + if config_type == "bot": + return {"messages": "all", "raw_packets": "none"} + if config_type in ("webhook", "apprise"): + messages = scope.get("messages", "all") + if messages not in ("all", "none") and not isinstance(messages, dict): + raise HTTPException( + status_code=400, + detail="scope.messages must be 'all', 'none', or a filter object", + ) + return {"messages": messages, "raw_packets": "none"} + # For mqtt_private, validate scope values + messages = scope.get("messages", "all") + if messages not in ("all", "none") and not isinstance(messages, dict): + raise HTTPException( + status_code=400, + detail="scope.messages must be 'all', 'none', or a filter object", + ) + raw_packets = scope.get("raw_packets", "all") + if raw_packets not in ("all", "none"): + raise HTTPException( + status_code=400, + detail="scope.raw_packets must be 'all' or 'none'", + ) + return {"messages": messages, "raw_packets": raw_packets} + + +@router.get("") +async def list_fanout_configs() -> list[dict]: + """List all fanout configs.""" + return await FanoutConfigRepository.get_all() + + +@router.post("") +async def create_fanout_config(body: FanoutConfigCreate) -> dict: + """Create a new fanout config.""" + if body.type not in _VALID_TYPES: + raise HTTPException( + status_code=400, + detail=f"Invalid type '{body.type}'. Must be one of: {', '.join(sorted(_VALID_TYPES))}", + ) + + if body.type == "bot" and server_settings.disable_bots: + raise HTTPException(status_code=403, detail="Bot system disabled by server configuration") + + # Only validate config when creating as enabled — disabled configs + # are drafts the user hasn't finished configuring yet. + if body.enabled: + if body.type == "mqtt_private": + _validate_mqtt_private_config(body.config) + elif body.type == "mqtt_community": + _validate_mqtt_community_config(body.config) + elif body.type == "bot": + _validate_bot_config(body.config) + elif body.type == "webhook": + _validate_webhook_config(body.config) + elif body.type == "apprise": + _validate_apprise_config(body.config) + + scope = _enforce_scope(body.type, body.scope) + + cfg = await FanoutConfigRepository.create( + config_type=body.type, + name=body.name, + config=body.config, + scope=scope, + enabled=body.enabled, + ) + + # Start the module if enabled + if cfg["enabled"]: + from app.fanout.manager import fanout_manager + + await fanout_manager.reload_config(cfg["id"]) + + logger.info("Created fanout config %s (type=%s, name=%s)", cfg["id"], body.type, body.name) + return cfg + + +@router.patch("/{config_id}") +async def update_fanout_config(config_id: str, body: FanoutConfigUpdate) -> dict: + """Update a fanout config. Triggers module reload.""" + existing = await FanoutConfigRepository.get(config_id) + if existing is None: + raise HTTPException(status_code=404, detail="Fanout config not found") + + if existing["type"] == "bot" and server_settings.disable_bots: + raise HTTPException(status_code=403, detail="Bot system disabled by server configuration") + + kwargs = {} + if body.name is not None: + kwargs["name"] = body.name + if body.enabled is not None: + kwargs["enabled"] = body.enabled + if body.config is not None: + kwargs["config"] = body.config + if body.scope is not None: + kwargs["scope"] = _enforce_scope(existing["type"], body.scope) + + # Validate config when the result will be enabled + will_be_enabled = body.enabled if body.enabled is not None else existing["enabled"] + if will_be_enabled: + config_to_validate = body.config if body.config is not None else existing["config"] + if existing["type"] == "mqtt_private": + _validate_mqtt_private_config(config_to_validate) + elif existing["type"] == "mqtt_community": + _validate_mqtt_community_config(config_to_validate) + elif existing["type"] == "bot": + _validate_bot_config(config_to_validate) + elif existing["type"] == "webhook": + _validate_webhook_config(config_to_validate) + elif existing["type"] == "apprise": + _validate_apprise_config(config_to_validate) + + updated = await FanoutConfigRepository.update(config_id, **kwargs) + if updated is None: + raise HTTPException(status_code=404, detail="Fanout config not found") + + # Reload the module to pick up changes + from app.fanout.manager import fanout_manager + + await fanout_manager.reload_config(config_id) + + logger.info("Updated fanout config %s", config_id) + return updated + + +@router.delete("/{config_id}") +async def delete_fanout_config(config_id: str) -> dict: + """Delete a fanout config.""" + existing = await FanoutConfigRepository.get(config_id) + if existing is None: + raise HTTPException(status_code=404, detail="Fanout config not found") + + # Stop the module first + from app.fanout.manager import fanout_manager + + await fanout_manager.remove_config(config_id) + await FanoutConfigRepository.delete(config_id) + + logger.info("Deleted fanout config %s", config_id) + return {"deleted": True} diff --git a/app/routers/health.py b/app/routers/health.py index 5be6e6b8..39a53c94 100644 --- a/app/routers/health.py +++ b/app/routers/health.py @@ -1,4 +1,5 @@ import os +from typing import Any from fastapi import APIRouter from pydantic import BaseModel @@ -16,8 +17,7 @@ class HealthResponse(BaseModel): connection_info: str | None database_size_mb: float oldest_undecrypted_timestamp: int | None - mqtt_status: str | None = None - community_mqtt_status: str | None = None + fanout_statuses: dict[str, dict[str, str]] = {} bots_disabled: bool = False @@ -36,27 +36,12 @@ async def build_health_data(radio_connected: bool, connection_info: str | None) except RuntimeError: pass # Database not connected - # MQTT status - mqtt_status: str | None = None + # Fanout module statuses + fanout_statuses: dict[str, Any] = {} try: - from app.mqtt import mqtt_publisher + from app.fanout.manager import fanout_manager - if mqtt_publisher._is_configured(): - mqtt_status = "connected" if mqtt_publisher.connected else "disconnected" - else: - mqtt_status = "disabled" - except Exception: - pass - - # Community MQTT status - community_mqtt_status: str | None = None - try: - from app.community_mqtt import community_publisher - - if community_publisher._is_configured(): - community_mqtt_status = "connected" if community_publisher.connected else "disconnected" - else: - community_mqtt_status = "disabled" + fanout_statuses = fanout_manager.get_statuses() except Exception: pass @@ -66,8 +51,7 @@ async def build_health_data(radio_connected: bool, connection_info: str | None) "connection_info": connection_info, "database_size_mb": db_size_mb, "oldest_undecrypted_timestamp": oldest_ts, - "mqtt_status": mqtt_status, - "community_mqtt_status": community_mqtt_status, + "fanout_statuses": fanout_statuses, "bots_disabled": settings.disable_bots, } diff --git a/app/routers/messages.py b/app/routers/messages.py index 757b0ba6..9aabb6f7 100644 --- a/app/routers/messages.py +++ b/app/routers/messages.py @@ -1,4 +1,3 @@ -import asyncio import logging import time @@ -176,25 +175,9 @@ async def send_direct_message(request: SendDirectMessageRequest) -> Message: ) # Broadcast so all connected clients (not just sender) see the outgoing message immediately. + # Fanout modules (including bots) are triggered via broadcast_event's realtime dispatch. broadcast_event("message", message.model_dump()) - # Trigger bots for outgoing DMs (runs in background, doesn't block response) - from app.bot import run_bot_for_message - - asyncio.create_task( - run_bot_for_message( - sender_name=None, - sender_key=db_contact.public_key.lower(), - message_text=request.text, - is_dm=True, - channel_key=None, - channel_name=None, - sender_timestamp=now, - path=None, - is_outgoing=True, - ) - ) - return message @@ -313,6 +296,7 @@ async def send_channel_message(request: SendChannelMessageRequest) -> Message: acked=0, sender_name=radio_name or None, sender_key=our_public_key, + channel_name=db_channel.name, ).model_dump(), ) @@ -333,23 +317,7 @@ async def send_channel_message(request: SendChannelMessageRequest) -> Message: paths=paths, sender_name=radio_name or None, sender_key=our_public_key, - ) - - # Trigger bots for outgoing channel messages (runs in background, doesn't block response) - from app.bot import run_bot_for_message - - asyncio.create_task( - run_bot_for_message( - sender_name=radio_name or None, - sender_key=None, - message_text=request.text, - is_dm=False, - channel_key=channel_key_upper, - channel_name=db_channel.name, - sender_timestamp=now, - path=None, - is_outgoing=True, - ) + channel_name=db_channel.name, ) return message @@ -478,6 +446,7 @@ async def resend_channel_message( acked=0, sender_name=radio_name or None, sender_key=resend_public_key, + channel_name=db_channel.name, ).model_dump(), ) diff --git a/app/routers/packets.py b/app/routers/packets.py index 734e8787..b08754b4 100644 --- a/app/routers/packets.py +++ b/app/routers/packets.py @@ -71,7 +71,7 @@ async def _run_historical_channel_decryption( timestamp=result.timestamp, received_at=packet_timestamp, path=path_hex, - trigger_bot=False, # Historical decryption should not trigger bot + realtime=False, # Historical decryption should not trigger fanout ) if msg_id is not None: diff --git a/app/routers/settings.py b/app/routers/settings.py index 55e20b3f..7f220a35 100644 --- a/app/routers/settings.py +++ b/app/routers/settings.py @@ -1,40 +1,17 @@ import asyncio import logging -import re from typing import Literal -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter from pydantic import BaseModel, Field -from app.config import settings as server_settings -from app.models import AppSettings, BotConfig +from app.models import AppSettings from app.repository import AppSettingsRepository logger = logging.getLogger(__name__) router = APIRouter(prefix="/settings", tags=["settings"]) -def validate_bot_code(code: str, bot_name: str | None = None) -> None: - """Validate bot code syntax. Raises HTTPException on error.""" - if not code or not code.strip(): - return # Empty code is valid (disables bot) - - try: - compile(code, "", "exec") - except SyntaxError as e: - name_part = f"'{bot_name}' " if bot_name else "" - raise HTTPException( - status_code=400, - detail=f"Bot {name_part}has syntax error at line {e.lineno}: {e.msg}", - ) from None - - -def validate_all_bots(bots: list[BotConfig]) -> None: - """Validate all bots' code syntax. Raises HTTPException on first error.""" - for bot in bots: - validate_bot_code(bot.code, bot.name) - - class AppSettingsUpdate(BaseModel): max_radio_contacts: int | None = Field( default=None, @@ -57,70 +34,6 @@ class AppSettingsUpdate(BaseModel): ge=0, description="Periodic advertisement interval in seconds (0 = disabled, minimum 3600)", ) - bots: list[BotConfig] | None = Field( - default=None, - description="List of bot configurations", - ) - mqtt_broker_host: str | None = Field( - default=None, - description="MQTT broker hostname (empty = disabled)", - ) - mqtt_broker_port: int | None = Field( - default=None, - ge=1, - le=65535, - description="MQTT broker port", - ) - mqtt_username: str | None = Field( - default=None, - description="MQTT username (optional)", - ) - mqtt_password: str | None = Field( - default=None, - description="MQTT password (optional)", - ) - mqtt_use_tls: bool | None = Field( - default=None, - description="Whether to use TLS for MQTT connection", - ) - mqtt_tls_insecure: bool | None = Field( - default=None, - description="Skip TLS certificate verification (for self-signed certs)", - ) - mqtt_topic_prefix: str | None = Field( - default=None, - description="MQTT topic prefix", - ) - mqtt_publish_messages: bool | None = Field( - default=None, - description="Whether to publish decrypted messages to MQTT", - ) - mqtt_publish_raw_packets: bool | None = Field( - default=None, - description="Whether to publish raw packets to MQTT", - ) - community_mqtt_enabled: bool | None = Field( - default=None, - description="Whether to publish raw packets to the community MQTT broker", - ) - community_mqtt_iata: str | None = Field( - default=None, - description="IATA region code for community MQTT topic routing (3 alpha chars)", - ) - community_mqtt_broker_host: str | None = Field( - default=None, - description="Community MQTT broker hostname", - ) - community_mqtt_broker_port: int | None = Field( - default=None, - ge=1, - le=65535, - description="Community MQTT broker port", - ) - community_mqtt_email: str | None = Field( - default=None, - description="Email address for node claiming on the community aggregator", - ) flood_scope: str | None = Field( default=None, description="Outbound flood scope / region name (empty = disabled)", @@ -201,62 +114,6 @@ async def update_settings(update: AppSettingsUpdate) -> AppSettings: logger.info("Updating advert_interval to %d", interval) kwargs["advert_interval"] = interval - if update.bots is not None: - if server_settings.disable_bots: - raise HTTPException( - status_code=403, detail="Bot system disabled by server configuration" - ) - validate_all_bots(update.bots) - logger.info("Updating bots (count=%d)", len(update.bots)) - kwargs["bots"] = update.bots - - # MQTT fields - mqtt_fields = [ - "mqtt_broker_host", - "mqtt_broker_port", - "mqtt_username", - "mqtt_password", - "mqtt_use_tls", - "mqtt_tls_insecure", - "mqtt_topic_prefix", - "mqtt_publish_messages", - "mqtt_publish_raw_packets", - ] - mqtt_changed = False - for field in mqtt_fields: - value = getattr(update, field) - if value is not None: - kwargs[field] = value - mqtt_changed = True - - # Community MQTT fields - community_mqtt_changed = False - if update.community_mqtt_enabled is not None: - kwargs["community_mqtt_enabled"] = update.community_mqtt_enabled - community_mqtt_changed = True - - if update.community_mqtt_iata is not None: - iata = update.community_mqtt_iata.upper().strip() - if iata and not re.fullmatch(r"[A-Z]{3}", iata): - raise HTTPException( - status_code=400, - detail="IATA code must be exactly 3 uppercase alphabetic characters", - ) - kwargs["community_mqtt_iata"] = iata - community_mqtt_changed = True - - if update.community_mqtt_broker_host is not None: - kwargs["community_mqtt_broker_host"] = update.community_mqtt_broker_host - community_mqtt_changed = True - - if update.community_mqtt_broker_port is not None: - kwargs["community_mqtt_broker_port"] = update.community_mqtt_broker_port - community_mqtt_changed = True - - if update.community_mqtt_email is not None: - kwargs["community_mqtt_email"] = update.community_mqtt_email - community_mqtt_changed = True - # Block lists if update.blocked_keys is not None: kwargs["blocked_keys"] = [k.lower() for k in update.blocked_keys] @@ -270,34 +127,9 @@ async def update_settings(update: AppSettingsUpdate) -> AppSettings: kwargs["flood_scope"] = stripped flood_scope_changed = True - # Require IATA when enabling community MQTT - if kwargs.get("community_mqtt_enabled", False): - # Check the IATA value being set, or fall back to current settings - iata_value = kwargs.get("community_mqtt_iata") - if iata_value is None: - current = await AppSettingsRepository.get() - iata_value = current.community_mqtt_iata - if not iata_value or not re.fullmatch(r"[A-Z]{3}", iata_value): - raise HTTPException( - status_code=400, - detail="A valid IATA region code is required to enable community sharing", - ) - if kwargs: result = await AppSettingsRepository.update(**kwargs) - # Restart MQTT publisher if any MQTT settings changed - if mqtt_changed: - from app.mqtt import mqtt_publisher - - await mqtt_publisher.restart(result) - - # Restart community MQTT publisher if any community settings changed - if community_mqtt_changed: - from app.community_mqtt import community_publisher - - await community_publisher.restart(result) - # Apply flood scope to radio immediately if changed if flood_scope_changed: from app.radio import radio_manager diff --git a/app/websocket.py b/app/websocket.py index 16380357..3ceb705e 100644 --- a/app/websocket.py +++ b/app/websocket.py @@ -92,21 +92,26 @@ class WebSocketManager: ws_manager = WebSocketManager() -def broadcast_event(event_type: str, data: dict) -> None: +def broadcast_event(event_type: str, data: dict, *, realtime: bool = True) -> None: """Schedule a broadcast without blocking. Convenience function that creates an asyncio task to broadcast - an event to all connected WebSocket clients and forward to MQTT. + an event to all connected WebSocket clients and forward to fanout modules. + + Args: + event_type: Event type string (e.g. "message", "raw_packet") + data: Event payload dict + realtime: If False, skip fanout dispatch (used for historical decryption) """ asyncio.create_task(ws_manager.broadcast(event_type, data)) - from app.mqtt import mqtt_broadcast + if realtime: + from app.fanout.manager import fanout_manager - mqtt_broadcast(event_type, data) - - from app.community_mqtt import community_mqtt_broadcast - - community_mqtt_broadcast(event_type, data) + if event_type == "message": + asyncio.create_task(fanout_manager.broadcast_message(data)) + elif event_type == "raw_packet": + asyncio.create_task(fanout_manager.broadcast_raw(data)) def broadcast_error(message: str, details: str | None = None) -> None: diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index 61bcc5e4..095bd53c 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -84,9 +84,8 @@ frontend/src/ │ │ ├── settingsConstants.ts # Settings section type, ordering, labels │ │ ├── SettingsRadioSection.tsx # Name, keys, advert interval, max contacts, radio preset, freq/bw/sf/cr, txPower, lat/lon, reboot │ │ ├── SettingsLocalSection.tsx # Browser-local settings: theme, local label, reopen last conversation -│ │ ├── SettingsMqttSection.tsx # MQTT broker config, TLS, publish toggles +│ │ ├── SettingsFanoutSection.tsx # Fanout integrations: MQTT, bots, config CRUD │ │ ├── SettingsDatabaseSection.tsx # DB size, cleanup, auto-decrypt, local label -│ │ ├── SettingsBotSection.tsx # Bot list, code editor, add/delete/reset │ │ ├── SettingsStatisticsSection.tsx # Read-only mesh network stats │ │ ├── SettingsAboutSection.tsx # Version, author, license, links │ │ └── ThemeSelector.tsx # Color theme picker @@ -242,15 +241,14 @@ LocalStorage migration helpers for favorites; canonical favorites are server-sid - `preferences_migrated` - `advert_interval` - `last_advert_time` -- `bots` -- `mqtt_broker_host`, `mqtt_broker_port`, `mqtt_username`, `mqtt_password` -- `mqtt_use_tls`, `mqtt_tls_insecure`, `mqtt_topic_prefix`, `mqtt_publish_messages`, `mqtt_publish_raw_packets` -- `community_mqtt_enabled`, `community_mqtt_iata`, `community_mqtt_broker_host`, `community_mqtt_broker_port`, `community_mqtt_email` - `flood_scope` - `blocked_keys`, `blocked_names` -`HealthStatus` includes `mqtt_status` (`"connected"`, `"disconnected"`, `"disabled"`, or `null`). -`HealthStatus` also includes `community_mqtt_status` with the same status values. +Note: MQTT, bot, and community MQTT settings were migrated to the `fanout_configs` table (managed via `/api/fanout`). They are no longer part of `AppSettings`. + +`HealthStatus` includes `fanout_statuses: Record` mapping config IDs to `{name, type, status}`. Also includes `bots_disabled: boolean`. + +`FanoutConfig` represents a single fanout integration: `{id, type, name, enabled, config, scope, sort_order, created_at}`. `RawPacket.decrypted_info` includes `channel_key` and `contact_key` for MQTT topic routing. diff --git a/frontend/src/api.ts b/frontend/src/api.ts index d2ca63a0..b4b4d113 100644 --- a/frontend/src/api.ts +++ b/frontend/src/api.ts @@ -8,6 +8,7 @@ import type { ContactAdvertPath, ContactAdvertPathSummary, ContactDetail, + FanoutConfig, Favorite, HealthStatus, MaintenanceResult, @@ -280,6 +281,37 @@ export const api = { body: JSON.stringify(request), }), + // Fanout + getFanoutConfigs: () => fetchJson('/fanout'), + createFanoutConfig: (config: { + type: string; + name: string; + config: Record; + scope: Record; + enabled?: boolean; + }) => + fetchJson('/fanout', { + method: 'POST', + body: JSON.stringify(config), + }), + updateFanoutConfig: ( + id: string, + update: { + name?: string; + config?: Record; + scope?: Record; + enabled?: boolean; + } + ) => + fetchJson(`/fanout/${id}`, { + method: 'PATCH', + body: JSON.stringify(update), + }), + deleteFanoutConfig: (id: string) => + fetchJson<{ deleted: boolean }>(`/fanout/${id}`, { + method: 'DELETE', + }), + // Statistics getStatistics: () => fetchJson('/statistics'), diff --git a/frontend/src/components/SettingsModal.tsx b/frontend/src/components/SettingsModal.tsx index dc948014..3cbb8513 100644 --- a/frontend/src/components/SettingsModal.tsx +++ b/frontend/src/components/SettingsModal.tsx @@ -11,9 +11,8 @@ import { SETTINGS_SECTION_LABELS, type SettingsSection } from './settings/settin import { SettingsRadioSection } from './settings/SettingsRadioSection'; import { SettingsLocalSection } from './settings/SettingsLocalSection'; -import { SettingsMqttSection } from './settings/SettingsMqttSection'; +import { SettingsFanoutSection } from './settings/SettingsFanoutSection'; import { SettingsDatabaseSection } from './settings/SettingsDatabaseSection'; -import { SettingsBotSection } from './settings/SettingsBotSection'; import { SettingsStatisticsSection } from './settings/SettingsStatisticsSection'; import { SettingsAboutSection } from './settings/SettingsAboutSection'; @@ -78,9 +77,8 @@ export function SettingsModal(props: SettingsModalProps) { const [expandedSections, setExpandedSections] = useState>({ radio: false, local: false, - mqtt: false, + fanout: false, database: false, - bot: false, statistics: false, about: false, }); @@ -217,29 +215,13 @@ export function SettingsModal(props: SettingsModalProps) { )} - {shouldRenderSection('bot') && ( + {shouldRenderSection('fanout') && (
- {renderSectionHeader('bot')} - {isSectionVisible('bot') && appSettings && ( - - )} -
- )} - - {shouldRenderSection('mqtt') && ( -
- {renderSectionHeader('mqtt')} - {isSectionVisible('mqtt') && appSettings && ( - )} diff --git a/frontend/src/components/settings/SettingsBotSection.tsx b/frontend/src/components/settings/SettingsBotSection.tsx deleted file mode 100644 index c69b2707..00000000 --- a/frontend/src/components/settings/SettingsBotSection.tsx +++ /dev/null @@ -1,335 +0,0 @@ -import { useState, useEffect, lazy, Suspense } from 'react'; -import { Label } from '../ui/label'; -import { Button } from '../ui/button'; -import { Separator } from '../ui/separator'; -import { toast } from '../ui/sonner'; -import type { AppSettings, AppSettingsUpdate, BotConfig, HealthStatus } from '../../types'; - -const BotCodeEditor = lazy(() => - import('../BotCodeEditor').then((m) => ({ default: m.BotCodeEditor })) -); - -const DEFAULT_BOT_CODE = `def bot( - sender_name: str | None, - sender_key: str | None, - message_text: str, - is_dm: bool, - channel_key: str | None, - channel_name: str | None, - sender_timestamp: int | None, - path: str | None, - is_outgoing: bool = False, -) -> str | list[str] | None: - """ - Process messages and optionally return a reply. - - Args: - sender_name: Display name of sender (may be None) - sender_key: 64-char hex public key (None for channel msgs) - message_text: The message content - is_dm: True for direct messages, False for channel - channel_key: 32-char hex key for channels, None for DMs - channel_name: Channel name with hash (e.g. "#bot"), None for DMs - sender_timestamp: Sender's timestamp (unix seconds, may be None) - path: Hex-encoded routing path (may be None) - is_outgoing: True if this is our own outgoing message - - Returns: - None for no reply, a string for a single reply, - or a list of strings to send multiple messages in order - """ - # Don't reply to our own outgoing messages - if is_outgoing: - return None - - # Example: Only respond in #bot channel to "!pling" command - if channel_name == "#bot" and "!pling" in message_text.lower(): - return "[BOT] Plong!" - return None`; - -export function SettingsBotSection({ - appSettings, - health, - isMobileLayout, - onSaveAppSettings, - className, -}: { - appSettings: AppSettings; - health: HealthStatus | null; - isMobileLayout: boolean; - onSaveAppSettings: (update: AppSettingsUpdate) => Promise; - className?: string; -}) { - const [bots, setBots] = useState([]); - const [expandedBotId, setExpandedBotId] = useState(null); - const [editingNameId, setEditingNameId] = useState(null); - const [editingNameValue, setEditingNameValue] = useState(''); - - const [busy, setBusy] = useState(false); - const [error, setError] = useState(null); - - useEffect(() => { - setBots(appSettings.bots || []); - }, [appSettings]); - - const handleSave = async () => { - setBusy(true); - setError(null); - - try { - await onSaveAppSettings({ bots }); - toast.success('Bot settings saved'); - } catch (err) { - console.error('Failed to save bot settings:', err); - const errorMsg = err instanceof Error ? err.message : 'Failed to save'; - setError(errorMsg); - toast.error(errorMsg); - } finally { - setBusy(false); - } - }; - - const handleAddBot = () => { - const newBot: BotConfig = { - id: crypto.randomUUID(), - name: `Bot ${bots.length + 1}`, - enabled: false, - code: DEFAULT_BOT_CODE, - }; - setBots([...bots, newBot]); - setExpandedBotId(newBot.id); - }; - - const handleDeleteBot = (botId: string) => { - const bot = bots.find((b) => b.id === botId); - if (bot && bot.code.trim() && bot.code !== DEFAULT_BOT_CODE) { - if (!confirm(`Delete "${bot.name}"? This will remove all its code.`)) { - return; - } - } - setBots(bots.filter((b) => b.id !== botId)); - if (expandedBotId === botId) { - setExpandedBotId(null); - } - }; - - const handleToggleBotEnabled = (botId: string) => { - setBots(bots.map((b) => (b.id === botId ? { ...b, enabled: !b.enabled } : b))); - }; - - const handleBotCodeChange = (botId: string, code: string) => { - setBots(bots.map((b) => (b.id === botId ? { ...b, code } : b))); - }; - - const handleStartEditingName = (bot: BotConfig) => { - setEditingNameId(bot.id); - setEditingNameValue(bot.name); - }; - - const handleFinishEditingName = () => { - if (editingNameId && editingNameValue.trim()) { - setBots( - bots.map((b) => (b.id === editingNameId ? { ...b, name: editingNameValue.trim() } : b)) - ); - } - setEditingNameId(null); - setEditingNameValue(''); - }; - - const handleResetBotCode = (botId: string) => { - setBots(bots.map((b) => (b.id === botId ? { ...b, code: DEFAULT_BOT_CODE } : b))); - }; - - if (health?.bots_disabled) { - return ( -
-

Bot system disabled by server startup flag.

-
- ); - } - - return ( -
-
-

- Experimental: This is an alpha feature and introduces automated message - sending to your radio; unexpected behavior may occur. Use with caution, and please report - any bugs! -

-
- -
-

- Security Warning: This feature executes arbitrary Python code on the - server. Only run trusted code, and be cautious of arbitrary usage of message parameters. -

-
- -
-

- Don't wreck the mesh! Bots process ALL messages, including their - own. Be careful of creating infinite loops! -

-
- -
- - -
- - {bots.length === 0 ? ( -
-

No bots configured

- -
- ) : ( -
- {bots.map((bot) => ( -
- - - - {expandedBotId === bot.id && ( -
-
-

- Define a bot() function that - receives message data and optionally returns a reply. -

- -
- - Loading editor... -
- } - > - handleBotCodeChange(bot.id, code)} - id={`bot-code-${bot.id}`} - height={isMobileLayout ? '256px' : '384px'} - /> - -
- )} -
- ))} -
- )} - - - -
-

- Available: Standard Python libraries and any modules installed in the - server environment. -

-

- Limits: 10 second timeout per bot. -

-

- Note: Bots respond to all messages, including your own. For channel - messages, sender_key is None. Multiple enabled bots run - serially, with a two-second delay between messages to prevent repeater collision. -

-
- - {error && ( -
- {error} -
- )} - - - - ); -} diff --git a/frontend/src/components/settings/SettingsFanoutSection.tsx b/frontend/src/components/settings/SettingsFanoutSection.tsx new file mode 100644 index 00000000..4c2c1375 --- /dev/null +++ b/frontend/src/components/settings/SettingsFanoutSection.tsx @@ -0,0 +1,1165 @@ +import { useState, useEffect, useCallback, lazy, Suspense } from 'react'; +import { Input } from '../ui/input'; +import { Label } from '../ui/label'; +import { Button } from '../ui/button'; +import { Separator } from '../ui/separator'; +import { toast } from '../ui/sonner'; +import { cn } from '@/lib/utils'; +import { api } from '../../api'; +import type { Channel, Contact, FanoutConfig, HealthStatus } from '../../types'; + +const BotCodeEditor = lazy(() => + import('../BotCodeEditor').then((m) => ({ default: m.BotCodeEditor })) +); + +const TYPE_LABELS: Record = { + mqtt_private: 'Private MQTT', + mqtt_community: 'Community MQTT', + bot: 'Bot', + webhook: 'Webhook', + apprise: 'Apprise', +}; + +const TYPE_OPTIONS = [ + { value: 'mqtt_private', label: 'Private MQTT' }, + { value: 'mqtt_community', label: 'Community MQTT' }, + { value: 'bot', label: 'Bot' }, + { value: 'webhook', label: 'Webhook' }, + { value: 'apprise', label: 'Apprise' }, +]; + +const DEFAULT_BOT_CODE = `def bot( + sender_name: str | None, + sender_key: str | None, + message_text: str, + is_dm: bool, + channel_key: str | None, + channel_name: str | None, + sender_timestamp: int | None, + path: str | None, + is_outgoing: bool = False, +) -> str | list[str] | None: + """ + Process messages and optionally return a reply. + + Args: + sender_name: Display name of sender (may be None) + sender_key: 64-char hex public key (None for channel msgs) + message_text: The message content + is_dm: True for direct messages, False for channel + channel_key: 32-char hex key for channels, None for DMs + channel_name: Channel name with hash (e.g. "#bot"), None for DMs + sender_timestamp: Sender's timestamp (unix seconds, may be None) + path: Hex-encoded routing path (may be None) + is_outgoing: True if this is our own outgoing message + + Returns: + None for no reply, a string for a single reply, + or a list of strings to send multiple messages in order + """ + # Don't reply to our own outgoing messages + if is_outgoing: + return None + + # Example: Only respond in #bot channel to "!pling" command + if channel_name == "#bot" and "!pling" in message_text.lower(): + return "[BOT] Plong!" + return None`; + +function getStatusLabel(status: string | undefined, type?: string) { + if (status === 'connected') + return type === 'bot' || type === 'webhook' || type === 'apprise' ? 'Active' : 'Connected'; + if (status === 'error') return 'Error'; + if (status === 'disconnected') return 'Disconnected'; + return 'Inactive'; +} + +function getStatusColor(status: string | undefined, enabled?: boolean) { + if (enabled === false) return 'bg-warning shadow-[0_0_6px_hsl(var(--warning)/0.5)]'; + if (status === 'connected') + return 'bg-status-connected shadow-[0_0_6px_hsl(var(--status-connected)/0.5)]'; + if (status === 'error') return 'bg-destructive shadow-[0_0_6px_hsl(var(--destructive)/0.5)]'; + return 'bg-muted-foreground'; +} + +function MqttPrivateConfigEditor({ + config, + scope, + onChange, + onScopeChange, +}: { + config: Record; + scope: Record; + onChange: (config: Record) => void; + onScopeChange: (scope: Record) => void; +}) { + return ( +
+

+ Forward mesh data to your own MQTT broker for home automation, logging, or alerting. +

+ +
+ Outgoing messages (DMs and group messages) will be reported to private MQTT brokers in + decrypted/plaintext form. +
+ +
+
+ + onChange({ ...config, broker_host: e.target.value })} + /> +
+
+ + + onChange({ ...config, broker_port: parseInt(e.target.value, 10) || 1883 }) + } + /> +
+
+ +
+
+ + onChange({ ...config, username: e.target.value })} + /> +
+
+ + onChange({ ...config, password: e.target.value })} + /> +
+
+ + + + {!!config.use_tls && ( + + )} + + + +
+ + onChange({ ...config, topic_prefix: e.target.value })} + /> +
+ + + + +
+ ); +} + +function MqttCommunityConfigEditor({ + config, + onChange, +}: { + config: Record; + onChange: (config: Record) => void; +}) { + return ( +
+

+ Share raw packet data with the MeshCore community for coverage mapping and network analysis. + Only raw RF packets are shared — never decrypted messages. +

+ +
+
+ + onChange({ ...config, broker_host: e.target.value })} + /> +
+
+ + + onChange({ ...config, broker_port: parseInt(e.target.value, 10) || 443 }) + } + /> +
+
+ +
+ + onChange({ ...config, iata: e.target.value.toUpperCase() })} + className="w-32" + /> +

+ Your nearest airport's IATA code (required) +

+
+ +
+ + onChange({ ...config, email: e.target.value })} + /> +

+ Used to claim your node on the community aggregator +

+
+
+ ); +} + +function BotConfigEditor({ + config, + onChange, +}: { + config: Record; + onChange: (config: Record) => void; +}) { + const code = (config.code as string) || ''; + return ( +
+
+

+ Experimental: This is an alpha feature and introduces automated message + sending to your radio; unexpected behavior may occur. Use with caution, and please report + any bugs! +

+
+ +
+

+ Security Warning: This feature executes arbitrary Python code on the + server. Only run trusted code, and be cautious of arbitrary usage of message parameters. +

+
+ +
+

+ Don't wreck the mesh! Bots process ALL messages, including their + own. Be careful of creating infinite loops! +

+
+ +
+

+ Define a bot() function that receives + message data and optionally returns a reply. +

+ +
+ + + Loading editor... +
+ } + > + onChange({ ...config, code: c })} /> + + +
+

+ Available: Standard Python libraries and any modules installed in the + server environment. +

+

+ Limits: 10 second timeout per bot. +

+

+ Note: Bots respond to all messages, including your own. For channel + messages, sender_key is None. Multiple enabled bots run + concurrently. Outgoing messages are serialized with a two-second delay between sends to + prevent repeater collision. +

+
+ + ); +} + +type ScopeMode = 'all' | 'none' | 'only' | 'except'; + +function getScopeMode(value: unknown): ScopeMode { + if (value === 'all') return 'all'; + if (value === 'none') return 'none'; + if (typeof value === 'object' && value !== null) { + // Check if either channels or contacts uses the {except: [...]} shape + const obj = value as Record; + const ch = obj.channels; + const co = obj.contacts; + if ( + (typeof ch === 'object' && ch !== null && !Array.isArray(ch)) || + (typeof co === 'object' && co !== null && !Array.isArray(co)) + ) { + return 'except'; + } + return 'only'; + } + return 'all'; +} + +/** Extract the key list from a filter value, whether it's a plain list or {except: [...]} */ +function getFilterKeys(filter: unknown): string[] { + if (Array.isArray(filter)) return filter as string[]; + if (typeof filter === 'object' && filter !== null && 'except' in filter) + return ((filter as Record).except as string[]) ?? []; + return []; +} + +function ScopeSelector({ + scope, + onChange, + showRawPackets = false, +}: { + scope: Record; + onChange: (scope: Record) => void; + showRawPackets?: boolean; +}) { + const [channels, setChannels] = useState([]); + const [contacts, setContacts] = useState([]); + + useEffect(() => { + api.getChannels().then(setChannels).catch(console.error); + + // Paginate to fetch all contacts (API caps at 1000 per request) + (async () => { + const all: Contact[] = []; + const pageSize = 1000; + let offset = 0; + + while (true) { + const page = await api.getContacts(pageSize, offset); + all.push(...page); + if (page.length < pageSize) break; + offset += pageSize; + } + setContacts(all); + })().catch(console.error); + }, []); + + const messages = scope.messages ?? 'all'; + const rawMode = getScopeMode(messages); + // When raw packets aren't offered, "none" is not a valid choice — treat as "all" + const mode = !showRawPackets && rawMode === 'none' ? 'all' : rawMode; + const isListMode = mode === 'only' || mode === 'except'; + + const selectedChannels: string[] = + isListMode && typeof messages === 'object' && messages !== null + ? getFilterKeys((messages as Record).channels) + : []; + const selectedContacts: string[] = + isListMode && typeof messages === 'object' && messages !== null + ? getFilterKeys((messages as Record).contacts) + : []; + + /** Wrap channel/contact key lists in the right shape for the current mode */ + const buildMessages = (chKeys: string[], coKeys: string[]) => { + if (mode === 'except') { + return { + channels: { except: chKeys }, + contacts: { except: coKeys }, + }; + } + return { channels: chKeys, contacts: coKeys }; + }; + + const handleModeChange = (newMode: ScopeMode) => { + if (newMode === 'all' || newMode === 'none') { + onChange({ ...scope, messages: newMode }); + } else if (newMode === 'only') { + onChange({ ...scope, messages: { channels: [], contacts: [] } }); + } else { + onChange({ + ...scope, + messages: { channels: { except: [] }, contacts: { except: [] } }, + }); + } + }; + + const toggleChannel = (key: string) => { + const current = [...selectedChannels]; + const idx = current.indexOf(key); + if (idx >= 0) current.splice(idx, 1); + else current.push(key); + onChange({ ...scope, messages: buildMessages(current, selectedContacts) }); + }; + + const toggleContact = (key: string) => { + const current = [...selectedContacts]; + const idx = current.indexOf(key); + if (idx >= 0) current.splice(idx, 1); + else current.push(key); + onChange({ ...scope, messages: buildMessages(selectedChannels, current) }); + }; + + // Exclude repeaters (2), rooms (3), and sensors (4) + const filteredContacts = contacts.filter((c) => c.type === 0 || c.type === 1); + + const modeDescriptions: Record = { + all: 'All messages', + none: 'No messages', + only: 'Only listed channels/contacts', + except: 'All except listed channels/contacts', + }; + + const rawEnabled = showRawPackets && scope.raw_packets === 'all'; + + // Warn when the effective scope matches nothing + const messagesEffectivelyNone = + mode === 'none' || + (mode === 'only' && selectedChannels.length === 0 && selectedContacts.length === 0) || + (mode === 'except' && + channels.length > 0 && + filteredContacts.length > 0 && + selectedChannels.length >= channels.length && + selectedContacts.length >= filteredContacts.length); + const showEmptyScopeWarning = messagesEffectivelyNone && !rawEnabled; + + const isChannelChecked = (key: string) => selectedChannels.includes(key); + const isContactChecked = (key: string) => selectedContacts.includes(key); + + const listHint = + mode === 'only' + ? 'Newly added channels or contacts will not be automatically included.' + : 'Newly added channels or contacts will be automatically included unless excluded here.'; + + const checkboxLabel = mode === 'except' ? 'exclude' : 'include'; + + const messageModes: ScopeMode[] = showRawPackets + ? ['all', 'none', 'only', 'except'] + : ['all', 'only', 'except']; + + return ( +
+ + + {showRawPackets && ( + + )} + +
+ {messageModes.map((m) => ( + + ))} +
+ + {showEmptyScopeWarning && ( +
+ Nothing is selected — this integration will not forward any data. +
+ )} + + {isListMode && ( + <> +

{listHint}

+ + {channels.length > 0 && ( +
+
+ + + + / + + +
+
+ {channels.map((ch) => ( + + ))} +
+
+ )} + + {filteredContacts.length > 0 && ( +
+
+ + + + / + + +
+
+ {filteredContacts.map((c) => ( + + ))} +
+
+ )} + + )} +
+ ); +} + +function AppriseConfigEditor({ + config, + scope, + onChange, + onScopeChange, +}: { + config: Record; + scope: Record; + onChange: (config: Record) => void; + onScopeChange: (scope: Record) => void; +}) { + return ( +
+

+ Send push notifications via{' '} + + Apprise + {' '} + when messages are received. Supports Discord, Slack, Telegram, email, and{' '} + + 100+ other services + + . +

+ +
+ +