From 202acacfe18d3794544e40409f4feff911ce8b80 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sat, 28 Mar 2026 17:08:32 -0700 Subject: [PATCH] Enhance MQTT implementation by adding optional SNMP support, allowing for monitoring and management of MQTT statistics. Update configuration options to include SNMP settings, and modify the MQTTBridge to integrate SNMP agent functionality. Additionally, improve documentation for the MQTT observer setup and expand the number of concurrent MQTT connection slots from 3 to 5, enhancing overall flexibility and performance. --- MQTT_IMPLEMENTATION.md | 233 +++++++++++++++++++--------- MQTT_SNMP.md | 149 ++++++++++++++++++ examples/simple_repeater/MyMesh.cpp | 33 ++++ examples/simple_repeater/MyMesh.h | 7 + src/helpers/CommonCLI.cpp | 28 +++- src/helpers/CommonCLI.h | 8 +- src/helpers/SNMPAgent.cpp | 123 +++++++++++++++ src/helpers/SNMPAgent.h | 79 ++++++++++ src/helpers/bridges/MQTTBridge.cpp | 108 +++++++++++-- src/helpers/bridges/MQTTBridge.h | 21 ++- variants/heltec_v3/platformio.ini | 5 +- variants/heltec_v4/platformio.ini | 5 +- variants/station_g2/platformio.ini | 3 + 13 files changed, 706 insertions(+), 96 deletions(-) create mode 100644 MQTT_SNMP.md create mode 100644 src/helpers/SNMPAgent.cpp create mode 100644 src/helpers/SNMPAgent.h diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index 3b5bdbba..0a3408ca 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -4,29 +4,55 @@ This document describes the MQTT bridge implementation that allows MeshCore repe ## Quick Start Guide -### Essential Commands to Get MQTT Repeater Running +### Essential Commands to Get MQTT Observer Running -**1. Connect to device console via repeater login or serial console (115200 baud)** +**1. Flash the observer firmware to your device** -**2. Configure WiFi Credentials** +Use one of the observer build targets (e.g., `heltec_v4_repeater_observer_mqtt`). After flashing, connect to the device console via serial (115200 baud) or repeater login. + +**2. Configure radio settings** + +If this is a fresh flash or full erase, configure your radio parameters first. These must match other nodes in your mesh: + +```bash +set freq 906.0 +set sf 12 +set cr 5 +set bw 250 +set tx 22 +``` + +**3. Configure device identity** + +```bash +set name MyObserver +set mqtt.iata SEA +``` + +If migrating from an existing node (e.g., a Raspberry Pi gateway), restore the private key to keep the same identity: +```bash +set prv.key +``` + +**4. Configure WiFi credentials** ```bash set wifi.ssid YourWiFiNetwork set wifi.pwd YourWiFiPassword ``` -If you wish to upload to the MeshCore Analyzer, also `set mqtt.iata XXX` to a valid IATA airport code. +**5. (Optional) Disable packet repeating** -**3. Reboot to Connect to WiFi** +If this observer is receive-only (e.g., using a PCB antenna in a location where repeating would be harmful), disable forwarding: +```bash +set repeat off +``` + +**6. Reboot to connect** ```bash reboot ``` -**4. Toggle bridge.source to rx** -```bash -set bridge.source rx -``` - -**5. Verify Configuration** +**7. Verify configuration** ```bash get wifi.ssid get bridge.enabled @@ -36,16 +62,7 @@ get mqtt.iata get mqtt1.preset get mqtt2.preset get mqtt3.preset -``` - -**6. Restart Bridge (if needed)** -```bash -# Option A: Toggle bridge off then on -set bridge.enabled off -set bridge.enabled on - -# Option B: Full device reboot -reboot +get mqtt.status ``` **That's it!** The device will now: @@ -59,8 +76,8 @@ reboot ## Overview The MQTT bridge implementation provides: -- Up to 3 concurrent MQTT connection slots with built-in presets -- Built-in presets for LetsMesh Analyzer (US/EU) and MeshMapper +- Up to 6 MQTT connection slots with built-in presets +- Built-in presets for LetsMesh Analyzer (US/EU), MeshMapper, MeshRank, Waev, Meshomatic, and CascadiaMesh - Custom broker support with username/password authentication - JWT (Ed25519 device signing) authentication for preset brokers - WSS (WebSocket Secure) and direct MQTT transport @@ -73,7 +90,7 @@ The MQTT bridge implementation provides: ### Slot-Based Preset System -The MQTT bridge uses a slot-based architecture with up to 3 concurrent connections. Each slot can be configured with a built-in preset or custom broker settings. +The MQTT bridge uses a slot-based architecture with up to 6 concurrent connections. Each slot can be configured with a built-in preset or custom broker settings. **Built-in Presets:** @@ -83,17 +100,20 @@ The MQTT bridge uses a slot-based architecture with up to 3 concurrent connectio | `analyzer-eu` | mqtt-eu-v1.letsmesh.net:443 | JWT (Ed25519) | WSS | | `meshmapper` | mqtt.meshmapper.cc:443 | JWT (Ed25519) | WSS | | `meshrank` | meshrank.net:8883 | None (token in topic) | MQTT over TLS | +| `waev` | mqtt.waev.app:443 | JWT (Ed25519) | WSS | +| `meshomatic` | us-east.meshomatic.net:443 | JWT (Ed25519) | WSS | +| `cascadiamesh` | cascadiamesh.ddns.net:1883 | None | MQTT (unencrypted) | | `custom` | User-configured | Username/Password | MQTT or WSS | | `none` | (disabled) | — | — | **Default Configuration:** - Slot 1: `analyzer-us` - Slot 2: `analyzer-eu` -- Slot 3: `none` +- Slots 3-6: `none` **Memory Limits:** -- With PSRAM: All 3 slots can be active simultaneously -- Without PSRAM: Maximum 2 active slots (each WSS/TLS connection requires ~40KB internal heap) +- With PSRAM: All slots can be active simultaneously +- Without PSRAM: Maximum 2 active TLS/WSS slots (each WSS/TLS connection requires ~40KB internal heap) - If more slots are configured than the device supports, excess slots show as `(inactive)` in `get mqtt.status` - Slot configurations are preserved in preferences — moving firmware to a PSRAM device activates all slots @@ -130,39 +150,41 @@ pio run -e Station_G2_repeater_observer_mqtt ### Build Flags - `WITH_MQTT_BRIDGE=1` - Enable MQTT bridge (required) +- `WITH_SNMP=1` - Enable SNMP agent (optional, see [MQTT_SNMP.md](MQTT_SNMP.md)) - `MQTT_DEBUG=1` - Enable debug logging (optional) - `MQTT_WIFI_TX_POWER` - WiFi TX power level (default: `WIFI_POWER_11dBm`) -- `MQTT_WIFI_POWER_SAVE_DEFAULT` - Default WiFi power save mode (0=min, 1=none, 2=max) +- ~~`MQTT_WIFI_POWER_SAVE_DEFAULT`~~ - Removed; all builds now default to `none` (no power save) ## Default Configuration -The MQTT bridge comes with the following defaults: -- **Origin**: Device name (set automatically) -- **IATA**: (must be configured) +The MQTT bridge comes with the following defaults for fresh installs: +- **Origin**: Device name (set automatically from `set name`) +- **IATA**: (blank — must be configured for Analyzer presets) - **Status Messages**: Enabled - **Packet Messages**: Enabled - **Raw Messages**: Disabled - **TX Messages**: Disabled (RX only by default) - **Status Interval**: 5 minutes (300000 ms) -- **Slot 1**: `analyzer-us` (mqtt-us-v1.letsmesh.net:443) -- **Slot 2**: `analyzer-eu` (mqtt-eu-v1.letsmesh.net:443) -- **Slot 3**: `none` (disabled) -- **WiFi SSID**: "ssid_here" (must be configured) -- **WiFi Password**: "password_here" (must be configured) -- **WiFi Power Save**: "min" (minimum power saving, balanced performance and power) -- **Timezone**: "America/Los_Angeles" (Pacific Time with DST support) -- **Timezone Offset**: -8 hours (fallback) +- **Slot 1**: `analyzer-us` +- **Slot 2**: `analyzer-eu` +- **Slots 3-6**: `none` (disabled) +- **WiFi SSID**: (blank — must be configured) +- **WiFi Password**: (blank — must be configured) +- **WiFi Power Save**: `none` (no power save) +- **Timezone**: (blank — uses UTC until configured) +- **Timezone Offset**: 0 (fallback, no offset) +- **Repeat (forwarding)**: On (set `repeat off` for receive-only observers) ## CLI Commands ### MQTT Slot Commands -Each slot (1-3) supports the following commands: +Each slot (1-6) supports the following commands: #### Get Commands - `get mqtt1.preset` - Get slot 1 preset name - `get mqtt2.preset` - Get slot 2 preset name -- `get mqtt3.preset` - Get slot 3 preset name +- `get mqttN.preset` - Get slot N preset name (N = 1-6) - `get mqttN.server` - Get custom server hostname for slot N - `get mqttN.port` - Get custom server port for slot N - `get mqttN.username` - Get custom username for slot N @@ -171,12 +193,15 @@ Each slot (1-3) supports the following commands: - `get mqttN.topic` - Get custom topic template for slot N #### Set Commands -- `set mqtt1.preset analyzer-us` - Set slot 1 to LetsMesh Analyzer US -- `set mqtt1.preset analyzer-eu` - Set slot 1 to LetsMesh Analyzer EU -- `set mqtt1.preset meshmapper` - Set slot 1 to MeshMapper -- `set mqtt1.preset meshrank` - Set slot 1 to MeshRank (requires token) -- `set mqtt1.preset custom` - Set slot 1 to custom broker (configure server/port/username/password) -- `set mqtt1.preset none` - Disable slot 1 +- `set mqttN.preset analyzer-us` - Set slot N to LetsMesh Analyzer US +- `set mqttN.preset analyzer-eu` - Set slot N to LetsMesh Analyzer EU +- `set mqttN.preset meshmapper` - Set slot N to MeshMapper +- `set mqttN.preset meshrank` - Set slot N to MeshRank (requires token) +- `set mqttN.preset waev` - Set slot N to Waev +- `set mqttN.preset meshomatic` - Set slot N to Meshomatic +- `set mqttN.preset cascadiamesh` - Set slot N to CascadiaMesh +- `set mqttN.preset custom` - Set slot N to custom broker (configure server/port/username/password) +- `set mqttN.preset none` - Disable slot N - `set mqttN.server ` - Set custom server hostname for slot N - `set mqttN.port ` - Set custom server port for slot N (1-65535) - `set mqttN.username ` - Set custom username for slot N @@ -229,7 +254,7 @@ When a slot's preset is `custom`, you can define a custom topic template using p If no custom topic is set, custom slots default to: `meshcore/{iata}/{device}/{type}` -**Note:** Topic templates only apply to `custom` preset slots. Built-in presets (analyzer-us, analyzer-eu, meshmapper, meshrank) always use their hardcoded topic format. +**Note:** Topic templates only apply to `custom` preset slots. Built-in presets (analyzer-us, analyzer-eu, meshmapper, meshrank, etc.) always use their hardcoded topic format. ### MQTT Shared Commands @@ -270,7 +295,7 @@ These settings apply across all MQTT slots: - `set wifi.pwd ` - Set WiFi password - `set wifi.powersave none|min|max` - Set WiFi power save mode - `none` - No power saving (best performance, highest power consumption) - - `min` - Minimum power saving (default, balanced performance and power) + - `min` - Minimum power saving (balanced performance and power) - `max` - Maximum power saving (lowest power consumption, may affect performance) ### Timezone Commands @@ -288,6 +313,26 @@ These settings apply across all MQTT slots: - **Common abbreviations**: `PDT`, `PST`, `MDT`, `MST`, `CDT`, `CST`, `EDT`, `EST`, `BST`, `GMT`, `CEST`, `CET` - **UTC offsets**: `UTC-8`, `UTC+5`, `+5`, `-8`, etc. +### Device & Radio Commands + +These are standard MeshCore commands, not MQTT-specific, but important for observer setup: + +#### Get Commands +- `get name` - Get device name +- `get repeat` - Get repeat (forwarding) status (on/off) +- `get freq` - Get radio frequency +- `get public.key` - Get device public key (for migration) + +#### Set Commands +- `set name ` - Set device name (also sets MQTT origin) +- `set repeat on|off` - Enable/disable packet forwarding (use `off` for receive-only observers) +- `set prv.key <64-hex-char-key>` - Restore private key (for migrating identity from another device) +- `set freq ` - Set radio frequency +- `set sf <5-12>` - Set LoRa spreading factor +- `set bw ` - Set LoRa bandwidth +- `set cr <5-8>` - Set LoRa coding rate +- `set tx ` - Set transmit power + ### Bridge Commands #### Get Commands @@ -298,6 +343,18 @@ These settings apply across all MQTT slots: - `set bridge.source rx|tx` - Set packet source (rx for received, tx for transmitted) - `set bridge.enabled on|off` - Enable/disable bridge +### SNMP Commands + +#### Get Commands +- `get snmp` - Get SNMP agent status (on/off) +- `get snmp.community` - Get SNMP community string + +#### Set Commands +- `set snmp on|off` - Enable/disable SNMP agent (restart required) +- `set snmp.community ` - Set SNMP community string (restart required, default: `public`) + +See [MQTT_SNMP.md](MQTT_SNMP.md) for full SNMP documentation. + ## Command Architecture The CLI commands are organized into two levels: @@ -309,7 +366,7 @@ The CLI commands are organized into two levels: ### Bridge-Specific Commands (`mqtt.*`, `mqttN.*`, `wifi.*`, `timezone.*`) **Implementation-specific settings** - These only apply to the MQTT bridge: -- `mqttN.*` - Per-slot MQTT broker configuration (N = 1, 2, or 3) +- `mqttN.*` - Per-slot MQTT broker configuration (N = 1-6) - `mqtt.*` - Shared MQTT settings (message types, origin, IATA, etc.) - `wifi.*` - WiFi connection settings for MQTT connectivity - `timezone.*` - Timezone configuration for accurate timestamps @@ -381,12 +438,13 @@ Minimal raw packet data for map integration. ## Key Features ### Slot-Based Preset System -- Up to 3 concurrent MQTT connections (with PSRAM), 2 without PSRAM -- Built-in presets for LetsMesh Analyzer (US/EU), MeshMapper, and MeshRank +- Up to 6 concurrent MQTT connections (with PSRAM), 2 without PSRAM +- Built-in presets for LetsMesh Analyzer (US/EU), MeshMapper, MeshRank, Waev, Meshomatic, and CascadiaMesh - Custom broker support with username/password auth and custom topic templates - JWT (Ed25519) authentication for preset brokers, token-in-topic for MeshRank - WSS (WebSocket Secure) and direct MQTT over TLS transport - Automatic reconnection with exponential backoff per slot +- Circuit breaker pattern with periodic probes for recovery from prolonged outages - JWT token buffers only allocated for JWT-auth slots (memory efficient) - Deferred construction: MQTTBridge is heap-allocated in `begin()` to avoid ESP32 static init crashes @@ -413,7 +471,7 @@ Minimal raw packet data for map integration. - Proper UTC system time handling ### Authentication -- **JWT Authentication**: Ed25519-signed tokens for secure MQTT authentication (used by all built-in presets) +- **JWT Authentication**: Ed25519-signed tokens for secure MQTT authentication (used by all built-in presets except meshrank and cascadiamesh) - **Username/Password**: Standard MQTT authentication for custom brokers - **Username Format** (JWT): `v1_{UPPERCASE_PUBLIC_KEY}` - **Automatic Token Renewal**: Tokens are renewed before expiration @@ -432,48 +490,72 @@ The migration happens automatically on first boot after firmware update. No manu ## First-Time Setup ### Prerequisites -- MeshCore device with MQTT bridge firmware flashed +- MeshCore device with observer MQTT firmware flashed - WiFi network credentials -- LoRa-capable device for configuration (repeater console) +- Serial console access (115200 baud) or repeater login via companion app -### Step 1: Configure WiFi +### Step 1: Configure Radio (after fresh flash/full erase) + +If this is a fresh flash, radio parameters must be set to match your mesh network: +``` +set freq 906.0 +set sf 12 +set cr 5 +set bw 250 +set tx 22 +``` + +### Step 2: Configure Device Identity +``` +set name MyObserver +set mqtt.iata SEA +``` + +If migrating from an existing device, restore the private key to keep the same identity: +``` +set prv.key +``` + +### Step 3: Configure WiFi ``` set wifi.ssid YourWiFiNetwork set wifi.pwd YourWiFiPassword reboot ``` -### Step 2: Configure Device Identity +### Step 4: Configure Timezone (optional) ``` -set mqtt.iata SEA -get mqtt.origin +set timezone America/New_York ``` -### Step 3: Verify Slot Configuration +Or use an offset as a fallback: +``` +set timezone.offset -5 +``` + +### Step 5: (Optional) Disable Repeating + +For receive-only observers (e.g., using a PCB antenna or in a location where repeating is not desired): +``` +set repeat off +``` + +### Step 6: Verify Slot Configuration ``` get mqtt1.preset # Should show: analyzer-us get mqtt2.preset # Should show: analyzer-eu get mqtt3.preset # Should show: none ``` -### Step 4: (Optional) Add MeshMapper +### Step 7: (Optional) Add Additional Presets ``` set mqtt3.preset meshmapper ``` -### Step 5: (Optional) Configure Custom Broker +### Step 8: Verify Connection ``` -set mqtt3.preset custom -set mqtt3.server your-broker.example.com -set mqtt3.port 1883 -set mqtt3.username your-username -set mqtt3.password your-password -``` - -### Step 6: Verify Connection -``` -set bridge.source rx get bridge.enabled +get bridge.source get mqtt.status get wifi.status ``` @@ -492,6 +574,8 @@ reboot ``` get bridge.enabled set bridge.enabled on +get bridge.source # Should be "rx" +set bridge.source rx get mqtt.status # Check per-slot connection status get mqtt1.preset # Verify slots are configured get mqtt.iata # IATA must be set for Analyzer presets @@ -505,6 +589,10 @@ set timezone EST # Abbreviation set timezone UTC-5 # UTC offset ``` +## SNMP Monitoring + +Observer nodes include an optional SNMP v2c agent that exposes radio stats, MQTT connectivity, memory usage, and network information to standard monitoring tools. See [MQTT_SNMP.md](MQTT_SNMP.md) for setup and OID reference. + ## Dependencies - **PsychicMqttClient**: MQTT client library (supports WSS and direct MQTT) @@ -514,3 +602,4 @@ set timezone UTC-5 # UTC offset - **WiFi**: ESP32 WiFi functionality - **Ed25519**: Cryptographic library for JWT token signing - **JWTHelper**: Custom JWT token generation for device authentication +- **SNMP_Agent**: Optional SNMPv2c agent (0neblock/SNMP_Agent, observer builds only) diff --git a/MQTT_SNMP.md b/MQTT_SNMP.md new file mode 100644 index 00000000..82740092 --- /dev/null +++ b/MQTT_SNMP.md @@ -0,0 +1,149 @@ +# SNMP Agent for MeshCore Observer Firmware + +This document describes the optional SNMP v2c agent that allows network monitoring tools to poll MeshCore observer nodes for health and performance metrics. + +## Overview + +The SNMP agent exposes radio statistics, MQTT connectivity status, memory usage, and network information via standard SNMP v2c GET/GETNEXT/GETBULK operations. It is designed for use with monitoring tools such as Nagios, Zabbix, LibreNMS, PRTG, or any SNMP-capable system. + +- **Protocol**: SNMPv2c (read-only, no SET support) +- **Port**: UDP 161 (standard SNMP) +- **Library**: [0neblock/SNMP_Agent](https://github.com/0neblock/Arduino_SNMP) +- **Disabled by default** -- must be explicitly enabled via CLI + +## Quick Start + +```bash +# Enable SNMP +set snmp on +reboot + +# Verify from a host on the same network +snmpwalk -v2c -c public 1.3.6.1.4.1.99999 +``` + +## CLI Commands + +| Command | Description | +|---|---| +| `get snmp` | Show SNMP agent status (`on` / `off`) | +| `set snmp on` | Enable SNMP agent (restart required) | +| `set snmp off` | Disable SNMP agent (restart required) | +| `get snmp.community` | Show SNMP community string | +| `set snmp.community ` | Set community string (restart required, default: `public`) | + +## OID Reference + +All OIDs are under the private enterprise subtree `1.3.6.1.4.1.99999` (`.iso.org.dod.internet.private.enterprises.99999`). + +### System (.1.x.0) + +| OID | Type | Description | +|---|---|---| +| `.1.1.0` | INTEGER | System uptime (seconds) | +| `.1.2.0` | STRING | Firmware version | +| `.1.3.0` | STRING | Node name | + +### Radio (.2.x.0) + +| OID | Type | Description | +|---|---|---| +| `.2.1.0` | INTEGER | Total packets received | +| `.2.2.0` | INTEGER | Total packets sent | +| `.2.3.0` | INTEGER | Receive errors (CRC failures, etc.) | +| `.2.4.0` | INTEGER | Noise floor (dBm) | +| `.2.5.0` | INTEGER | Last RSSI (dBm) | +| `.2.6.0` | INTEGER | Last SNR (dB x 4) | +| `.2.7.0` | INTEGER | Flood packets sent | +| `.2.8.0` | INTEGER | Direct packets sent | +| `.2.9.0` | INTEGER | Flood packets received | +| `.2.10.0` | INTEGER | Direct packets received | +| `.2.11.0` | INTEGER | Total air time (seconds) | + +### MQTT (.3.x.0) + +| OID | Type | Description | +|---|---|---| +| `.3.1.0` | INTEGER | Connected MQTT slot count | +| `.3.2.0` | INTEGER | Packet queue depth | +| `.3.3.0` | INTEGER | Skipped publishes (memory pressure) | + +### Memory (.4.x.0) + +| OID | Type | Description | +|---|---|---| +| `.4.1.0` | INTEGER | Free heap (bytes) | +| `.4.2.0` | INTEGER | Max allocatable block (bytes) | +| `.4.3.0` | INTEGER | Free internal RAM (bytes) | +| `.4.4.0` | INTEGER | Free PSRAM (bytes, 0 if no PSRAM) | + +### Network (.5.x.0) + +| OID | Type | Description | +|---|---|---| +| `.5.1.0` | INTEGER | WiFi RSSI (dBm) | + +## Example Output + +``` +$ snmpwalk -v2c -c public 192.168.50.95 1.3.6.1.4.1.99999 + +SNMPv2-SMI::enterprises.99999.1.1.0 = INTEGER: 18 # uptime 18s +SNMPv2-SMI::enterprises.99999.1.2.0 = STRING: "v1.14.1" # firmware +SNMPv2-SMI::enterprises.99999.1.3.0 = STRING: "Heltec Repeater" +SNMPv2-SMI::enterprises.99999.2.1.0 = INTEGER: 3 # 3 packets received +SNMPv2-SMI::enterprises.99999.2.4.0 = INTEGER: -96 # noise floor -96 dBm +SNMPv2-SMI::enterprises.99999.2.5.0 = INTEGER: -25 # last RSSI -25 dBm +SNMPv2-SMI::enterprises.99999.4.1.0 = INTEGER: 199612 # ~195 KB free heap +SNMPv2-SMI::enterprises.99999.4.2.0 = INTEGER: 188404 # ~184 KB max alloc +SNMPv2-SMI::enterprises.99999.5.1.0 = INTEGER: -22 # WiFi RSSI -22 dBm +``` + +## Architecture + +The SNMP agent runs inside the existing MQTT FreeRTOS task on Core 0 (the WiFi/network core). No additional task is created. + +- **Radio stats** are pushed from Core 1 (mesh task) every 2 seconds via `updateRadioStats()` +- **MQTT stats** are updated from Core 0 each MQTT task loop iteration +- **Memory and WiFi stats** are read directly in the SNMP loop on Core 0 + +The agent starts automatically once WiFi connects and `snmp_enabled = 1` in prefs. It adds approximately 2-3 KB RAM overhead plus per-OID handler storage. + +## Build Configuration + +SNMP is enabled at compile time with the `WITH_SNMP=1` build flag and included in observer firmware targets: + +- `heltec_v3_repeater_observer_mqtt` +- `heltec_v4_repeater_observer_mqtt` +- `Station_G2_repeater_observer_mqtt` + +To add SNMP to another observer variant, add the following to its `platformio.ini`: + +```ini +build_flags = + ... + -D WITH_SNMP=1 + +build_src_filter = + ... + + + +lib_deps = + ... + 0neblock/SNMP_Agent +``` + +## Files + +| File | Description | +|---|---| +| `src/helpers/SNMPAgent.h` | SNMP agent wrapper class declaration | +| `src/helpers/SNMPAgent.cpp` | OID registration and stats update implementation | +| `src/helpers/CommonCLI.h` | `NodePrefs` struct (`snmp_enabled`, `snmp_community` fields) | +| `src/helpers/CommonCLI.cpp` | CLI command parsing and prefs persistence | +| `src/helpers/bridges/MQTTBridge.cpp` | SNMP loop integration in `mqttTaskLoop()` | +| `examples/simple_repeater/MyMesh.cpp` | Radio stats push and agent initialization | + +## Private Enterprise Number + +The OIDs currently use a temporary unregistered enterprise number (`99999`). A proper Private Enterprise Number (PEN) can be registered with IANA at no cost at https://pen.iana.org/pen/PenApplication.page. diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 48adfb04..aa05bdf0 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -919,6 +919,9 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc _prefs.flood_advert_interval = 12; // 12 hours _prefs.flood_max = 64; _prefs.interference_threshold = 0; // disabled +#ifdef WITH_MQTT_BRIDGE + _prefs.agc_reset_interval = 7; // 28 seconds (secs/4) — prevents AGC drift on long-running observers +#endif // bridge defaults _prefs.bridge_enabled = 1; // enabled @@ -929,6 +932,10 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc StrHelper::strncpy(_prefs.bridge_secret, "LVSITANOS", sizeof(_prefs.bridge_secret)); + // SNMP defaults + _prefs.snmp_enabled = 0; + StrHelper::strncpy(_prefs.snmp_community, "public", sizeof(_prefs.snmp_community)); + // GPS defaults _prefs.gps_enabled = 0; _prefs.gps_interval = 0; @@ -1010,6 +1017,13 @@ void MyMesh::begin(FILESYSTEM *fs) { #ifdef WITH_MQTT_BRIDGE // Set stats sources for automatic stats collection bridge->setStatsSources(this, _radio, _cli.getBoard(), _ms); +#ifdef WITH_SNMP + if (_prefs.snmp_enabled) { + _snmp_agent.setNodeName(_prefs.node_name); + _snmp_agent.setFirmwareVersion(getFirmwareVer()); + bridge->setSNMPAgent(&_snmp_agent); + } +#endif #endif bridge->begin(); @@ -1434,6 +1448,25 @@ void MyMesh::loop() { uint32_t now = millis(); uptime_millis += now - last_millis; last_millis = now; + +#ifdef WITH_SNMP + // Push radio stats to SNMP agent every 2 seconds + if (_snmp_agent.isRunning()) { + static unsigned long last_snmp_stats = 0; + if (now - last_snmp_stats >= 2000) { + last_snmp_stats = now; + _snmp_agent.updateRadioStats( + radio_driver.getPacketsRecv(), radio_driver.getPacketsSent(), + radio_driver.getPacketsRecvErrors(), + (int16_t)_radio->getNoiseFloor(), + (int16_t)radio_driver.getLastRSSI(), + (int16_t)(radio_driver.getLastSNR() * 4), + getNumSentFlood(), getNumSentDirect(), + getNumRecvFlood(), getNumRecvDirect(), + getTotalAirTime() / 1000, uptime_millis / 1000); + } + } +#endif } // To check if there is pending work diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index d8625bd9..d957f6d9 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -29,6 +29,10 @@ #define WITH_BRIDGE #endif +#ifdef WITH_SNMP +#include "helpers/SNMPAgent.h" +#endif + #include #include #include @@ -122,6 +126,9 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { #elif defined(WITH_MQTT_BRIDGE) MQTTBridge* bridge; #endif +#ifdef WITH_SNMP + MeshSNMPAgent _snmp_agent; +#endif void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); void sendNodeDiscoverReq(); diff --git a/src/helpers/CommonCLI.cpp b/src/helpers/CommonCLI.cpp index c0ce70f1..4d4a0ff1 100644 --- a/src/helpers/CommonCLI.cpp +++ b/src/helpers/CommonCLI.cpp @@ -159,7 +159,9 @@ void CommonCLI::loadPrefsInt(FILESYSTEM* fs, const char* filename) { remaining -= to_read; } file.read((uint8_t *)&_prefs->rx_boosted_gain, sizeof(_prefs->rx_boosted_gain)); // 290 - // next: 291 + file.read((uint8_t *)&_prefs->snmp_enabled, sizeof(_prefs->snmp_enabled)); // 291 + file.read((uint8_t *)&_prefs->snmp_community, sizeof(_prefs->snmp_community)); // 292 + // next: 316 // sanitise bad pref values _prefs->rx_delay_base = constrain(_prefs->rx_delay_base, 0, 20.0f); @@ -189,6 +191,8 @@ void CommonCLI::loadPrefsInt(FILESYSTEM* fs, const char* filename) { _prefs->advert_loc_policy = constrain(_prefs->advert_loc_policy, 0, 2); _prefs->rx_boosted_gain = constrain(_prefs->rx_boosted_gain, 0, 1); // boolean + _prefs->snmp_enabled = constrain(_prefs->snmp_enabled, 0, 1); + _prefs->snmp_community[sizeof(_prefs->snmp_community) - 1] = '\0'; // ensure null terminated file.close(); } @@ -274,7 +278,9 @@ void CommonCLI::savePrefs(FILESYSTEM* fs) { remaining -= to_write; } file.write((uint8_t *)&_prefs->rx_boosted_gain, sizeof(_prefs->rx_boosted_gain)); // 290 - // next: 291 + file.write((uint8_t *)&_prefs->snmp_enabled, sizeof(_prefs->snmp_enabled)); // 291 + file.write((uint8_t *)&_prefs->snmp_community, sizeof(_prefs->snmp_community)); // 292 + // next: 316 file.close(); } @@ -304,11 +310,7 @@ static void setMQTTPrefsDefaults(MQTTPrefs* prefs) { strncpy(prefs->mqtt_slot_preset[i], "none", sizeof(prefs->mqtt_slot_preset[i]) - 1); prefs->mqtt_slot_preset[i][sizeof(prefs->mqtt_slot_preset[i]) - 1] = '\0'; } - #ifdef MQTT_WIFI_POWER_SAVE_DEFAULT - prefs->wifi_power_save = MQTT_WIFI_POWER_SAVE_DEFAULT; // 0=min, 1=none, 2=max - #else - prefs->wifi_power_save = 0; // Default to WIFI_PS_MIN_MODEM (0=min) - #endif + prefs->wifi_power_save = 1; // Default to none (0=min, 1=none, 2=max) // String fields are already zero-initialized by memset } @@ -714,6 +716,10 @@ void CommonCLI::handleCommand(uint32_t sender_timestamp, const char* command, ch } else { strcpy(reply, "> strict"); } + } else if (memcmp(config, "snmp.community", 14) == 0) { + sprintf(reply, "> %s", _prefs->snmp_community); + } else if (memcmp(config, "snmp", 4) == 0 && (config[4] == '\0' || config[4] == '\n' || config[4] == '\r')) { + strcpy(reply, _prefs->snmp_enabled ? "> on" : "> off"); } else if (memcmp(config, "tx", 2) == 0 && (config[2] == 0 || config[2] == ' ')) { sprintf(reply, "> %d", (int32_t) _prefs->tx_power_dbm); } else if (memcmp(config, "freq", 4) == 0) { @@ -1104,6 +1110,14 @@ void CommonCLI::handleCommand(uint32_t sender_timestamp, const char* command, ch savePrefs(); strcpy(reply, "OK"); } + } else if (memcmp(config, "snmp.community ", 15) == 0) { + StrHelper::strncpy(_prefs->snmp_community, &config[15], sizeof(_prefs->snmp_community)); + savePrefs(); + strcpy(reply, "OK - restart to apply"); + } else if (memcmp(config, "snmp ", 5) == 0) { + _prefs->snmp_enabled = memcmp(&config[5], "on", 2) == 0; + savePrefs(); + strcpy(reply, "OK - restart to apply"); } else if (memcmp(config, "tx ", 3) == 0) { _prefs->tx_power_dbm = atoi(&config[3]); savePrefs(); diff --git a/src/helpers/CommonCLI.h b/src/helpers/CommonCLI.h index 209b13dc..982b6eab 100644 --- a/src/helpers/CommonCLI.h +++ b/src/helpers/CommonCLI.h @@ -72,7 +72,7 @@ struct NodePrefs { // persisted to file // WiFi settings char wifi_ssid[32]; // WiFi SSID char wifi_password[64]; // WiFi password - uint8_t wifi_power_save; // WiFi power save mode: 0=min, 1=none, 2=max (default: 0=min) + uint8_t wifi_power_save; // WiFi power save mode: 0=min, 1=none, 2=max (default: 1=none) // Timezone settings char timezone_string[32]; // Timezone string (e.g., "America/Los_Angeles") @@ -96,6 +96,10 @@ struct NodePrefs { // persisted to file char mqtt_slot_topic[MAX_MQTT_SLOTS][96]; // Per-slot custom topic template (custom preset only) uint8_t loop_detect; + + // SNMP settings (optional, only used when WITH_SNMP is defined) + uint8_t snmp_enabled; // boolean: 0=off, 1=on + char snmp_community[24]; // community string (default "public") }; #ifdef WITH_MQTT_BRIDGE @@ -137,7 +141,7 @@ struct MQTTPrefs { // WiFi settings char wifi_ssid[32]; // WiFi SSID char wifi_password[64]; // WiFi password - uint8_t wifi_power_save; // WiFi power save mode: 0=min, 1=none, 2=max (default: 0=min) + uint8_t wifi_power_save; // WiFi power save mode: 0=min, 1=none, 2=max (default: 1=none) // Timezone settings char timezone_string[32]; // Timezone string (e.g., "America/Los_Angeles") diff --git a/src/helpers/SNMPAgent.cpp b/src/helpers/SNMPAgent.cpp new file mode 100644 index 00000000..a70716f5 --- /dev/null +++ b/src/helpers/SNMPAgent.cpp @@ -0,0 +1,123 @@ +#ifdef WITH_SNMP + +#include "SNMPAgent.h" +#include + +#define SNMP_PORT 161 + +MeshSNMPAgent::MeshSNMPAgent() + : _snmp("public"), + _running(false), + _uptime_secs(0), + _packets_recv(0), _packets_sent(0), _recv_errors(0), + _noise_floor(0), _last_rssi(0), _last_snr(0), + _sent_flood(0), _sent_direct(0), _recv_flood(0), _recv_direct(0), + _total_air_time_secs(0), + _mqtt_connected_slots(0), _mqtt_queue_depth(0), _mqtt_skipped_publishes(0), + _free_heap(0), _max_alloc(0), _internal_free(0), _psram_free(0), + _wifi_rssi(0) +{ + _firmware_version[0] = '\0'; + _node_name[0] = '\0'; +} + +void MeshSNMPAgent::begin(const char* community) { + if (_running) return; + + _snmp = SNMPAgent(community); + _snmp.setUDP(&_udp); + _snmp.begin(); + + // System group (.1.x.0) + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".1.1.0", &_uptime_secs); + _snmp.addReadOnlyStaticStringHandler(MESHCORE_OID_BASE ".1.2.0", _firmware_version, sizeof(_firmware_version)); + _snmp.addReadOnlyStaticStringHandler(MESHCORE_OID_BASE ".1.3.0", _node_name, sizeof(_node_name)); + + // Radio group (.2.x.0) + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.1.0", &_packets_recv); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.2.0", &_packets_sent); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.3.0", &_recv_errors); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.4.0", &_noise_floor); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.5.0", &_last_rssi); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.6.0", &_last_snr); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.7.0", &_sent_flood); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.8.0", &_sent_direct); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.9.0", &_recv_flood); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.10.0", &_recv_direct); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".2.11.0", &_total_air_time_secs); + + // MQTT group (.3.x.0) + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".3.1.0", &_mqtt_connected_slots); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".3.2.0", &_mqtt_queue_depth); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".3.3.0", &_mqtt_skipped_publishes); + + // Memory group (.4.x.0) + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".4.1.0", &_free_heap); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".4.2.0", &_max_alloc); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".4.3.0", &_internal_free); + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".4.4.0", &_psram_free); + + // Network group (.5.x.0) + _snmp.addIntegerHandler(MESHCORE_OID_BASE ".5.1.0", &_wifi_rssi); + + _snmp.sortHandlers(); + _running = true; +} + +void MeshSNMPAgent::loop() { + if (!_running) return; + + // Update memory and network stats locally (we're on Core 0 with WiFi) + _free_heap = (int)ESP.getFreeHeap(); + _max_alloc = (int)ESP.getMaxAllocHeap(); + _internal_free = (int)heap_caps_get_free_size(MALLOC_CAP_INTERNAL); +#ifdef BOARD_HAS_PSRAM + _psram_free = (int)heap_caps_get_free_size(MALLOC_CAP_SPIRAM); +#else + _psram_free = 0; +#endif + + if (WiFi.isConnected()) { + _wifi_rssi = (int)WiFi.RSSI(); + } + + _snmp.loop(); +} + +void MeshSNMPAgent::updateRadioStats( + uint32_t packets_recv, uint32_t packets_sent, uint32_t recv_errors, + int16_t noise_floor, int16_t last_rssi, int16_t last_snr, + uint32_t sent_flood, uint32_t sent_direct, + uint32_t recv_flood, uint32_t recv_direct, + uint32_t total_air_time_secs, uint32_t uptime_secs) { + _packets_recv = (int)packets_recv; + _packets_sent = (int)packets_sent; + _recv_errors = (int)recv_errors; + _noise_floor = (int)noise_floor; + _last_rssi = (int)last_rssi; + _last_snr = (int)last_snr; + _sent_flood = (int)sent_flood; + _sent_direct = (int)sent_direct; + _recv_flood = (int)recv_flood; + _recv_direct = (int)recv_direct; + _total_air_time_secs = (int)total_air_time_secs; + _uptime_secs = (int)uptime_secs; +} + +void MeshSNMPAgent::updateMQTTStats(int connected_slots, int queue_depth, int skipped_publishes) { + _mqtt_connected_slots = connected_slots; + _mqtt_queue_depth = queue_depth; + _mqtt_skipped_publishes = skipped_publishes; +} + +void MeshSNMPAgent::setNodeName(const char* name) { + strncpy(_node_name, name, sizeof(_node_name) - 1); + _node_name[sizeof(_node_name) - 1] = '\0'; +} + +void MeshSNMPAgent::setFirmwareVersion(const char* version) { + strncpy(_firmware_version, version, sizeof(_firmware_version) - 1); + _firmware_version[sizeof(_firmware_version) - 1] = '\0'; +} + +#endif // WITH_SNMP diff --git a/src/helpers/SNMPAgent.h b/src/helpers/SNMPAgent.h new file mode 100644 index 00000000..c1bfe0f5 --- /dev/null +++ b/src/helpers/SNMPAgent.h @@ -0,0 +1,79 @@ +#pragma once + +#ifdef WITH_SNMP + +#include +#include +#include + +// Temporary private enterprise OID base — replace with registered PEN when available. +// All MeshCore OIDs live under this subtree. +#define MESHCORE_OID_BASE ".1.3.6.1.4.1.99999" + +// OID layout: +// .1.x.0 = system (uptime, version, node name) +// .2.x.0 = radio (packets, RSSI, SNR, noise floor, air time) +// .3.x.0 = mqtt (connected slots, queue depth, skipped publishes) +// .4.x.0 = memory (free heap, max alloc, internal free, PSRAM free) +// .5.x.0 = network (WiFi RSSI) + +class MeshSNMPAgent { +public: + MeshSNMPAgent(); + void begin(const char* community); + void loop(); + + // Called from the mesh task (Core 1) to push fresh stats into SNMP-visible variables. + // Copies are atomic for 32-bit aligned ints on ESP32, so no mutex needed. + void updateRadioStats(uint32_t packets_recv, uint32_t packets_sent, uint32_t recv_errors, + int16_t noise_floor, int16_t last_rssi, int16_t last_snr, + uint32_t sent_flood, uint32_t sent_direct, + uint32_t recv_flood, uint32_t recv_direct, + uint32_t total_air_time_secs, uint32_t uptime_secs); + + void updateMQTTStats(int connected_slots, int queue_depth, int skipped_publishes); + + void setNodeName(const char* name); + void setFirmwareVersion(const char* version); + + bool isRunning() const { return _running; } + +private: + WiFiUDP _udp; + SNMPAgent _snmp; + bool _running; + + // System OIDs + int _uptime_secs; + char _firmware_version[32]; + char _node_name[32]; + + // Radio OIDs + int _packets_recv; + int _packets_sent; + int _recv_errors; + int _noise_floor; + int _last_rssi; + int _last_snr; + int _sent_flood; + int _sent_direct; + int _recv_flood; + int _recv_direct; + int _total_air_time_secs; + + // MQTT OIDs + int _mqtt_connected_slots; + int _mqtt_queue_depth; + int _mqtt_skipped_publishes; + + // Memory OIDs (updated in loop() since we're on Core 0 with WiFi) + int _free_heap; + int _max_alloc; + int _internal_free; + int _psram_free; + + // Network OIDs + int _wifi_rssi; +}; + +#endif // WITH_SNMP diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index f4e2e5fa..725b3214 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -4,6 +4,10 @@ #include #include +#ifdef WITH_SNMP +#include "../SNMPAgent.h" +#endif + #ifdef ESP_PLATFORM #include #include @@ -168,8 +172,12 @@ MQTTBridge::MQTTBridge(NodePrefs *prefs, mesh::PacketManager *mgr, mesh::RTCCloc _cached_has_connected_slots(false), _last_memory_check(0), _skipped_publishes(0), _last_fragmentation_recovery(0), _fragmentation_pressure_since(0), _last_critical_check_run(0), - _last_no_broker_log(0), _last_config_warning(0), + _last_no_broker_log(0), _queue_disconnected_since(0), _all_tripped_since(0), + _last_config_warning(0), _dispatcher(nullptr), _radio(nullptr), _board(nullptr), _ms(nullptr), +#ifdef WITH_SNMP + _snmp_agent(nullptr), +#endif _last_wifi_check(0), _last_wifi_status(WL_DISCONNECTED), _wifi_status_initialized(false), _wifi_disconnected_time(0), _last_wifi_reconnect_attempt(0), _wifi_reconnect_backoff_attempt(0) #ifdef ESP_PLATFORM @@ -673,6 +681,25 @@ void MQTTBridge::mqttTaskLoop() { // Process packet queue processPacketQueue(); +#ifdef WITH_SNMP + // SNMP agent loop — process incoming UDP requests + if (_snmp_agent) { + if (!_snmp_agent->isRunning() && WiFi.isConnected() && _prefs->snmp_enabled) { + _snmp_agent->begin(_prefs->snmp_community); + MQTT_DEBUG_PRINTLN("SNMP agent started on port 161 (community: %s)", _prefs->snmp_community); + } + if (_snmp_agent->isRunning()) { + // Update MQTT stats from this core + int connected = 0; + for (int i = 0; i < MAX_MQTT_SLOTS; i++) { + if (_slots[i].enabled && _slots[i].connected) connected++; + } + _snmp_agent->updateMQTTStats(connected, _queue_count, _skipped_publishes); + _snmp_agent->loop(); + } + } +#endif + // Periodic configuration check (throttled to avoid spam) checkConfigurationMismatch(); @@ -937,6 +964,9 @@ void MQTTBridge::maintainSlotConnections() { // Only allow one reconnect attempt per maintenance cycle to avoid // multiple simultaneous TLS handshakes blocking the network stack bool reconnect_attempted_this_cycle = false; + // Only allow one full teardown+setup per cycle to limit heap fragmentation + // when multiple slots fail simultaneously + bool teardown_attempted_this_cycle = false; for (int i = 0; i < MAX_MQTT_SLOTS; i++) { if (!_slots[i].enabled || !_slots[i].client) continue; @@ -946,11 +976,11 @@ void MQTTBridge::maintainSlotConnections() { continue; } - maintainSlotConnection(i, now_millis, current_time, time_synced, reconnect_attempted_this_cycle); + maintainSlotConnection(i, now_millis, current_time, time_synced, reconnect_attempted_this_cycle, teardown_attempted_this_cycle); } } -void MQTTBridge::maintainSlotConnection(int index, unsigned long now_millis, unsigned long current_time, bool time_synced, bool& reconnect_attempted) { +void MQTTBridge::maintainSlotConnection(int index, unsigned long now_millis, unsigned long current_time, bool time_synced, bool& reconnect_attempted, bool& teardown_attempted) { MQTTSlot& slot = _slots[index]; if (slot.connected) { @@ -1037,13 +1067,23 @@ void MQTTBridge::maintainSlotConnection(int index, unsigned long now_millis, uns } slot.client->connect(); } else { - // Token expired or near expiry — full teardown for fresh TLS + new token - bool saved_tripped = slot.circuit_breaker_tripped; - MQTT_DEBUG_PRINTLN("MQTT%d token expired/near expiry, full teardown+setup for probe", index + 1); - teardownSlot(index); - setupSlot(index); - _slots[index].circuit_breaker_tripped = saved_tripped; - _slots[index].last_reconnect_attempt = now_millis; + // Token expired — regenerate token but avoid teardown+setup + // which would allocate new TLS context on potentially fragmented heap + if (slot.client) { + if (createSlotAuthToken(index)) { + slot.client->setCredentials(_jwt_username, slot.auth_token); + MQTT_DEBUG_PRINTLN("MQTT%d circuit breaker probe (regenerated expired token)", index + 1); + } + slot.client->connect(); + } else { + // Client was destroyed — must do full setup + bool saved_tripped = slot.circuit_breaker_tripped; + MQTT_DEBUG_PRINTLN("MQTT%d circuit breaker probe (full setup, no client)", index + 1); + teardownSlot(index); + setupSlot(index); + _slots[index].circuit_breaker_tripped = saved_tripped; + _slots[index].last_reconnect_attempt = now_millis; + } } } else { slot.client->connect(); @@ -1094,6 +1134,12 @@ void MQTTBridge::maintainSlotConnection(int index, unsigned long now_millis, uns slot.client->reconnect(); } else { // Full teardown: token expired/near expiry, or lightweight reconnect failed (backoff 3+) + if (teardown_attempted) { + // Defer to next cycle to limit heap fragmentation from simultaneous teardowns + slot.last_reconnect_attempt = now_millis; + return; + } + teardown_attempted = true; uint8_t saved_backoff = slot.reconnect_backoff; uint8_t saved_failures = slot.max_backoff_failures; MQTT_DEBUG_PRINTLN("MQTT%d full teardown+setup for reconnect (backoff %d)", index + 1, saved_backoff); @@ -1110,6 +1156,11 @@ void MQTTBridge::maintainSlotConnection(int index, unsigned long now_millis, uns slot.client->reconnect(); } else { // Full teardown after lightweight reconnect failed (backoff 3+) + if (teardown_attempted) { + slot.last_reconnect_attempt = now_millis; + return; + } + teardown_attempted = true; uint8_t saved_backoff = slot.reconnect_backoff; uint8_t saved_failures = slot.max_backoff_failures; MQTT_DEBUG_PRINTLN("MQTT%d full teardown+setup for reconnect (backoff %d)", index + 1, saved_backoff); @@ -1769,6 +1820,7 @@ void MQTTBridge::processPacketQueue() { _queue_count = uxQueueMessagesWaiting(_packet_queue_handle); if (_queue_count == 0) { + _queue_disconnected_since = 0; return; } @@ -1782,10 +1834,21 @@ void MQTTBridge::processPacketQueue() { MQTT_DEBUG_PRINTLN("Queue has %d packets but no slots connected", _queue_count); _last_no_broker_log = now; } + // Flush stale packets after extended disconnect + if (_queue_disconnected_since == 0) { + _queue_disconnected_since = now; + } else if ((now - _queue_disconnected_since) >= QUEUE_STALE_MS) { + QueuedPacket discard; + while (xQueueReceive(_packet_queue_handle, &discard, 0) == pdTRUE) {} + _queue_count = 0; + MQTT_DEBUG_PRINTLN("Flushed stale packet queue after %lu ms disconnected", now - _queue_disconnected_since); + _queue_disconnected_since = now; + } } return; } + _queue_disconnected_since = 0; _last_no_broker_log = 0; // Adaptive drain: burst-process when queue has backlog, gentle otherwise @@ -2273,6 +2336,31 @@ void MQTTBridge::runCriticalMemoryCheckAndRecovery() { MQTT_DEBUG_PRINTLN("Fragmentation recovery: recreating MQTT clients (max_alloc=%d, pressure %lu min)", (int)max_alloc, (unsigned long)(required_window_ms / 60000)); recreateMqttClientsForFragmentationRecovery(); } + + // Last resort: if ALL enabled slots have circuit breakers tripped for >1 hour, + // heap is likely too fragmented for TLS to ever succeed — reboot. + bool all_tripped = true; + int enabled_count = 0; + for (int i = 0; i < MAX_MQTT_SLOTS; i++) { + if (_slots[i].enabled) { + enabled_count++; + if (!_slots[i].circuit_breaker_tripped) { + all_tripped = false; + break; + } + } + } + if (enabled_count > 0 && all_tripped) { + if (_all_tripped_since == 0) { + _all_tripped_since = now; + } else if ((now - _all_tripped_since) >= 3600000UL) { + MQTT_DEBUG_PRINTLN("All MQTT slots circuit-breaker tripped for >1 hour. Restarting ESP."); + delay(100); + ESP.restart(); + } + } else { + _all_tripped_since = 0; + } } #endif diff --git a/src/helpers/bridges/MQTTBridge.h b/src/helpers/bridges/MQTTBridge.h index 643a7aca..94696ab6 100644 --- a/src/helpers/bridges/MQTTBridge.h +++ b/src/helpers/bridges/MQTTBridge.h @@ -10,6 +10,10 @@ #include "helpers/JWTHelper.h" #include "helpers/MQTTPresets.h" +#ifdef WITH_SNMP +class MeshSNMPAgent; // Forward declaration +#endif + #ifdef ESP_PLATFORM #include #include @@ -186,6 +190,17 @@ private: // Cached connection status (updated in callbacks to avoid redundant checks) bool _cached_has_connected_slots; + // Queue staleness tracking + unsigned long _queue_disconnected_since; // 0 = has connected slots + static const unsigned long QUEUE_STALE_MS = 300000UL; // Flush queue after 5 min disconnected + + // Recovery: restart ESP after prolonged total failure + unsigned long _all_tripped_since; // 0 = not all tripped + +#ifdef WITH_SNMP + MeshSNMPAgent* _snmp_agent; +#endif + // Throttle logging unsigned long _last_no_broker_log; static const unsigned long NO_BROKER_LOG_INTERVAL = 30000; // Log every 30 seconds max @@ -216,7 +231,7 @@ private: void setupSlot(int index); // Create/destroy client for a slot based on its preset void teardownSlot(int index); // Disconnect and free slot resources void maintainSlotConnections(); // Maintain all slot connections (token renewal, reconnect) - void maintainSlotConnection(int index, unsigned long now_millis, unsigned long current_time, bool time_synced, bool& reconnect_attempted); + void maintainSlotConnection(int index, unsigned long now_millis, unsigned long current_time, bool time_synced, bool& reconnect_attempted, bool& teardown_attempted); bool createSlotAuthToken(int index); // Create/renew JWT token for a slot bool publishToSlot(int index, const char* topic, const char* payload, bool retained = false); bool publishToAllSlots(const char* topic, const char* payload, bool retained = false); @@ -304,6 +319,10 @@ public: void setStatsSources(mesh::Dispatcher* dispatcher, mesh::Radio* radio, mesh::MainBoard* board, mesh::MillisecondClock* ms); + +#ifdef WITH_SNMP + void setSNMPAgent(MeshSNMPAgent* agent) { _snmp_agent = agent; } +#endif }; #endif diff --git a/variants/heltec_v3/platformio.ini b/variants/heltec_v3/platformio.ini index 68791843..28c7e26a 100644 --- a/variants/heltec_v3/platformio.ini +++ b/variants/heltec_v3/platformio.ini @@ -126,7 +126,7 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_19_5dBm - -D MQTT_WIFI_POWER_SAVE_DEFAULT=1 + -D WITH_SNMP=1 # -D WIFI_SSID='"ssid"' # -D WIFI_PWD='"password"' # -D MQTT_SERVER='"your-mqtt-broker.com"' @@ -137,6 +137,7 @@ build_src_filter = ${Heltec_lora32_v3.build_src_filter} + + + + + + +<../examples/simple_repeater> lib_deps = @@ -146,6 +147,7 @@ lib_deps = bblanchon/ArduinoJson arduino-libraries/NTPClient JChristensen/Timezone + 0neblock/SNMP_Agent paulstoffregen/Time@1.6.1 [env:Heltec_v3_room_server] @@ -186,7 +188,6 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_19_5dBm - -D MQTT_WIFI_POWER_SAVE_DEFAULT=1 build_src_filter = ${Heltec_lora32_v3.build_src_filter} + + diff --git a/variants/heltec_v4/platformio.ini b/variants/heltec_v4/platformio.ini index f5bb4ccc..ec1e6199 100644 --- a/variants/heltec_v4/platformio.ini +++ b/variants/heltec_v4/platformio.ini @@ -144,7 +144,7 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_19_5dBm - -D MQTT_WIFI_POWER_SAVE_DEFAULT=1 + -D WITH_SNMP=1 # -D WIFI_SSID='"ssid"' # -D WIFI_PWD='"password"' # -D MQTT_SERVER='"your-mqtt-broker.com"' @@ -155,6 +155,7 @@ build_src_filter = ${heltec_v4_oled.build_src_filter} + + + + + + +<../examples/simple_repeater> lib_deps = @@ -165,6 +166,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:heltec_v4_room_server_observer_mqtt] extends = heltec_v4_oled @@ -185,7 +187,6 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_19_5dBm - -D MQTT_WIFI_POWER_SAVE_DEFAULT=1 build_src_filter = ${heltec_v4_oled.build_src_filter} + + diff --git a/variants/station_g2/platformio.ini b/variants/station_g2/platformio.ini index 928201df..925eb464 100644 --- a/variants/station_g2/platformio.ini +++ b/variants/station_g2/platformio.ini @@ -297,6 +297,7 @@ build_flags = -D MESH_DEBUG=1 -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 + -D WITH_SNMP=1 # -D WIFI_SSID='"ssid"' # -D WIFI_PWD='"password"' # -D MQTT_SERVER='"your-mqtt-broker.com"' @@ -307,6 +308,7 @@ build_src_filter = ${Station_G2.build_src_filter} + + + + + + +<../examples/simple_repeater> lib_deps = @@ -317,3 +319,4 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent