MeshCore-mqtt-observer/MQTT_IMPLEMENTATION.md

MQTT Bridge Implementation for MeshCore

This document describes the MQTT bridge implementation that allows MeshCore repeaters to uplink packet data to multiple MQTT brokers.

Quick Start Guide

Essential Commands to Get MQTT Observer Running

1. Flash the observer firmware to your device

Use one of the observer build targets (e.g., heltec_v4_repeater_observer_mqtt). After flashing, connect to the device console via serial (115200 baud) or repeater login.

2. Configure radio settings

If this is a fresh flash or full erase, configure your radio parameters first. These must match other nodes in your mesh:

set freq 906.0
set sf 12
set cr 5
set bw 250
set tx 22

3. Configure device identity

set name MyObserver
set mqtt.iata SEA

If migrating from an existing node (e.g., a Raspberry Pi gateway), restore the private key to keep the same identity:

set prv.key <your_64_hex_char_private_key>

4. Configure WiFi credentials

set wifi.ssid YourWiFiNetwork
set wifi.pwd YourWiFiPassword

5. (Optional) Disable packet repeating

If this observer is receive-only (e.g., using a PCB antenna in a location where repeating would be harmful), disable forwarding:

set repeat off

6. Reboot to connect

reboot

7. Verify configuration

get wifi.ssid
get bridge.enabled
get bridge.source
get mqtt.origin
get mqtt.iata
get mqtt1.preset
get mqtt2.preset
get mqtt3.preset
get mqtt.status

That's it! The device will now:

• Connect to WiFi automatically
• Start uplinking mesh packets to configured MQTT brokers
• By default, publish to Let's Mesh Analyzer US (slot 1) and EU (slot 2)
• Use device name as MQTT origin (set automatically)

---

Overview

The MQTT bridge implementation provides:

• Up to 6 MQTT connection slots with built-in presets
• Built-in presets for LetsMesh Analyzer (US/EU), MeshMapper, MeshRank, Waev, Meshomatic, and CascadiaMesh
• Custom broker support with username/password authentication
• JWT (Ed25519 device signing) authentication for preset brokers
• WSS (WebSocket Secure) and direct MQTT transport
• Automatic reconnection with exponential backoff
• JSON message formatting for status, packets, and raw data
• Packet queuing during connection issues
• Automatic migration from old configuration format

Architecture

Slot-Based Preset System

The MQTT bridge uses a slot-based architecture with up to 6 concurrent connections. Each slot can be configured with a built-in preset or custom broker settings.

Built-in Presets:

Preset	Server	Auth	Transport
analyzer-us	mqtt-us-v1.letsmesh.net:443	JWT (Ed25519)	WSS
analyzer-eu	mqtt-eu-v1.letsmesh.net:443	JWT (Ed25519)	WSS
meshmapper	mqtt.meshmapper.cc:443	JWT (Ed25519)	WSS
meshrank	meshrank.net:8883	None (token in topic)	MQTT over TLS
waev	mqtt.waev.app:443	JWT (Ed25519)	WSS
meshomatic	us-east.meshomatic.net:443	JWT (Ed25519)	WSS
cascadiamesh	cascadiamesh.ddns.net:1883	None	MQTT (unencrypted)
custom	User-configured	Username/Password	MQTT or WSS
none	(disabled)	—	—

Default Configuration:

• Slot 1: analyzer-us
• Slot 2: analyzer-eu
• Slots 3-6: none

Memory Limits:

• With PSRAM: All slots can be active simultaneously
• Without PSRAM: Maximum 2 active TLS/WSS slots (each WSS/TLS connection requires ~40KB internal heap)
• If more slots are configured than the device supports, excess slots show as (inactive) in get mqtt.status
• Slot configurations are preserved in preferences — moving firmware to a PSRAM device activates all slots

Files

Core Implementation

• src/helpers/bridges/MQTTBridge.h - MQTT bridge class definition
• src/helpers/bridges/MQTTBridge.cpp - MQTT bridge implementation
• src/helpers/MQTTPresets.h - Preset definitions, CA certificates, and lookup functions
• src/helpers/MQTTMessageBuilder.h - JSON message formatting utilities
• src/helpers/MQTTMessageBuilder.cpp - JSON message formatting implementation
• src/helpers/JWTHelper.h - JWT token generation for Ed25519-based authentication

Integration

• Updated examples/simple_repeater/MyMesh.h - Added MQTT bridge support
• Updated examples/simple_repeater/MyMesh.cpp - Added MQTT bridge integration and raw radio data capture
• Updated src/helpers/CommonCLI.h - MQTT slot preferences, WiFi, and timezone fields
• Updated src/helpers/CommonCLI.cpp - MQTT slot CLI commands, migration logic

Build Configuration

To build the MQTT bridge firmware:

# Heltec V3
pio run -e Heltec_v3_repeater_observer_mqtt

# Heltec V4
pio run -e heltec_v4_repeater_observer_mqtt

# Station G2
pio run -e Station_G2_repeater_observer_mqtt

Build Flags

• WITH_MQTT_BRIDGE=1 - Enable MQTT bridge (required)
• WITH_SNMP=1 - Enable SNMP agent (optional, see MQTT_SNMP.md)
• MQTT_DEBUG=1 - Enable debug logging (optional)
• MQTT_WIFI_TX_POWER - WiFi TX power level (default: WIFI_POWER_11dBm)
• MQTT_WIFI_POWER_SAVE_DEFAULT - Removed; all builds now default to none (no power save)

Default Configuration

The MQTT bridge comes with the following defaults for fresh installs:

• Origin: Device name (set automatically from set name)
• IATA: (blank — must be configured for Analyzer presets)
• Status Messages: Enabled
• Packet Messages: Enabled
• Raw Messages: Disabled
• TX Messages: Disabled (RX only by default)
• Status Interval: 5 minutes (300000 ms)
• Slot 1: analyzer-us
• Slot 2: analyzer-eu
• Slots 3-6: none (disabled)
• WiFi SSID: (blank — must be configured)
• WiFi Password: (blank — must be configured)
• WiFi Power Save: none (no power save)
• Timezone: (blank — uses UTC until configured)
• Timezone Offset: 0 (fallback, no offset)
• Repeat (forwarding): On (set repeat off for receive-only observers)

CLI Commands

MQTT Slot Commands

Each slot (1-6) supports the following commands:

Get Commands

• get mqtt1.preset - Get slot 1 preset name
• get mqtt2.preset - Get slot 2 preset name
• get mqttN.preset - Get slot N preset name (N = 1-6)
• get mqttN.server - Get custom server hostname for slot N
• get mqttN.port - Get custom server port for slot N
• get mqttN.username - Get custom username for slot N
• get mqttN.password - Get custom password for slot N
• get mqttN.token - Get per-slot token (e.g., MeshRank account token)
• get mqttN.topic - Get custom topic template for slot N

Set Commands

• set mqttN.preset analyzer-us - Set slot N to LetsMesh Analyzer US
• set mqttN.preset analyzer-eu - Set slot N to LetsMesh Analyzer EU
• set mqttN.preset meshmapper - Set slot N to MeshMapper
• set mqttN.preset meshrank - Set slot N to MeshRank (requires token)
• set mqttN.preset waev - Set slot N to Waev
• set mqttN.preset meshomatic - Set slot N to Meshomatic
• set mqttN.preset cascadiamesh - Set slot N to CascadiaMesh
• set mqttN.preset custom - Set slot N to custom broker (configure server/port/username/password)
• set mqttN.preset none - Disable slot N
• set mqttN.server <hostname> - Set custom server hostname for slot N
• set mqttN.port <port> - Set custom server port for slot N (1-65535)
• set mqttN.username <username> - Set custom username for slot N
• set mqttN.password <password> - Set custom password for slot N
• set mqttN.token <token> - Set per-slot token (required for MeshRank preset)
• set mqttN.topic <template> - Set custom topic template (custom preset only, see below)

Note: Custom server/port/username/password settings only apply when the slot's preset is custom.

Example: Configure MeshRank on Slot 3

set mqtt3.preset meshrank
set mqtt3.token FE1B34242C5938C39225310081FD6718

The token is generated on the MeshRank website and is tied to your account. MeshRank only receives packet data (no status or raw messages).

Example: Configure MeshMapper on Slot 3

set mqtt3.preset meshmapper

Example: Configure Custom Broker on Slot 3

set mqtt3.preset custom
set mqtt3.server your-broker.example.com
set mqtt3.port 1883
set mqtt3.username your-username
set mqtt3.password your-password

Example: Custom Broker with Custom Topic Template

set mqtt3.preset custom
set mqtt3.server my-broker.local
set mqtt3.port 1883
set mqtt3.topic mynetwork/{device}/{type}

Custom Topic Templates

When a slot's preset is custom, you can define a custom topic template using placeholders:

Placeholder	Value	Example
{iata}	IATA airport code	SEA
{device}	Device public key (64 hex chars)	CC5D3CFD...
{token}	Per-slot token from mqttN.token	FE1B3424...
{type}	Message type	status, packets, or raw

If no custom topic is set, custom slots default to: meshcore/{iata}/{device}/{type}

Note: Topic templates only apply to custom preset slots. Built-in presets (analyzer-us, analyzer-eu, meshmapper, meshrank, etc.) always use their hardcoded topic format.

MQTT Shared Commands

These settings apply across all MQTT slots:

Get Commands

• get mqtt.origin - Get device origin name
• get mqtt.iata - Get IATA code
• get mqtt.status - Get MQTT status summary (connection info per slot)
• get mqtt.packets - Get packet message setting (on/off)
• get mqtt.raw - Get raw message setting (on/off)
• get mqtt.tx - Get TX message setting (on/off)
• get mqtt.interval - Get status publish interval
• get mqtt.owner - Get owner public key (serial console only)
• get mqtt.email - Get owner email address (serial console only)

Set Commands

• set mqtt.origin <name> - Set device origin name
• set mqtt.iata <code> - Set IATA code (auto-uppercased)
• set mqtt.status on|off - Enable/disable status messages
• set mqtt.packets on|off - Enable/disable packet messages
• set mqtt.raw on|off - Enable/disable raw messages
• set mqtt.tx on|off - Enable/disable TX packet messages
• set mqtt.interval <minutes> - Set status publish interval (1-60 minutes)
• set mqtt.owner <64-hex-char-public-key> - Set owner public key
• set mqtt.email <email> - Set owner email address

WiFi Commands

Get Commands

• get wifi.ssid - Get WiFi SSID
• get wifi.pwd - Get WiFi password
• get wifi.status - Get WiFi connection status, IP, RSSI, and uptime
• get wifi.powersave - Get WiFi power save mode (none/min/max)

Set Commands

• set wifi.ssid <ssid> - Set WiFi SSID
• set wifi.pwd <password> - Set WiFi password
• set wifi.powersave none|min|max - Set WiFi power save mode
  • none - No power saving (best performance, highest power consumption)
  • min - Minimum power saving (balanced performance and power)
  • max - Maximum power saving (lowest power consumption, may affect performance)

Timezone Commands

Get Commands

• get timezone - Get timezone string (e.g., "America/Los_Angeles")
• get timezone.offset - Get timezone offset in hours (-12 to +14)

Set Commands

• set timezone <string> - Set timezone string (IANA format or abbreviation)
• set timezone.offset <offset> - Set timezone offset in hours (-12 to +14)

Supported Timezone Formats

• IANA strings: America/Los_Angeles, Europe/London, Asia/Tokyo, etc.
• Common abbreviations: PDT, PST, MDT, MST, CDT, CST, EDT, EST, BST, GMT, CEST, CET
• UTC offsets: UTC-8, UTC+5, +5, -8, etc.

Device & Radio Commands

These are standard MeshCore commands, not MQTT-specific, but important for observer setup:

Get Commands

• get name - Get device name
• get repeat - Get repeat (forwarding) status (on/off)
• get freq - Get radio frequency
• get public.key - Get device public key (for migration)

Set Commands

• set name <name> - Set device name (also sets MQTT origin)
• set repeat on|off - Enable/disable packet forwarding (use off for receive-only observers)
• set prv.key <64-hex-char-key> - Restore private key (for migrating identity from another device)
• set freq <MHz> - Set radio frequency
• set sf <5-12> - Set LoRa spreading factor
• set bw <kHz> - Set LoRa bandwidth
• set cr <5-8> - Set LoRa coding rate
• set tx <dBm> - Set transmit power

Bridge Commands

Get Commands

• get bridge.source - Get packet source (rx/tx)
• get bridge.enabled - Get bridge enabled status (on/off)

Set Commands

• set bridge.source rx|tx - Set packet source (rx for received, tx for transmitted)
• set bridge.enabled on|off - Enable/disable bridge

SNMP Commands

Get Commands

• get snmp - Get SNMP agent status (on/off)
• get snmp.community - Get SNMP community string

Set Commands

• set snmp on|off - Enable/disable SNMP agent (restart required)
• set snmp.community <string> - Set SNMP community string (restart required, default: public)

See MQTT_SNMP.md for full SNMP documentation.

Command Architecture

The CLI commands are organized into two levels:

Bridge Commands (bridge.*)

Low-level bridge control - These settings apply to all bridge types (MQTT, RS232, ESP-NOW, etc.):

• bridge.enabled - Master switch for the entire bridge system
• bridge.source - Controls which packet events to capture (RX vs TX)

Bridge-Specific Commands (mqtt.*, mqttN.*, wifi.*, timezone.*)

Implementation-specific settings - These only apply to the MQTT bridge:

• mqttN.* - Per-slot MQTT broker configuration (N = 1-6)
• mqtt.* - Shared MQTT settings (message types, origin, IATA, etc.)
• wifi.* - WiFi connection settings for MQTT connectivity
• timezone.* - Timezone configuration for accurate timestamps

MQTT Topics

The bridge publishes to three main topics with the following structure:

Status Topic: meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/status

Device connection status and metadata (retained messages).

Packets Topic: meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/packets

Full packet data with RF characteristics and metadata.

Raw Topic: meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/raw

Minimal raw packet data for map integration.

Note: {DEVICE_PUBLIC_KEY} is the device's public key in hexadecimal format (64 characters).

JSON Message Formats

Status Message

{
  "status": "online|offline",
  "timestamp": "2024-01-01T12:00:00.000000",
  "origin": "Device Name",
  "origin_id": "DEVICE_PUBLIC_KEY",
  "model": "device_model",
  "firmware_version": "firmware_version",
  "radio": "radio_info",
  "client_version": "meshcore-custom-repeater/{build_date}"
}

Packet Message

{
  "origin": "MeshCore-HOWL",
  "origin_id": "A1B2C3D4E5F67890...",
  "timestamp": "2024-01-01T12:00:00.000000",
  "type": "PACKET",
  "direction": "rx|tx",
  "time": "12:00:00",
  "date": "01/01/2024",
  "len": "45",
  "packet_type": "4",
  "route": "F|D|T|U",
  "payload_len": "32",
  "raw": "F5930103807E5F1E...",
  "SNR": "12.5",
  "RSSI": "-65",
  "hash": "A1B2C3D4E5F67890",
  "path": "node1,node2,node3"
}

Raw Message

{
  "origin": "MeshCore-HOWL",
  "origin_id": "A1B2C3D4E5F67890...",
  "timestamp": "2024-01-01T12:00:00.000000",
  "type": "RAW",
  "data": "F5930103807E5F1E..."
}

Key Features

Slot-Based Preset System

• Up to 6 concurrent MQTT connections (with PSRAM), 2 without PSRAM
• Built-in presets for LetsMesh Analyzer (US/EU), MeshMapper, MeshRank, Waev, Meshomatic, and CascadiaMesh
• Custom broker support with username/password auth and custom topic templates
• JWT (Ed25519) authentication for preset brokers, token-in-topic for MeshRank
• WSS (WebSocket Secure) and direct MQTT over TLS transport
• Automatic reconnection with exponential backoff per slot
• Circuit breaker pattern with periodic probes for recovery from prolonged outages
• JWT token buffers only allocated for JWT-auth slots (memory efficient)
• Deferred construction: MQTTBridge is heap-allocated in begin() to avoid ESP32 static init crashes

Raw Radio Data Capture

• Captures actual raw radio transmission data (including radio headers)
• Uses proper MeshCore packet hashing (SHA256-based)
• Provides accurate SNR/RSSI values from actual radio reception
• Supports both RX and TX packet uplinking (configurable)

Timezone Support

• Full timezone support with automatic DST handling
• Supports IANA timezone strings, common abbreviations, and UTC offsets
• Separates local time (for timestamps) and UTC time (for time/date fields)
• Uses JChristensen/Timezone library for accurate timezone conversions

WiFi Configuration

• Runtime WiFi credential management via CLI
• Persistent storage across reboots
• Automatic reconnection with exponential backoff

NTP Time Synchronization

• Automatic time synchronization with NTP servers
• Periodic time updates (every hour)
• Proper UTC system time handling

Authentication

• JWT Authentication: Ed25519-signed tokens for secure MQTT authentication (used by all built-in presets except meshrank and cascadiamesh)
• Username/Password: Standard MQTT authentication for custom brokers
• Username Format (JWT): v1_{UPPERCASE_PUBLIC_KEY}
• Automatic Token Renewal: Tokens are renewed before expiration

Migration from Old Configuration

When upgrading from a firmware version that used the old MQTT configuration format (mqtt.analyzer.us, mqtt.analyzer.eu, mqtt.server, mqtt.port, mqtt.username, mqtt.password), the device automatically migrates settings:

• mqtt.analyzer.us = on → Slot 1 preset: analyzer-us
• mqtt.analyzer.eu = on → Slot 2 preset: analyzer-eu
• Custom server configured → Slot 3 preset: custom with host/port/username/password preserved
• All other settings (origin, IATA, message types, WiFi, timezone) are preserved as-is

The migration happens automatically on first boot after firmware update. No manual intervention is needed.

First-Time Setup

Prerequisites

• MeshCore device with observer MQTT firmware flashed
• WiFi network credentials
• Serial console access (115200 baud) or repeater login via companion app

Step 1: Configure Radio (after fresh flash/full erase)

If this is a fresh flash, radio parameters must be set to match your mesh network:

set freq 906.0
set sf 12
set cr 5
set bw 250
set tx 22

Step 2: Configure Device Identity

set name MyObserver
set mqtt.iata SEA

If migrating from an existing device, restore the private key to keep the same identity:

set prv.key <your_64_hex_char_private_key>

Step 3: Configure WiFi

set wifi.ssid YourWiFiNetwork
set wifi.pwd YourWiFiPassword
reboot

Step 4: Configure Timezone (optional)

set timezone America/New_York

Or use an offset as a fallback:

set timezone.offset -5

Step 5: (Optional) Disable Repeating

For receive-only observers (e.g., using a PCB antenna or in a location where repeating is not desired):

set repeat off

Step 6: Verify Slot Configuration

get mqtt1.preset    # Should show: analyzer-us
get mqtt2.preset    # Should show: analyzer-eu
get mqtt3.preset    # Should show: none

Step 7: (Optional) Add Additional Presets

set mqtt3.preset meshmapper

Step 8: Verify Connection

get bridge.enabled
get bridge.source
get mqtt.status
get wifi.status

Troubleshooting

Device Won't Connect to WiFi

get wifi.ssid
get wifi.pwd
set wifi.powersave none    # Try disabling power saving
reboot

No MQTT Messages Appearing

get bridge.enabled
set bridge.enabled on
get bridge.source          # Should be "rx"
set bridge.source rx
get mqtt.status            # Check per-slot connection status
get mqtt1.preset           # Verify slots are configured
get mqtt.iata              # IATA must be set for Analyzer presets

Timezone Issues

get timezone
set timezone America/New_York    # IANA format
set timezone EST                 # Abbreviation
set timezone UTC-5               # UTC offset

SNMP Monitoring

Observer nodes include an optional SNMP v2c agent that exposes radio stats, MQTT connectivity, memory usage, and network information to standard monitoring tools. See MQTT_SNMP.md for setup and OID reference.

Dependencies

• PsychicMqttClient: MQTT client library (supports WSS and direct MQTT)
• ArduinoJson: JSON message formatting
• NTPClient: Network time protocol client
• Timezone: Timezone conversion library (JChristensen/Timezone)
• WiFi: ESP32 WiFi functionality
• Ed25519: Cryptographic library for JWT token signing
• JWTHelper: Custom JWT token generation for device authentication
• SNMP_Agent: Optional SNMPv2c agent (0neblock/SNMP_Agent, observer builds only)