MeshCore-mqtt-observer/MQTT_LIBRARY_ALTERNATIVES.md

MQTT Library Alternatives with WebSocket Support

Current Library

• PsychicMqttClient (elims/PsychicMqttClient@^0.2.4)
• WebSocket Support: ✅ Yes (WSS://)
• Memory Issue: ESP-IDF esp_mqtt_client_enqueue() copies payloads internally
• Platform: ESP32 only

Alternative Libraries with WebSocket Support

1. AsyncMqttClient ⭐ RECOMMENDED

• Author: Marvin Roger
• GitHub: https://github.com/marvinroger/AsyncMqttClient
• PlatformIO: marvinroger/AsyncMqttClient
• WebSocket Support: ✅ Yes (WSS://)
• Memory Management:
  • Uses ESP-IDF MQTT client (same underlying library as PsychicMqttClient)
  • Same memory issue: ESP-IDF copies payloads internally
• Pros:
  • Well-established, widely used
  • Similar API to PsychicMqttClient (almost drop-in replacement)
  • Good documentation
• Cons:
  • Same underlying ESP-IDF library = same memory fragmentation issue
  • May not solve the memory problem

2. ESP-IDF MQTT Client (Direct Usage)

• Library: Built into ESP-IDF framework
• WebSocket Support: ✅ Yes (WSS://)
• Memory Management:
  • Same as PsychicMqttClient (it's a wrapper)
  • Can configure buffer sizes via esp_mqtt_client_config_t
  • May allow more control over memory
• Pros:
  • No wrapper overhead
  • Direct control over configuration
  • Can set buffer sizes, queue depths
• Cons:
  • More complex API
  • Requires ESP-IDF knowledge
  • Still copies payloads internally
• Configuration Options:

  esp_mqtt_client_config_t mqtt_cfg = {
      .buffer.size = 1024,  // Can reduce this
      .buffer.out_size = 1024,  // Can reduce this
      // ... other config
  };
  

3. lwmqtt (Lightweight MQTT)

• GitHub: https://github.com/256dpi/lwmqtt
• WebSocket Support: ❌ NO - TCP only
• Memory Management:
  • Zero-copy design
  • No dynamic allocations
  • Fixed buffers
• Pros:
  • Excellent memory efficiency
  • Zero-copy, no fragmentation
  • Very lightweight
• Cons:
  • No WebSocket support (deal breaker for analyzer servers)
  • Would need separate WebSocket implementation
  • More complex integration

4. PubSubClient

• PlatformIO: knolleary/PubSubClient
• WebSocket Support: ❌ NO - TCP only
• Memory Management:
  • Uses fixed buffer (configurable size)
  • Copies payloads into buffer
• Pros:
  • Simple API
  • Widely used
  • Predictable memory usage
• Cons:
  • No WebSocket support (deal breaker)
  • Synchronous (blocks)
  • Less efficient than async libraries

5. Custom WebSocket + MQTT Implementation

• Approach: Use ESP-IDF WebSocket client + custom MQTT protocol layer
• WebSocket Support: ✅ Yes (full control)
• Memory Management:
  • Full control over allocations
  • Can implement zero-copy
  • Custom buffer management
• Pros:
  • Complete control over memory
  • Can optimize for our use case
  • No unnecessary copies
• Cons:
  • Significant development effort
  • Need to implement MQTT protocol
  • Testing and maintenance burden

Recommendation

Option A: Stay with PsychicMqttClient + Optimize Usage ⭐ BEST SHORT-TERM

• Why: All ESP32 MQTT libraries use ESP-IDF underneath = same memory issue
• Actions:
  1. Reduce number of publishes (single analyzer server)
  2. Test synchronous publishes (async=false)
  3. Configure ESP-IDF buffer sizes via PsychicMqttClient
  4. Keep memory pressure monitoring

Option B: Switch to AsyncMqttClient

• Why: More mature, better documented, similar API
• Trade-off: Same memory issue (uses ESP-IDF)
• Effort: Medium (API is similar, mostly drop-in)

Option C: Use ESP-IDF MQTT Client Directly

• Why: More control over configuration
• Actions:
  • Bypass wrapper library
  • Configure buffer sizes directly
  • May reduce some overhead
• Effort: High (need to rewrite MQTT bridge code)
• Benefit: Can tune ESP-IDF buffer sizes

Option D: Custom Implementation (Long-term)

• Why: Complete control over memory
• Effort: Very High (weeks of development)
• Benefit: Optimal memory usage, zero-copy possible

Key Finding

⚠️ All ESP32 MQTT libraries that support WebSockets use ESP-IDF esp_mqtt_client underneath, which copies payloads internally. This is a limitation of the ESP-IDF framework, not the wrapper libraries.

Options to reduce memory impact:

1. ✅ Reduce number of publishes (already identified)
2. ✅ Memory pressure monitoring (already implemented)
3. ⚠️ Configure ESP-IDF buffer sizes (may help)
4. ⚠️ Use synchronous publishes (may reduce queue overhead)
5. ⚠️ Custom implementation (significant effort)

Next Steps

1. Test ESP-IDF Buffer Configuration:

   • Try reducing setBufferSize() in PsychicMqttClient
   • May reduce per-client memory but won't fix publish allocations

2. Test Synchronous Publishes:

   • Try async=false to bypass queue
   • May reduce memory but blocks execution

3. Reduce Publishes:

   • Implement single analyzer server
   • Measure memory improvement

4. Consider ESP-IDF Direct Usage:

   • If other optimizations don't help enough
   • More control but more complexity