MeshCore-mqtt-observer/MQTT_MEMORY_ANALYSIS.md

MQTT Library Memory Analysis

Library Information

• Library: elims/PsychicMqttClient@^0.2.4
• Type: External PlatformIO library (wrapper around ESP-IDF MQTT client)
• Platform: ESP32
• Protocol: MQTT over WebSocket (WSS) for analyzer servers, standard MQTT for custom brokers
• Underlying Library: ESP-IDF esp_mqtt_client (part of ESP-IDF framework)
• Source Location: .pio/libdeps/*/PsychicMqttClient/src/

Current Usage Patterns

Memory Allocation Points

1. Client Instance Creation

   _mqtt_client = new PsychicMqttClient();  // Heap allocation
   _analyzer_us_client = new PsychicMqttClient();  // Heap allocation
   _analyzer_eu_client = new PsychicMqttClient();  // Heap allocation
   

   • Impact: 3 instances × ~few KB each = ~10-15KB base memory

2. setServer() Calls ✅ OPTIMIZED - NO ALLOCATION IN WRAPPER

   // From PsychicMqttClient.cpp line 215-223
   PsychicMqttClient &setServer(const char *uri) {
       _mqtt_cfg.broker.address.uri = uri;  // Just stores pointer
       return *this;
   }
   

   • Finding: Wrapper does NOT copy URI (just stores pointer)
   • Current Optimization: We only call setServer() when URI changes (using static tracking) ✅
   • ESP-IDF Behavior: When connect() is called, ESP-IDF likely copies the URI internally
   • Impact: Minimal (only on connection, not per publish)

3. setCredentials() Calls ✅ NO ALLOCATION IN WRAPPER

   // From PsychicMqttClient.cpp line 166-178
   PsychicMqttClient &setCredentials(const char *username, const char *password) {
       _mqtt_cfg.credentials.username = username;  // Just stores pointers
       _mqtt_cfg.credentials.authentication.password = password;
       return *this;
   }
   

   • Finding: Wrapper does NOT copy credentials (just stores pointers)
   • Frequency: Once per broker connection
   • ESP-IDF Behavior: Likely copies when connecting
   • Impact: Minimal (only on connection, not per publish)

4. publish() Calls ⚠️ CONFIRMED MAIN CULPRIT

   // From PsychicMqttClient.cpp line 370-389
   int publish(const char *topic, int qos, bool retain, const char *payload, int length, bool async)
   {
       if (async) {
           return esp_mqtt_client_enqueue(_client, topic, payload, length, qos, retain, true);
       } else {
           return esp_mqtt_client_publish(_client, topic, payload, length, qos, retain);
       }
   }
   

   • Frequency: High (every packet, status updates)
   • Parameters:
     • topic: String pointer (passed to ESP-IDF)
     • payload: Buffer pointer + length (passed to ESP-IDF)
   • Critical Finding:
     • PsychicMqttClient wrapper does NOT copy payload (passes pointers directly)
     • BUT: ESP-IDF esp_mqtt_client_enqueue() and esp_mqtt_client_publish() DO copy internally
     • ESP-IDF copies both topic and payload into internal buffers for async processing
   • Memory Impact:
     • Each publish: strlen(topic) + payload_len bytes allocated by ESP-IDF
     • For 2KB JSON payloads + ~50 byte topics: ~2KB per publish
   • Multiple Publishes Per Packet:
     • Custom brokers: 1-3 publishes (one per broker)
     • Analyzer servers: 2 publishes (US + EU)
     • Total: Up to 5 publishes per packet × 2KB = 10KB+ allocations per packet
   • Async vs Sync:
     • We use async=true (default), which calls esp_mqtt_client_enqueue()
     • Enqueue likely allocates more (needs queue buffer) than direct publish

Memory Fragmentation Evidence

Observed Behavior

• When MQTT Active: Max alloc drops to ~54KB (severe fragmentation)
• When MQTT Disconnected: Max alloc recovers to ~88KB
• Pattern: Memory recovers when publishes stop

Root Cause - CONFIRMED ✅

1. ESP-IDF esp_mqtt_client_enqueue() copies payload internally:

   • PsychicMqttClient wrapper passes pointers (no copy in wrapper)
   • ESP-IDF MQTT client copies topic + payload for async queue processing
   • Each publish: topic_len + payload_len bytes allocated by ESP-IDF
   • This is the main source of fragmentation

2. Multiple publishes per packet multiply allocations:

   • 1 packet → 5 publishes (3 brokers + 2 analyzers) = 5× allocations
   • Each allocation: ~2KB (topic + payload)
   • Total: ~10KB per packet in heap allocations

3. Frequent publishes prevent heap coalescing:

   • New allocations before old ones are freed
   • ESP-IDF async queue holds messages until sent
   • Creates "holes" in heap that can't be coalesced

Recommendations

1. Investigate Library Source Code

Since the library is external, we need to:

• Check if library source is available in PlatformIO cache
• Review publish() implementation for memory allocations
• Look for zero-copy or buffer reuse options

2. Potential Optimizations

A. Reduce Number of Publishes ⭐ HIGHEST IMPACT

• Current: Publish to all brokers + analyzers separately (5 publishes per packet)
• Option 1: Publish to one analyzer server only (reduce from 2 to 1)
  • Impact: 20% reduction in allocations (2KB saved per packet)
• Option 2: Use single custom broker with forwarding
  • Impact: 60% reduction (from 3 brokers to 1)
• Option 3: Combine both (1 broker + 1 analyzer)
  • Impact: 60% reduction (from 5 to 2 publishes)
• Trade-off: Less redundancy, simpler architecture

B. Reduce Payload Sizes

• Current: Up to 2KB JSON per packet
• Option: Compress or reduce JSON size
• Impact: Reduces allocation size per publish
• Trade-off: Less data per message

C. Throttle Publishes (Already Implemented) ✅

• Current: Skip publishes when Max alloc < 60KB
• Status: ✅ Implemented
• Effect: Prevents further fragmentation when memory is low

D. Use Synchronous Publishes (Blocking)

• Current: async=true (default) uses esp_mqtt_client_enqueue()
• Option: Use async=false to call esp_mqtt_client_publish() directly
• Impact: May reduce queue overhead, but blocks until sent
• Trade-off: Blocks execution, may impact responsiveness
• Code Change: publish(topic, qos, retain, payload, len, false)

E. Reduce Buffer Size Configuration

• Current: Default buffer size is 1024 bytes (from setBufferSize())
• Option: Reduce buffer size if messages are smaller
• Impact: Less memory per client instance
• Note: May cause message fragmentation if payloads exceed buffer

F. Investigate ESP-IDF MQTT Client Configuration

• ESP-IDF MQTT client has internal buffer configuration
• May be able to reduce queue depth or buffer sizes
• Requires: ESP-IDF documentation review

3. Library Alternatives (If Issues Persist)

Consider lightweight MQTT libraries with better memory management:

• lwmqtt: Zero-copy, no dynamic allocations
• coreMQTT: Fixed buffers, predictable memory
• Custom implementation: Full control over memory

Next Steps

1. ✅ Locate Library Source: Found in .pio/libdeps/*/PsychicMqttClient/src/

2. ✅ Analyze publish() Implementation:

   • Wrapper doesn't copy (passes pointers)
   • ESP-IDF esp_mqtt_client_enqueue() copies internally
   • This is the root cause

3. Test Synchronous Publishes:

   • Try async=false to use esp_mqtt_client_publish() directly
   • May have different memory behavior (no queue)
   • Measure fragmentation impact

4. Reduce Number of Publishes:

   • Implement single analyzer server (US or EU)
   • Or reduce custom broker count
   • Measure memory improvement

5. Profile ESP-IDF MQTT Client:

   • Check ESP-IDF documentation for buffer configuration
   • Look for queue depth settings
   • Investigate if we can reduce internal buffers

6. Consider ESP-IDF MQTT Client Direct Usage:

   • Bypass PsychicMqttClient wrapper
   • Use ESP-IDF API directly with custom memory management
   • More control, but more complex

7. Alternative: Switch to Different Library:

   • Consider lwmqtt (zero-copy, no dynamic allocations)
   • Or coreMQTT (fixed buffers)
   • Requires significant refactoring

Current Mitigations

✅ Memory Pressure Monitoring: Skip publishes when fragmented ✅ setServer() Optimization: Only call when URI changes
✅ JSON Buffer Reuse: Build once, publish to multiple destinations ✅ Analyzer Server Throttling: Skip when memory low

Remaining Risk

⚠️ Library Internal Allocations: If publish() copies payloads internally, we can't control this without:

• Library modification
• Library replacement
• Library API changes (if supported)