Sporadically receiving MQTTRecvFailed error [Azure IoT middleware]

Question

Hello,

I'm doing endurance testing on the Azure IoT middleware for the FreeRTOS SDK (link). I am thinking of using the library for our application. I've modified the esp32 sample_azure_iot_pnp example for the test. Currently, the endurance test sends 4 packets every 9.5 seconds, repeated 880 times.

When testing - often mid-flow - I get the following error:

I haven't been able to identify the cause of this error or why it occurs. Has anyone experienced this or know how to deal with it? I have not been able to find out what errno 119 corresponds to?

My current workaround is to disconnect and restart the Azure code when the error occurs. When I do that then I leak some memory from the heap:

Have I forgotten to call or close something? Here is the task:

static void prvAzureDemoTask(void *arg)
{
    base_event* received_ele;
    AzureIoTResult_t xResult;
    esp_err_t ret;
    while (1) {
        if ( ulTaskNotifyTake(pdTRUE, portMAX_DELAY) )
        {
            ESP_LOGW(LIBRARY_LOG_NAME, "Free heap memory in bytes: %d", heap_caps_get_free_size(MALLOC_CAP_8BIT));
            /* Attempt to establish TLS session with IoT Hub. If connection fails,
             * retry after a timeout. Timeout value will be exponentially increased
             * until  the maximum number of attempts are reached or the maximum timeout
             * value is reached. The function returns a failure status if the TCP
             * connection cannot be established to the IoT Hub after the configured
             * number of attempts. */
            ulStatus = prvConnectToServerWithBackoffRetries( ( const char * ) pucIotHubHostname,
                                                             democonfigIOTHUB_PORT,
                                                             &xNetworkCredentials, &xNetworkContext );
            configASSERT( ulStatus == 0 );
        
            /* Fill in Transport Interface send and receive function pointers. */
            xTransport.pxNetworkContext = &xNetworkContext;
            xTransport.xSend = TLS_Socket_Send;
            xTransport.xRecv = TLS_Socket_Recv;
            xResult = AzureIoTHubClient_OptionsInit( &xHubOptions );
            xResult = AzureIoTHubClient_Init( &xAzureIoTHubClient,
                                    pucIotHubHostname, pulIothubHostnameLength,
                                    pucIotHubDeviceId, pulIothubDeviceIdLength,
                                    &xHubOptions,
                                    ucMQTTMessageBuffer, sizeof( ucMQTTMessageBuffer ),
                                    ullGetUnixTime,
                                    &xTransport );
            configASSERT( xResult == eAzureIoTSuccess );
            xResult = AzureIoTHubClient_SetSymmetricKey( &xAzureIoTHubClient,
                                                            ( const uint8_t * ) democonfigDEVICE_SYMMETRIC_KEY,
                                                            sizeof( democonfigDEVICE_SYMMETRIC_KEY ) - 1,
                                                            Crypto_HMAC );
            configASSERT( xResult == eAzureIoTSuccess );
  
            /* Sends an MQTT Connect packet over the already established TLS connection,
             * and waits for connection acknowledgment (CONNACK) packet. */
            LogInfo( ( "Creating an MQTT connection to %s.", pucIotHubHostname ) );
            xResult = AzureIoTHubClient_Connect( &xAzureIoTHubClient,
                                                 false, &xSessionPresent,
                                                 CONNACK_RECV_TIMEOUT_MS );
            configASSERT( xResult == eAzureIoTSuccess );
            // subscribe to something
            xResult = AzureIoTHubClient_SubscribeCommand( &xAzureIoTHubClient, prvHandleCommand,
                                                          &xAzureIoTHubClient, sampleazureiotSUBSCRIBE_TIMEOUT );
            configASSERT( xResult == eAzureIoTSuccess );
            xResult = AzureIoTHubClient_SubscribeProperties( &xAzureIoTHubClient, prvHandlePropertiesMessage,
                                                             &xAzureIoTHubClient, sampleazureiotSUBSCRIBE_TIMEOUT );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Get property document after initial connection */
            xResult = AzureIoTHubClient_RequestPropertiesAsync( &xAzureIoTHubClient );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Create a bag of properties for the telemetry */
            xResult = AzureIoTMessage_PropertiesInit( &xPropertyBag, ucPropertyBuffer, 0, sizeof( ucPropertyBuffer ) );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Sending a default property (Content-Type). */
            xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag,
                                                        ( uint8_t * ) AZ_IOT_MESSAGE_PROPERTIES_CONTENT_TYPE, sizeof( AZ_IOT_MESSAGE_PROPERTIES_CONTENT_TYPE ) - 1,
                                                        ( uint8_t * ) sampleazureiotMESSAGE_CONTENT_TYPE, sizeof( sampleazureiotMESSAGE_CONTENT_TYPE ) - 1 );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Sending a default property (Content-Encoding). */
            xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag,
                                                        ( uint8_t * ) AZ_IOT_MESSAGE_PROPERTIES_CONTENT_ENCODING, sizeof( AZ_IOT_MESSAGE_PROPERTIES_CONTENT_ENCODING ) - 1,
                                                        ( uint8_t * ) sampleazureiotMESSAGE_CONTENT_ENCODING, sizeof( sampleazureiotMESSAGE_CONTENT_ENCODING ) - 1 );
            configASSERT( xResult == eAzureIoTSuccess );
            memset(ucScratchBuffer, 0, sizeof( ucScratchBuffer ));
            while (xQueueReceive(azure_queue, &received_ele, /*pdMS_TO_TICKS(200)*/portMAX_DELAY) == pdPASS){
                
                received_ele->create_json(ucScratchBuffer, CONFIG_AZURE_CONNECTIVITY_SCRATCH_BUFFERS, &ulScratchBufferLength);
				
                /* Publish messages with QoS1, send and process Keep alive messages. */
                xResult = AzureIoTHubClient_SendTelemetry( &xAzureIoTHubClient,
                                                            ucScratchBuffer, ulScratchBufferLength,
                                                            &xPropertyBag, eAzureIoTHubMessageQoS1, NULL );
                if(xResult != eAzureIoTSuccess){
                    ESP_LOGE(LIBRARY_LOG_NAME,"Failed to send the dato to the cloud.");
                }
                // you need to call this every push to the Iot Hub
                // https://learn.microsoft.com/en-us/answers/questions/694812/mqttnomemory-after-a-period-of-time-publishing
                xResult = AzureIoTHubClient_ProcessLoop( &xAzureIoTHubClient, sampleazureiotPROCESS_LOOP_TIMEOUT_MS );
                if(xResult != eAzureIoTSuccess){
                    ESP_LOGE(LIBRARY_LOG_NAME,"Process loop got an error: close and re-open a new session.");
                    
                    // enqueue back the element
                    azure_connectivity_enqueue(&received_ele,0);
                    // notify to start the init process again
                    xTaskNotifyGive(azure_task_hndl);
                    break;
                }
                else{
                    delete received_ele;
                }               
                ESP_LOGW(LIBRARY_LOG_NAME, "Free heap memory in bytes after process Loop event: %d", heap_caps_get_free_size(MALLOC_CAP_8BIT));
            }
       
            // unsubsribe to properties and commands
            xResult = AzureIoTHubClient_UnsubscribeProperties( &xAzureIoTHubClient );
            configASSERT( xResult == eAzureIoTSuccess );
            xResult = AzureIoTHubClient_UnsubscribeCommand( &xAzureIoTHubClient );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Send an MQTT Disconnect packet over the already connected TLS over
            * TCP connection. There is no corresponding response for the disconnect
            * packet. After sending disconnect, client must close the network
            * connection. */
            xResult = AzureIoTHubClient_Disconnect( &xAzureIoTHubClient );
            configASSERT( xResult == eAzureIoTSuccess );
            /* Close the network connection.  */
            TLS_Socket_Disconnect( &xNetworkContext );
            LogInfo( ( "Closing the connection." ) );
            ESP_LOGW(LIBRARY_LOG_NAME, "Free heap memory in bytes at the closing: %d", heap_caps_get_free_size(MALLOC_CAP_8BIT));
            
        } else {
            LogWarn( ( "Queue received returned with an error:") );
        } // if statement
    }// while loop

Thanks in advance for any advice!

Answer 1

Hi Klajderic Aljosa HSLU I

Thank you for the update.

Please let us know if the below suggestion helps.

1. Retry Logic: Optimize no of retries and delay in between concurrent requests.
2. Resetting TLS State
3. Increase TTL time to keep the session connected

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Thank you.