Concurrent Device Connections and Telemetry on Azure IoT Hub

Question

Details:

• SDK Versions:
  • azure-iot-device : 2.14.0
    • azure-iot-hub : 2.6.1
      • asyncio : 3.4.3
        • Python Version : 3.10.12
        • Azure IoTHub Details:
          • Tier: S1

Description of Issue: I have developed a Python script using the Azure IoT SDK to connect 670 devices to Azure IoT Hub. The script is designed to connect each device asynchronously, update the device twin, and send telemetry data in parallel.

 

However, I am facing a connectivity limitation when running the script for all 670 devices simultaneously:

• After 337 devices successfully connect, update their twins, and start sending telemetry, the remaining devices are unable to establish a connection.
• Logs indicate a new connection error for devices beyond the initial 337 connections, with the following details:Details:
  • SDK Versions:
    • azure-iot-device : 2.14.0
    • azure-iot-hub : 2.6.1
    • asyncio : 3.4.3
    • Python Version : 3.10.12
  • Azure IoTHub Details:
    • Tier: S1
  Description of Issue:
  I have developed a Python script using the Azure IoT SDK to connect 670 devices to Azure IoT Hub. The script is designed to connect each device asynchronously, update the device twin, and send telemetry data in parallel.   However, I am facing a connectivity limitation when running the script for all 670 devices simultaneously:
  • After 337 devices successfully connect, update their twins, and start sending telemetry, the remaining devices are unable to establish a connection.
  • Logs indicate a new connection error for devices beyond the initial 337 connections, with the following details:
• pipeline_stages_mqtt.py:83 - MQTTTransportStage(ConnectOperation): Connection watchdog expired. Cancelling op mqtt_transport.py:441 - disconnecting MQTT client async_adapter.py:72 - Callback completed with error Transport timeout on connection operation async_adapter.py:73 - ['azure.iot.device.common.pipeline.pipeline_exceptions.OperationTimeout: Transport timeout on connection operation\n'] mqtt_transport.py:150 - Creating client for connecting using MQTT over TCP

In the exception, the error message received is: Error: Could not complete operation before timeout.

 

Interestingly, when I divide the workload into two batches of 335 devices each, running them in separate instances of a python script from the same IP address, all connections are successful, and each device updates its twin and sends telemetry as expected. Only same python process is not able to connect to all devices.

Answer 1

Hello @Nayangiri Goswami ,

welcome to this moderated Azure community forum.

Have you checked the Azure IoT Hub limitations and operation throttling?

It's hard to describe what causes the exact throttling or even an error in your situation but the observation that splitting up the batch in sub batches works better is conform these limitations.

You can either break down the call in several batches (as you did already), slow down processing the initial batch to give the IoT Hub some time to time to react, or you can temporarily scale up to a higher tier during processing (from the top of my head, the billing of the IoT Hub is done per day).

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.