How can I store data from IoT Hub/Function App/Azure Digital Twins into a storage account in CSV format?

Question

I’m currently receiving live data from equipment and have set up IoT Hub, Function App, and Azure Digital Twins, successfully viewing the data across these layers. Now, I want to store this data in Azure Blob Storage or another storage solution in CSV or text file format, rather than in JSON, as the data from IoT Hub is in JSON format. Could you suggest the most effective method for achieving this? I previously tried using Stream Analytics but found it to be quite expensive. Any advice would be greatly appreciated. Thanks for your help!

Answer 1

Hello @Jayashri Goddanti ,

welcome to this moderated Azure community forum.

There are a few options to consider.

If you only want to forward raw IoT Hub messages to a storage account, you normally use a IoT Hub route towards an custom blob storage endpoint. Unfortunately, you want to use the CSV format, this is not supported by the IoT Hub routing endpoint (only AVRO and JSON).

If you still want CSV, you could create an Azure Function triggered when a file is created on the storage account, converting the file to CSV and deleting the original file.

If you are interested in events coming from the Azure Digital Twins Explorer, these are normally sent one-by-one to an Eventhub. So, batching them into separate CSV file will be a complex procedure.

A possible solution would be forwarding ADT message to an Azure Function, triggered by the EventHub. The function puts each new message in a storage queue. If there are enough messages collected, the queue is emptied andthe messages are saved in a CSV file.

An Alternative would be using Azure Data Factory, able to transform files too.

As you can see, your wish to use CSV makes it all a bit complex.

Why do you want to have it in CSV/text format?

Do you know JSON is also just another text format?

I want to propose a different approach.

You can forward all ADT twin changes and/or telemetry, and/or IoT Hub raw messages to EventHubs.

Have you considered the free Azure Data Explorer already?

It feel you want to do some data engineering on the incoming data by storing it for a longer period.

Normally Azure Data Explorer would be the perfect tool for this but an ADX cluster is not cheap if you have a simple project.

Microsoft provides a free ADX cluster, capable of ingesting streaming data coming from an Event Hub so you could try this out too.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.