Configure dataflow endpoints for local storage

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in preview. You shouldn't use this preview software in production environments.

You'll need to deploy a new Azure IoT Operations installation when a generally available release becomes available. You won't be able to upgrade a preview installation.

For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

To send data to local storage in Azure IoT Operations Preview, you can configure a dataflow endpoint. This configuration allows you to specify the endpoint, authentication, table, and other settings.

Prerequisites

• An instance of Azure IoT Operations Preview
• A configured dataflow profile
• A PersistentVolumeClaim (PVC)

Create a local storage dataflow endpoint

Use the local storage option to send data to a locally available persistent volume, through which you can upload data via Azure Container Storage enabled by Azure Arc edge volumes.

Portal

1. In the operations experience, select the Dataflow endpoints tab.

2. Under Create new dataflow endpoint, select Local Storage > New.

   

3. Enter the following settings for the endpoint:

   Setting	Description
   Name	The name of the dataflow endpoint.
   Persistent volume claim name	The name of the PersistentVolumeClaim (PVC) to use for local storage.

4. Select Apply to provision the endpoint.

Bicep

Create a Bicep .bicep file with the following content.

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'
param endpointName string = '<ENDPOINT_NAME>'
param persistentVCName string = '<PERSISTENT_VC_NAME>'

resource aioInstance 'Microsoft.IoTOperations/instances@2024-09-15-preview' existing = {
  name: aioInstanceName
}
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}
resource localStorageDataflowEndpoint 'Microsoft.IoTOperations/instances/dataflowEndpoints@2024-09-15-preview' = {
  parent: aioInstance
  name: endpointName
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    endpointType: 'LocalStorage'
    localStorageSettings: {
      persistentVolumeClaimRef: persistentVCName
    }
  }
}

Then, deploy via Azure CLI.

az deployment group create --resource-group <RESOURCE_GROUP> --template-file <FILE>.bicep

Kubernetes

Create a Kubernetes manifest .yaml file with the following content.

apiVersion: connectivity.iotoperations.azure.com/v1beta1
kind: DataflowEndpoint
metadata:
  name: <ENDPOINT_NAME>
  namespace: azure-iot-operations
spec:
  endpointType: localStorage
  localStorageSettings:
    persistentVolumeClaimRef: <PVC_NAME>

Then apply the manifest file to the Kubernetes cluster.

kubectl apply -f <FILE>.yaml

The PersistentVolumeClaim (PVC) must be in the same namespace as the DataflowEndpoint.

Supported serialization formats

The only supported serialization format is Parquet.

Use Azure Container Storage enabled by Azure Arc (ACSA)

You can use the local storage dataflow endpoint together with Azure Container Storage enabled by Azure Arc to store data locally or send data to a cloud destination.

Local shared volume

To write to a local shared volume, first create a PersistentVolumeClaim (PVC) according to the instructions from Local Shared Edge Volumes.

Then, when configuring your local storage dataflow endpoint, input the PVC name under persistentVolumeClaimRef.

Cloud ingest

To write your data to the cloud, follow the instructions in Cloud Ingest Edge Volumes configuration to create a PVC and attach a subvolume for your desired cloud destination.

Important

Don't forget to create the subvolume after creating the PVC, or else the dataflow fails to start and the logs show a "read-only file system" error.

Then, when configuring your local storage dataflow endpoint, input the PVC name under persistentVolumeClaimRef.

Finally, when you create the dataflow, the data destination parameter must match the spec.path parameter you created for your subvolume during configuration.

Next steps

To learn more about dataflows, see Create a dataflow.