Edit

Share via

Configure dataflow profile

  • Article

Important

This page includes instructions for managing Azure IoT Operations components using Kubernetes deployment manifests, which is in preview. This feature is provided with several limitations, and shouldn't be used for production workloads.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Dataflow profiles can be used to group dataflows together so that they share the same configuration. You can create multiple dataflow profiles to manage sets of different dataflow configurations.

The most important setting is the instance count, which determines the number of instances that run the dataflows. For example, you might have a dataflow profile with a single instance for development and testing, and another profile with multiple instances for production. Or, you might use a dataflow profile with low instance count for low-throughput dataflows and a profile with high instance count for high-throughput dataflows. Similarly, you can create a dataflow profile with different diagnostic settings for debugging purposes.

Default dataflow profile

By default, a dataflow profile named "default" is created when Azure IoT Operations is deployed. This dataflow profile has a single instance count. You can use this dataflow profile to get started with Azure IoT Operations.

Currently, when using the operations experience portal, the default dataflow profile is used for all dataflows.

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'

// Pointer to the Azure IoT Operations instance
resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
  name: aioInstanceName
}

// Pointer to your custom location where AIO is deployed
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}

// Pointer to the default dataflow profile
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' = {
  parent: aioInstance
  name: 'default'
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    instanceCount: 1
  }
}

Unless you need additional throughput or redundancy, you can use the default dataflow profile for your dataflows. If you need to adjust the instance count or other settings, you can create a new dataflow profile.

Create a new dataflow profile

To create a new dataflow profile, specify the name of the profile and the instance count.

resource dataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' = {
  parent: aioInstance
  name: '<NAME>'
  properties: {
    instanceCount: <COUNT>
  }
}

Scaling

You can scale the dataflow profile to adjust the number of instances that run the dataflows. Increasing the instance count can improve the throughput of the dataflows by creating multiple clients to process the data. When using dataflows with cloud services that have rate limits per client, increasing the instance count can help you stay within the rate limits.

Scaling can also improve the resiliency of the dataflows by providing redundancy in case of failures.

To manually scale the dataflow profile, specify the number of instances you want to run. For example, to set the instance count to 3:

properties: {
  instanceCount: 3
}

Diagnostic settings

You can configure other diagnostics settings for a dataflow profile such as log level and metrics interval.

In most cases, the default settings are sufficient. However, you can override the log level or other settings for debugging.

To learn how to configure these diagnostic settings, see ProfileDiagnostics.

For example, to set the log level to debug:

resource dataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' = {
  parent: aioInstance
  name: '<NAME>'
  properties: {
    instanceCount: 1
    diagnostics: {
      {
        logs: {
          level: 'debug'
        }
      }
    }
  }
}

Next steps

To learn more about dataflows, see Create a dataflow.