Microsoft.IoTOperations instances/dataflowProfiles/dataflows

• Latest
• 2024-11-01
• 2024-09-15-preview
• 2024-08-15-preview
• 2024-07-01-preview

Bicep resource definition

The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:

• Resource groups - See resource group deployment commands

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following Bicep to your template.

resource symbolicname 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-11-01' = {
  parent: resourceSymbolicName
  extendedLocation: {
    name: 'string'
    type: 'string'
  }
  name: 'string'
  properties: {
    mode: 'string'
    operations: [
      {
        builtInTransformationSettings: {
          datasets: [
            {
              description: 'string'
              expression: 'string'
              inputs: [
                'string'
              ]
              key: 'string'
              schemaRef: 'string'
            }
          ]
          filter: [
            {
              description: 'string'
              expression: 'string'
              inputs: [
                'string'
              ]
              type: 'string'
            }
          ]
          map: [
            {
              description: 'string'
              expression: 'string'
              inputs: [
                'string'
              ]
              output: 'string'
              type: 'string'
            }
          ]
          schemaRef: 'string'
          serializationFormat: 'string'
        }
        destinationSettings: {
          dataDestination: 'string'
          endpointRef: 'string'
        }
        name: 'string'
        operationType: 'string'
        sourceSettings: {
          assetRef: 'string'
          dataSources: [
            'string'
          ]
          endpointRef: 'string'
          schemaRef: 'string'
          serializationFormat: 'string'
        }
      }
    ]
  }
}

Property values

DataflowBuiltInTransformationDataset

Name	Description	Value
description	A user provided optional description of the dataset.	string Constraints: Max length = 256
expression	Condition to enrich data from Broker State Store. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string
inputs	List of fields for enriching from the Broker State Store.	string[] (required)
key	The key of the dataset.	string Constraints: Max length = 64 (required)
schemaRef	The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7.	string

DataflowBuiltInTransformationFilter

Name	Description	Value
description	A user provided optional description of the filter.	string Constraints: Max length = 256
expression	Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string (required)
inputs	List of fields for filtering in JSON path expression.	string[] (required)
type	The type of dataflow operation.	'Filter'

DataflowBuiltInTransformationMap

Name	Description	Value
description	A user provided optional description of the mapping function.	string Constraints: Max length = 256
expression	Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided)	string
inputs	List of fields for mapping in JSON path expression.	string[] (required)
output	Where and how the input fields to be organized in the output record.	string (required)
type	Type of transformation.	'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename'

DataflowBuiltInTransformationSettings

Name	Description	Value
datasets	Enrich data from Broker State Store. Dataset references a key in Broker State Store.	DataflowBuiltInTransformationDataset[]
filter	Filters input record or datapoints based on condition.	DataflowBuiltInTransformationFilter[]
map	Maps input to output message.	DataflowBuiltInTransformationMap[]
schemaRef	Reference to the schema that describes the output of the transformation.	string
serializationFormat	Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json	'Delta' 'Json' 'Parquet'

DataflowDestinationOperationSettings

Name	Description	Value
dataDestination	Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription.	string Constraints: Min length = 1 (required)
endpointRef	Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type.	string (required)

DataflowOperation

Name	Description	Value
builtInTransformationSettings	Built In Transformation configuration.	DataflowBuiltInTransformationSettings
destinationSettings	Destination configuration.	DataflowDestinationOperationSettings
name	Optional user provided name of the transformation.	string Constraints: Max length = 64
operationType	Type of operation.	'BuiltInTransformation' 'Destination' 'Source' (required)
sourceSettings	Source configuration.	DataflowSourceOperationSettings

DataflowProperties

Name	Description	Value
mode	Mode for Dataflow. Optional; defaults to Enabled.	'Disabled' 'Enabled'
operations	List of operations including source and destination references as well as transformation.	DataflowOperation[] (required)

DataflowSourceOperationSettings

Name	Description	Value
assetRef	Reference to the resource in Azure Device Registry where the data in the endpoint originates from.	string
dataSources	List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +.	string Constraints: Min length = 1[] (required)
endpointRef	Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type.	string (required)
schemaRef	Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match.	string
serializationFormat	Content is a JSON Schema. Allowed: JSON Schema/draft-7.	'Json'

ExtendedLocation

Name	Description	Value
name	The name of the extended location.	string (required)
type	Type of ExtendedLocation.	'CustomLocation' (required)

Microsoft.IoTOperations/instances/dataflowProfiles/dataflows

Name	Description	Value
extendedLocation	Edge location of the resource.	ExtendedLocation (required)
name	The resource name	string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required)
parent	In Bicep, you can specify the parent resource for a child resource. You only need to add this property when the child resource is declared outside of the parent resource. For more information, see Child resource outside parent resource.	Symbolic name for resource of type: instances/dataflowProfiles
properties	The resource-specific properties for this resource.	DataflowProperties

ARM template resource definition

The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:

• Resource groups - See resource group deployment commands

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following JSON to your template.

{
  "type": "Microsoft.IoTOperations/instances/dataflowProfiles/dataflows",
  "apiVersion": "2024-11-01",
  "name": "string",
  "extendedLocation": {
    "name": "string",
    "type": "string"
  },
  "properties": {
    "mode": "string",
    "operations": [
      {
        "builtInTransformationSettings": {
          "datasets": [
            {
              "description": "string",
              "expression": "string",
              "inputs": [ "string" ],
              "key": "string",
              "schemaRef": "string"
            }
          ],
          "filter": [
            {
              "description": "string",
              "expression": "string",
              "inputs": [ "string" ],
              "type": "string"
            }
          ],
          "map": [
            {
              "description": "string",
              "expression": "string",
              "inputs": [ "string" ],
              "output": "string",
              "type": "string"
            }
          ],
          "schemaRef": "string",
          "serializationFormat": "string"
        },
        "destinationSettings": {
          "dataDestination": "string",
          "endpointRef": "string"
        },
        "name": "string",
        "operationType": "string",
        "sourceSettings": {
          "assetRef": "string",
          "dataSources": [ "string" ],
          "endpointRef": "string",
          "schemaRef": "string",
          "serializationFormat": "string"
        }
      }
    ]
  }
}

Property values

DataflowBuiltInTransformationDataset

Name	Description	Value
description	A user provided optional description of the dataset.	string Constraints: Max length = 256
expression	Condition to enrich data from Broker State Store. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string
inputs	List of fields for enriching from the Broker State Store.	string[] (required)
key	The key of the dataset.	string Constraints: Max length = 64 (required)
schemaRef	The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7.	string

DataflowBuiltInTransformationFilter

Name	Description	Value
description	A user provided optional description of the filter.	string Constraints: Max length = 256
expression	Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string (required)
inputs	List of fields for filtering in JSON path expression.	string[] (required)
type	The type of dataflow operation.	'Filter'

DataflowBuiltInTransformationMap

Name	Description	Value
description	A user provided optional description of the mapping function.	string Constraints: Max length = 256
expression	Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided)	string
inputs	List of fields for mapping in JSON path expression.	string[] (required)
output	Where and how the input fields to be organized in the output record.	string (required)
type	Type of transformation.	'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename'

DataflowBuiltInTransformationSettings

Name	Description	Value
datasets	Enrich data from Broker State Store. Dataset references a key in Broker State Store.	DataflowBuiltInTransformationDataset[]
filter	Filters input record or datapoints based on condition.	DataflowBuiltInTransformationFilter[]
map	Maps input to output message.	DataflowBuiltInTransformationMap[]
schemaRef	Reference to the schema that describes the output of the transformation.	string
serializationFormat	Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json	'Delta' 'Json' 'Parquet'

DataflowDestinationOperationSettings

Name	Description	Value
dataDestination	Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription.	string Constraints: Min length = 1 (required)
endpointRef	Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type.	string (required)

DataflowOperation

Name	Description	Value
builtInTransformationSettings	Built In Transformation configuration.	DataflowBuiltInTransformationSettings
destinationSettings	Destination configuration.	DataflowDestinationOperationSettings
name	Optional user provided name of the transformation.	string Constraints: Max length = 64
operationType	Type of operation.	'BuiltInTransformation' 'Destination' 'Source' (required)
sourceSettings	Source configuration.	DataflowSourceOperationSettings

DataflowProperties

Name	Description	Value
mode	Mode for Dataflow. Optional; defaults to Enabled.	'Disabled' 'Enabled'
operations	List of operations including source and destination references as well as transformation.	DataflowOperation[] (required)

DataflowSourceOperationSettings

Name	Description	Value
assetRef	Reference to the resource in Azure Device Registry where the data in the endpoint originates from.	string
dataSources	List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +.	string Constraints: Min length = 1[] (required)
endpointRef	Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type.	string (required)
schemaRef	Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match.	string
serializationFormat	Content is a JSON Schema. Allowed: JSON Schema/draft-7.	'Json'

ExtendedLocation

Name	Description	Value
name	The name of the extended location.	string (required)
type	Type of ExtendedLocation.	'CustomLocation' (required)

Microsoft.IoTOperations/instances/dataflowProfiles/dataflows

Name	Description	Value
apiVersion	The api version	'2024-11-01'
extendedLocation	Edge location of the resource.	ExtendedLocation (required)
name	The resource name	string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required)
properties	The resource-specific properties for this resource.	DataflowProperties
type	The resource type	'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows'

Terraform (AzAPI provider) resource definition

The instances/dataflowProfiles/dataflows resource type can be deployed with operations that target:

• Resource groups

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.IoTOperations/instances/dataflowProfiles/dataflows resource, add the following Terraform to your template.

resource "azapi_resource" "symbolicname" {
  type = "Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-11-01"
  name = "string"
  body = jsonencode({
    extendedLocation = {
      name = "string"
      type = "string"
    }
    properties = {
      mode = "string"
      operations = [
        {
          builtInTransformationSettings = {
            datasets = [
              {
                description = "string"
                expression = "string"
                inputs = [
                  "string"
                ]
                key = "string"
                schemaRef = "string"
              }
            ]
            filter = [
              {
                description = "string"
                expression = "string"
                inputs = [
                  "string"
                ]
                type = "string"
              }
            ]
            map = [
              {
                description = "string"
                expression = "string"
                inputs = [
                  "string"
                ]
                output = "string"
                type = "string"
              }
            ]
            schemaRef = "string"
            serializationFormat = "string"
          }
          destinationSettings = {
            dataDestination = "string"
            endpointRef = "string"
          }
          name = "string"
          operationType = "string"
          sourceSettings = {
            assetRef = "string"
            dataSources = [
              "string"
            ]
            endpointRef = "string"
            schemaRef = "string"
            serializationFormat = "string"
          }
        }
      ]
    }
  })
}

Property values

DataflowBuiltInTransformationDataset

Name	Description	Value
description	A user provided optional description of the dataset.	string Constraints: Max length = 256
expression	Condition to enrich data from Broker State Store. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string
inputs	List of fields for enriching from the Broker State Store.	string[] (required)
key	The key of the dataset.	string Constraints: Max length = 64 (required)
schemaRef	The reference to the schema that describes the dataset. Allowed: JSON Schema/draft-7.	string

DataflowBuiltInTransformationFilter

Name	Description	Value
description	A user provided optional description of the filter.	string Constraints: Max length = 256
expression	Condition to filter data. Can reference input fields with {n} where n is the index of the input field starting from 1. Example: $1 < 0		$1 > $2 (Assuming inputs section $1 and $2 are provided)	string (required)
inputs	List of fields for filtering in JSON path expression.	string[] (required)
type	The type of dataflow operation.	'Filter'

DataflowBuiltInTransformationMap

Name	Description	Value
description	A user provided optional description of the mapping function.	string Constraints: Max length = 256
expression	Modify the inputs field(s) to the final output field. Example: $1 * 2.2 (Assuming inputs section $1 is provided)	string
inputs	List of fields for mapping in JSON path expression.	string[] (required)
output	Where and how the input fields to be organized in the output record.	string (required)
type	Type of transformation.	'BuiltInFunction' 'Compute' 'NewProperties' 'PassThrough' 'Rename'

DataflowBuiltInTransformationSettings

Name	Description	Value
datasets	Enrich data from Broker State Store. Dataset references a key in Broker State Store.	DataflowBuiltInTransformationDataset[]
filter	Filters input record or datapoints based on condition.	DataflowBuiltInTransformationFilter[]
map	Maps input to output message.	DataflowBuiltInTransformationMap[]
schemaRef	Reference to the schema that describes the output of the transformation.	string
serializationFormat	Serialization format. Optional; defaults to JSON. Allowed value JSON Schema/draft-7, Parquet. Default: Json	'Delta' 'Json' 'Parquet'

DataflowDestinationOperationSettings

Name	Description	Value
dataDestination	Destination location, can be a topic or table name. Supports dynamic values with $topic, $systemProperties, $userProperties, $payload, $context, and $subscription.	string Constraints: Min length = 1 (required)
endpointRef	Reference to the Endpoint CR. Can be of Broker, Kafka, Fabric, ADLS, ADX type.	string (required)

DataflowOperation

Name	Description	Value
builtInTransformationSettings	Built In Transformation configuration.	DataflowBuiltInTransformationSettings
destinationSettings	Destination configuration.	DataflowDestinationOperationSettings
name	Optional user provided name of the transformation.	string Constraints: Max length = 64
operationType	Type of operation.	'BuiltInTransformation' 'Destination' 'Source' (required)
sourceSettings	Source configuration.	DataflowSourceOperationSettings

DataflowProperties

Name	Description	Value
mode	Mode for Dataflow. Optional; defaults to Enabled.	'Disabled' 'Enabled'
operations	List of operations including source and destination references as well as transformation.	DataflowOperation[] (required)

DataflowSourceOperationSettings

Name	Description	Value
assetRef	Reference to the resource in Azure Device Registry where the data in the endpoint originates from.	string
dataSources	List of source locations. Can be Broker or Kafka topics. Supports wildcards # and +.	string Constraints: Min length = 1[] (required)
endpointRef	Reference to the Dataflow Endpoint resource. Can only be of Broker and Kafka type.	string (required)
schemaRef	Schema CR reference. Data will be deserialized according to the schema, and dropped if it doesn't match.	string
serializationFormat	Content is a JSON Schema. Allowed: JSON Schema/draft-7.	'Json'

ExtendedLocation

Name	Description	Value
name	The name of the extended location.	string (required)
type	Type of ExtendedLocation.	'CustomLocation' (required)

Microsoft.IoTOperations/instances/dataflowProfiles/dataflows

Name	Description	Value
extendedLocation	Edge location of the resource.	ExtendedLocation (required)
name	The resource name	string Constraints: Min length = 3 Max length = 63 Pattern = ^[a-z0-9][a-z0-9-]*[a-z0-9]$ (required)
parent_id	The ID of the resource that is the parent for this resource.	ID for resource of type: instances/dataflowProfiles
properties	The resource-specific properties for this resource.	DataflowProperties
type	The resource type	"Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2024-11-01"