Building Azure Stream Analytics Resources via ARM Templates - Part 2 - The Template.
Check out Part 1 of this post.
In this 2nd part of my post on building Azure Stream Analytics Resources using ARM Templates we are going to get down and actually build something :)
You've probably ended up here because you can't find a lot of really good help anywhere else on how to create Stream Analytics resources in ARM templates. Here is a good reason for this....it's not officially supported. Yes that's right, the reason you can't find any proper documentation on the MSDN or Azure documentation website is that it doesn't exist. Even the formal ARM template schemas which we discussed in part 1 don't exist.
For example, take a look at this snippet from a Stream Analytics template which you might have seen elsewhere on the intertubes:
{
"apiVersion": "2016-03-01",
"name": "[variables('streamAnalyticsJobName')]",
"location": "[resourceGroup().location]",
"type": "Microsoft.StreamAnalytics/StreamingJobs",
"dependsOn": [
"[concat('Microsoft.Devices/IotHubs/', parameters('iotHubName'))]"
],
.........
See that apiVersion bit and the type bit? Go and take a look at the schema reference file Microsoft.StreamAnalytics.json in the location I told you about last time, i.e. in the https://github.com/Azure/azure-resource-manager-schemas/tree/master/schemas/2016-03-01 folder.
Did you enjoy reading the file? Of course you didn't....it doesn't exist :( and so here-in lies the problem.
[Update on 11th September 2018 - I've just looked again and the file was added on the 14th October 2017 - https://github.com/Azure/azure-resource-manager-schemas/commits/master/schemas/2016-03-01/Microsoft.StreamAnalytics.json]
Quite why this was never published or made official is something I'm not able to answer. My assumption is that either 1) there is some (possibly technical) blocker which is preventing it from being published, 2) it's not seen as a priority. In either case you can provide feedback at https://feedback.azure.com/forums/270577-stream-analytics which is the official location where the Stream Analytics team look for new feature requests. There is already an item opened for just this issue: https://feedback.azure.com/forums/270577-stream-analytics/suggestions/18611194-make-stream-analitics-fully-configurable-from-arm - you'll probably want to go and vote it up.
But it's not all sad news :)
After a few days playing with examples I could find, some trial and error and a good dose of reverse engineering using https://resources.azure.com I was able to get a fully working template which created a Stream Analytics job and created both input and output locations (an IoTHub as input and PowerBI Streaming Dataset as output) . Creating the job was the easy bit, there are lots of places on the web which show this. Adding Inputs and Outputs took some more effort but I got it working. The only things which where not possible using a template where:-
- Starting the job. Once the input, query and output settings have been specified you'll have to manually start the job in the Azure Portal UI or by calling the REST API.
- Fully configuring the PowerBI OAuth credentials. My output is configured as a PowerBI (PBI) Streaming Dataset. In order to connect and send data the Stream Analytics jobs needs the correct permissions - specifically it needs an OAuth token. You can't specify this in the template because it's generated on the fly by AzureAD. You will need to deploy the template which creates the job and output which thinks it's connected to PBI using the user principal name you supply. Then you will need to via the Azure Portal go in to the output definition, and click on "Re-authorize". This makes the AzurePortal talk to AzureAD and obtain an OAuth token (which is then stored inside the job definition). It will be used whenever the job output needs to send data to PBI.
Of course that 2nd issue might not be a problem for you if you want to output to somewhere other than PBI.
How did I work out the template when it's not documented?
Blah..blah..to do when I get time/reminders/comments/requests etc.
How do I connect to input X or send to output Y?
Chances are you want to read from or send to another input or output and I didn't cover that here. If you need help, leave a comment and as soon as I get time I'll add it to this document (or probably start a part 3).
Show me your fully working example.
The original goal of the template was to:
- Read messages from an IoT Hub. Messages are JSON in UTF-8.
- Perform a simple query. Err, it just copies the input to the output. You can add your own query in place of mine.
- Output to a PowerBI Streaming Dataset. You specify in the template output the name of the dataset and table. You will see these appear automatically in the PBI Portal once the first message has been sent there (i.e. they don't get created when you create your Stream Analytics job output and configure the authentication). This caught me out a few times.
This template creates the IoT Hub, the Stream Analytics job, the Input to read from the IoT Hub, and defines the Output to send stuff to PBI. I've left my original parameters and variable references in place and I'm sure you'll be able to replace those with something suitable.
- iotHubName: The name of the IoTHub you want to create. Must be unique across the whole of Azure.
- userDisplayName: A human readable name to associate the OAuth token for connecting to the PBI Streaming Dataset. It doesn't seem to really matter what you enter here but something is required.
- userLoginName: The username in user@domain.com format that you want to use for authenticating against PowerBI.
- streamAnalyticsJobName: The name of the main job which will be created.
Now here it is in all it's glory......
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"iotHubName": {
"type": "string",
"minLength": 3,
"defaultValue": "wibblehub66",
"metadata": {
"description": "Name of the IoT Hub instance to provision."
}
},
"userDisplayName": {
"type": "string",
"minLength": 3,
"defaultValue": "Billy Bunter",
"metadata": {
"description": "Your name for connecting Stream Analytics to PowerBI."
}
},
"userLoginName": {
"type": "string",
"minLength": 3,
"defaultValue": "billybunter@microsoft.com",
"metadata": {
"description": "Your loginname/alias for connecting Stream Analytics to PowerBI."
}
}
},
"variables": {
"streamAnalyticsJobName": "MyAceJobName"
},
"resources": [
{
"apiVersion": "2016-02-03",
"location": "[resourceGroup().location]",
"name": "[parameters('iotHubName')]",
"type": "Microsoft.Devices/IotHubs",
"properties": {
"eventHubEndpoints": {
"events": {
"retentionTimeInDays": 1,
"partitionCount": 2
}
},
"cloudToDevice": {
"defaultTtlAsIso8601": "PT1H",
"maxDeliveryCount": 10,
"feedback": {
"maxDeliveryCount": 10,
"ttlAsIso8601": "PT1H",
"lockDurationAsIso8601": "PT60S"
}
},
"location": "[resourceGroup().location]"
},
"sku": {
"name": "S1",
"capacity": 2
}
},
{
"apiVersion": "2016-03-01",
"name": "[variables('streamAnalyticsJobName')]",
"location": "[resourceGroup().location]",
"type": "Microsoft.StreamAnalytics/StreamingJobs",
"dependsOn": [
"[concat('Microsoft.Devices/IotHubs/', parameters('iotHubName'))]"
],
"properties": {
"sku": {
"name": "standard"
},
"outputErrorPolicy": "stop",
"eventsOutOfOrderPolicy": "adjust",
"eventsOutOfOrderMaxDelayInSeconds": 0,
"eventsLateArrivalMaxDelayInSeconds": 5,
"dataLocale": "en-US",
"inputs": [
{
"Name": "IoTHub",
"Properties": {
"DataSource": {
"Properties": {
"iotHubNamespace": "[parameters('iothubname')]",
"sharedAccessPolicyKey": "[listkeys(resourceId('Microsoft.Devices/IotHubs/IotHubKeys',parameters('iothubname'), 'iothubowner'),'2016-02-03').primaryKey]",
"sharedAccessPolicyName": "iothubowner",
"endpoint": "messages/events"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
},
"Type": "Stream"
}
}
],
"transformation": {
"name": "Transformation",
"properties": {
"streamingUnits": 1,
"query": "SELECT\r\n *\r\nINTO\r\n [PBI]\r\nFROM\r\n [IoTHub]"
}
},
"outputs": [
{
"name": "PBI",
"properties": {
"dataSource": {
"type": "PowerBI",
"outputPowerBISource": {
"dataSet": "HeartDataSet",
"table": "HeartDataTable",
"groupId": "",
"groupName": "My Workspace",
"refreshToken": "dummytoken",
"tokenUserDisplayName": "[parameters('userDisplayName')]",
"tokenUserPrincipalName": "[parameters('userLoginName')]"
},
"properties": {
"dataSet": "HeartDataSet",
"table": "HeartDataTable",
"groupId": "",
"groupName": "My Workspace",
"refreshToken": "dummytoken",
"tokenUserDisplayName": "[parameters('userDisplayName')]",
"tokenUserPrincipalName": "[parameters('userLoginName')]"
}
},
"serialization": null,
"diagnostics": null
}
}
]
}
}
],
"outputs": {
"IoTHub": {
"type": "object",
"value": "[reference(resourceId('Microsoft.Devices/IotHubs',parameters('iotHubName')), '2016-02-03')]"
},
"SharedAccessPolicyKey": {
"value": "[listkeys(resourceId('Microsoft.Devices/IotHubs/IotHubKeys',parameters('iothubname'), 'iothubowner'),'2016-02-03').primaryKey]",
"type": "string"
},
"Hostname": {
"value": "[reference(resourceId('Microsoft.Devices/IotHubs',parameters('iotHubName')), '2016-02-03').hostname]",
"type": "string"
}
}
}
Comments
- Anonymous
July 25, 2017
Thank you Dave, very helpful! Can only imagine the hassle you've gone through.Could you extend the ARM template with Functions in the ASA job? - Anonymous
September 11, 2017
The comment has been removed - Anonymous
October 01, 2017
Is it possible to reference the code from a Visual Studio Azure Stream Analytics Project instead of inline the query?I don't want to copy my code from the script.asaql file into "query": "SELECT\r\n *\r\nINTO\r\n [PBI]\r\nFROM\r\n [IoTHub]". - Anonymous
October 05, 2017
Hi Dave,Great article! Incredibly useful :)I'm trying to connect up Cosmos DB as an output - any tips on how to configure the connection?Thanks!Anita - Anonymous
November 22, 2017
Now it's there:https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2016-03-01/Microsoft.StreamAnalytics.json - Anonymous
March 05, 2018
Thanks so much for this! - Graduate Developer PS. Please write about how you worked out the template and how to connect the input and output stuff. Once again, thank you! :) - Anonymous
March 12, 2018
This is a great bit of work. I'm disappointed that stream analytics isn't properly lit up for ARM templates, I've added my votes to the issue request and bugged people at MS too.