Infrastructure as Code with existing Azure Data Factory
I create an ARM template from already existing Azure Data Factory. I'd like to share the tips for that.
The following link is the useful resource to develop ARM template of Azure Data Factory from scratch.
1. Getting Azure Data Factory ARM Template
The easiest way is to use Azure Portal. Try to create a new Azure Data Factory from the portal. Unfortunately, you can not get "Automation Script" of Azure Data Factory from an existing Resource Group.
a. Click Automation options
b. Click Download button
Now you have Data Factory ARM template with some scripts.
2. Adding DataFactory configurations
If you have an Azure Data Factory instance, you might create several linked services, datasets, and Pipelines. You need to add these configurations to the ARM template which we downloaded at 1. Getting Azure Data Factory ARM Template.
You can get the configuration at the Data factory dashboard. Actions > Author and deploy. Then you can see the ARM template for the existing resource.
2.1. Copy and paste the configuration
Just copy paste the configurations to the ARM template which we downloaded at 1. However, it doesn't work as it is.
The ARM template structure is like this.
{
"$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string"
},
"location": {
"type": "string"
},
"apiVersion": {
"type": "string",
"defaultValue": "2015-01-01-preview"
}
},
"resources": [
{
"apiVersion": "[parameters('apiVersion')]",
"name": "[parameters('name')]",
"location": "[parameters('location')]",
"type": "Microsoft.DataFactory/dataFactories",
"properties": {},
"resources": [
COPY AND PASTE THE CONFIGURATIONS HERE!
]
}
]
}
No such thing like "Resources" after the "properties." You need to add that. Then copy and paste the configuration from the existing Azure Data Factory.
2.2. Edit
You need to edit several things from the configurations.
2.2.1. remove "hubname"
The existing configuration includes "hubname". Remove it. This is an example of mine.
{
"name": "AzureSqlLinkedService",
"properties": {
"description": "Azure Sql Database for testing Azure Data Factory telemetry",
"hubName": "spikeadf_hub", <----- REMOVE THIS
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "xxxxxxxxxxxxxx"
}
}
}
2.2.2. check the configuration of the connectionString
Your connection string will be masked in some part. Please convert it to the original value.
"connectionString": "Data Source=tcp:aaaaaa.database.windows.net,1433;Initial Catalog=aaaaa;Integrated Security=False;User ID=someuser@aaaaa.database.windows.net;Password=**********;Connect Timeout=30;Encrypt=True"
2.3. Debugging
Sometimes, you might get an error. You can see that is very hard to understand.
Don't worry, just go to the Azure portal and you can see the new Azure Data Factory is created.
If you check the "Actions > Author and deploy," you can see the detail cause of the Error.
2.4. Deployment
The ARM template which you downloaded at 1. It includes "./deploy.ps1" Just execute on your PowerShell.
Conclusion
Now you successfully create an ARM template for the existing Azure Data Factory instance. Enjoy.