Azure Data Factory JSON Changes in July 2015

Azure Data Factory factories are designed with a series of fairly simple JSON documents and uploaded to Azure using either the web interface, PowerShell, .Net, or Visual Studio.

If you were using the pre-release public preview of Azure Data Factory, you should be aware of a recent change in the SDK,  in order to make the transition as seamless as possible. Recently there have been there is a set of JSON Schema changes in Azure Data Factory that happen when the new PowerShell SDK (update as of July 17, 2015) is downloaded. The various Azure SDKs are released monthly I think, so the best way to know what you are using is to remember the date when you last downloaded it. https://azure.microsoft.com/en-us/downloads/

The hope is that the new JSON format makes more sense to users, so pardon our dust as we evolve toward making it more user friendly to develop with Azure Data Factory.

The changes should be easy to see and managed.

Three things you need to know:

1. When you are in the web portal https://portal.azure.com inside the Data Factory scripts, the changes are signaled with this informative notice. Your JSON should automatically be upgraded. When using the web portal to Author and Deploy, creating new objects, the web editor will know the new version of the JSON schema and can autofix the documents to the new format

The "Author and deploy" editor now uses new JSON format for data factory entities. Note: Properties specific to an entity type are now specified in "typeProperties" object. Type names have been changed. When a draft containing the old JSON format is opened, you will be prompted to automatically upgrade the JSON to the new format.

 

2. The same JSON change impact Visual Studio users. More info TBD

3. We have a freely downloadable tool that helps convert all JSON design documents in one directory as a batch, from the command line console:

• Main page on GitHub here https://github.com/Azure/Azure-DataFactory
• Direct download link https://github.com/Azure/Azure-DataFactory/releases/download/v1.0/Microsoft.Azure.DataFactoryJsonConversionTool.msi
• Example usage: JsonUpgradeTool.exe /sd <source directory> /td <target directory> [/v <target API version>]

 

Documentation References to observe

The documentation team has done a great job documenting the exact changes in the JSON schema that may impact your designs.

1. Release Notes https://azure.microsoft.com/en-us/documentation/articles/data-factory-release-notes/#notes-for-07172015-release-of-data-factory
2. JSON Documentation https://msdn.microsoft.com/en-us/library/azure/dn835050.aspx
3. The old schemas of the JSON format are mentioned in parallel for reference if you need to keep using the old PowerShell SDK version. https://msdn.microsoft.com/en-us/library/azure/mt185729.aspx

Hope this helps – we will update this blog if we notice any issues! ~JasonH