Announcing the General Availability of Azure Data Factory

This post is authored by Joseph Sirosh , Corporate Vice President of Information Management & Machine Learning at Microsoft.

I am thrilled to announce the general availability of the Microsoft Azure Data Factory (ADF) today. A key component of the recently announced Microsoft Cortana Analytics Suite, ADF is a fully managed cloud-based data integration service that orchestrates and automates the movement and transformation of data. It is part of a family of state-of-the-art cloud analytics products including Azure Data Catalog, Azure Machine Learning, HDInsight, Power BI, and Azure Stream Analytics that are helping our enterprise customers transform data into intelligent action.

With ADF, existing data processing services can be composed into data pipelines that are highly available and managed in the cloud. These data pipelines can be scheduled to ingest, prepare, transform, analyze, and publish data, and ADF will manage and orchestrate all of the complex data and processing dependencies without human intervention. Solutions can be quickly built and deployed in the cloud, connecting a growing number of on-premises and cloud data sources.

Using ADF, businesses can enjoy the benefits of using a fully managed cloud service without procuring hardware; reduce costs with automatic cloud resource management, efficiently move data using a globally deployed data transfer infrastructure, and easily monitor and manage complex schedule and data dependencies. All of this can be done through an intuitive monitoring and management UI available from the Azure portal, and developers are accelerated through a familiar Visual Studio plug-in experience for solution building and deployment.

Customers like Rockwell Automation, Alaska Airlines, and Milliman use ADF today as their data integration service to bring together their on-premises and cloud-based analytics. Rockwell uses it as part of their solution for predictive maintenance in industrial automation, and Alaska Airlines uses it in personalizing in-flight entertainment for their customers.

Milliman is one of the largest independent actuarial services providers for the insurance industry. They were already using Azure Storage and Azure Compute but needed a way to streamline their cloud and on-premises data integration, and also schedule and manage their Azure Compute jobs together with HDInsight jobs for an automated solution. For each of their numerous clients, they needed a daily aggregation, refinement, and management of terabytes of data for modeling and analytics data processing. Milliman started out by building in-house high-performance grid-computing capabilities and using custom code to stitch together a solution. This required them to maintain a specialized — and expensive — IT infrastructure and their solutions took months to deploy. Now, with ADF as their fundamental orchestration platform, Milliman is already working on enhancements that will further automate and improve end-to-end actuarial workflows, as well as evolving their cloud reporting capabilities with Power BI.

Milliman has now cut IT costs by 30% and reduced reporting time by 95%. At the same time, with the new cost efficiencies gained with the on-demand use of cloud resources, they were able to utilize 600% more compute hours and double their supported customer base. Milliman’s time to solution delivery dropped from months to weeks and their actuaries no longer had to spend 70% of their time on data management. “Actuaries should be focused on analyzing results and creating value for the company, not worrying about massaging data and managing reporting systems,” says Paul Maher, CTO of Milliman’s Integrate Solution.

We are proud to dramatically simplify and reduce the cost of implementing sophisticated analytics solutions for customers like Milliman so they can focus on transforming their data into intelligent actions that give them a competitive advantage.

Milliman is a great example of how enterprise customers can benefit from our cloud analytics services. However, businesses of every size face tough challenges to reach across data of all volumes, variety and velocities to simplify and enrich the analytics experience. Devices, customers and business processes produce a wealth of unused data that we believe can be tapped to create new experiences, business models, efficiencies, and even lead to new inventions given the right analytics tools and capabilities.

Data is pervasive but actionable intelligence is elusive. As a foundational component of the Cortana Analytics Suite, ADF stitches together and orchestrates a comprehensive set of services including Azure HDInsight, Azure ML, Power BI and more to bridge the gap between pervasive data and actionable intelligence.

The Cortana Analytics Suite will provide integrated, easy to build and deploy solution templates that span multiple services to accomplish intelligent tasks — availability and pricing for the suite will be announced in fall 2015. Meanwhile, note that ADF and other component services of the suite are ready to be used today directly from the Azure portal. To learn more about ADF visit www.azure.com/adf. To learn more about the Cortana Analytics Suite visit www.microsoft.com/cortanaanalytics.

Joseph
Follow me on twitter

Comments

• Anonymous
  August 08, 2015
  Good
• Anonymous
  August 13, 2015
  this is awesome, microsoft really seems to be stepping it up since the new CEO took over. Windows 10 rocks, edge rocks, azure rocks!