Azure Development... and Data? -- Architect your Data Story in the Cloud! (Content Spotlight)

First, I want to take a look into the Azure Application Architecture Guide (A3G):

• https://aka.ms/A3G

This is an Azure architecture guide (like a menu of what to use when) for developers!

 

Today I want to drill into the Big Data Architecture Style:

• Big data architecture style

A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional database systems.

Big data solutions typically involve one or more of the following types of workload:

• Batch processing of big data sources at rest.
• Real-time processing of big data in motion.
• Interactive exploration of big data.
• Predictive analytics and machine learning.

 

The new Azure Data Architecture Guide (ADAG) that I helped lead...

• https://aka.ms/ADAG

 

...which is structured around a basic pivot: The distinction between relational data and non-relational data.

ADAG heavily expands on this Big Data architecture style...

• Big data architectures

 

The following diagram shows the logical components that fit into a big data architecture. Individual solutions may not contain every item in this diagram.

Most big data architectures include some or all of the following components:

• Data sources. All big data solutions start with one or more data sources. Examples include:
  • Application data stores, such as relational databases.
  • Static files produced by applications, such as web server log files.
  • Real-time data sources, such as IoT devices.

 

• Data storage. Data for batch processing operations is typically stored in a distributed file store that can hold high volumes of large files in various formats. This kind of store is often called a data lake. Options for implementing this storage include Azure Data Lake Store or blob containers in Azure Storage.
  • See Choosing a big data storage technology in Azure

 

• Batch processing. Because the data sets are so large, often a big data solution must process data files using long-running batch jobs to filter, aggregate, and otherwise prepare the data for analysis. Usually these jobs involve reading source files, processing them, and writing the output to new files. Options include running U-SQL jobs in Azure Data Lake Analytics, using Hive, Pig, or custom Map/Reduce jobs in an HDInsight Hadoop cluster, or using Java, Scala, or Python programs in an HDInsight Spark cluster.
  • See Batch processing
  • See Choosing a batch processing technology in Azure

 

• Real-time message ingestion. If the solution includes real-time sources, the architecture must include a way to capture and store real-time messages for stream processing. This might be a simple data store, where incoming messages are dropped into a folder for processing. However, many solutions need a message ingestion store to act as a buffer for messages, and to support scale-out processing, reliable delivery, and other message queuing semantics. This portion of a streaming architecture is often referred to as stream buffering. Options include Azure Event Hubs, Azure IoT Hub, and Kafka.
  • See Real-time processing

 

• Stream processing. After capturing real-time messages, the solution must process them by filtering, aggregating, and otherwise preparing the data for analysis. The processed stream data is then written to an output sink. Azure Stream Analytics provides a managed stream processing service based on perpetually running SQL queries that operate on unbounded streams. You can also use open source Apache streaming technologies like Storm and Spark Streaming in an HDInsight cluster.
  • See Choosing a stream processing technology in Azure

 

• Analytical data store. Many big data solutions prepare data for analysis and then serve the processed data in a structured format that can be queried using analytical tools. The analytical data store used to serve these queries can be a Kimball-style relational data warehouse, as seen in most traditional business intelligence (BI) solutions. Alternatively, the data could be presented through a low-latency NoSQL technology such as HBase, or an interactive Hive database that provides a metadata abstraction over data files in the distributed data store. Azure SQL Data Warehouse provides a managed service for large-scale, cloud-based data warehousing. HDInsight supports Interactive Hive, HBase, and Spark SQL, which can also be used to serve data for analysis.
  • See Choosing an analytical data store in Azure

 

• Analysis and reporting. The goal of most big data solutions is to provide insights into the data through analysis and reporting. To empower users to analyze the data, the architecture may include a data modeling layer, such as a multidimensional OLAP cube or tabular data model in Azure Analysis Services. It might also support self-service BI, using the modeling and visualization technologies in Microsoft Power BI or Microsoft Excel. Analysis and reporting can also take the form of interactive data exploration by data scientists or data analysts. For these scenarios, many Azure services support analytical notebooks, such as Jupyter, enabling these users to leverage their existing skills with Python or R. For large-scale data exploration, you can use Microsoft R Server, either standalone or with Spark.
  • See Choosing a data analytics (and reporting) technology in Azure

 

• Orchestration. Most big data solutions consist of repeated data processing operations, encapsulated in workflows, that transform source data, move data between multiple sources and sinks, load the processed data into an analytical data store, or push the results straight to a report or dashboard. To automate these workflows, you can use an orchestration technology such Azure Data Factory or Apache Oozie and Sqoop.
  • See Choosing a data pipeline orchestration technology in Azure

 

So please browse the new content and reply to let us know how helpful it was.

The new Azure Data Architecture Guide (ADAG)...

• https://aka.ms/ADAG

 

Remember to keep your head in the Cloud!

- Ninja Ed

Comments

• Anonymous
  March 08, 2018
  Thanks Ed. The A3G is a great resource. I appreciate the hard work that has been put into it.
  • Anonymous
    March 24, 2018
    Thanks Chilberto! This comes from the original Microsoft patterns & practices team. The key contributors were Masashi Narumoto (PM) and Mike Wasson (Content Developer/Tech Writer). Here's the breakdown of these doc sets that I call "Architecture Guides" - http://aka.ms/A3G - For developershttp://aka.ms/ADAG - For data professionalshttp://aka.ms/ACAG - For IT Pros looking to migrate into the cloud And then I also published a few eBooks for IT Pros...http://aka.ms/VDC - Azure Virtual Datacenterhttp://aka.ms/AStack - Azure Stack hybrid configurationSo we're building a rich end-to-end content story for Azure.