Solution ideas
This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation of this architecture. Use this article as a starting point to design a well-architected solution that aligns with your workload's specific requirements.
This article presents a solution for automating data analysis and visualization using artificial intelligence (AI). Core components in the solution are Azure Functions, Azure AI services, and Azure Database for MySQL.
Architecture
Download a Visio file of this architecture.
Dataflow
- An Azure function activity allows you to trigger an Azure Functions App in the Azure Data Factory pipeline. You create a linked service connection and use the linked service with an activity to specify the Azure function you want to execute.
- Data comes from various sources such as Azure Storage or Azure Event Hubs for high-volume data. When the pipeline receives new data, it triggers the Azure Functions App.
- The Azure Functions App calls the Azure AI services API to analyze the data.
- The Azure AI services API returns the results of the analysis in JSON format to the Azure Functions App.
- The Azure Functions App stores the data and results from the Azure AI services API in Azure Database for MySQL.
- Azure Machine Learning uses custom machine learning algorithms to provide further insights into the data.
- The MySQL database connector for Power BI provides options for data visualization and analysis in Power BI or a custom web application.
Components
- Data Factory
- Functions
- Event Hubs
- Blob Storage
- Cognitive Services
- Cognitive Service for Language
- Azure Database for MySQL
- Machine Learning studio
- Power BI
Alternatives
- This solution uses Azure Functions to process data as it's received. If a large amount of data already exists in the data source, consider forms of batch processing.
- Azure Stream Analytics provides event processing for high volumes of fast-streaming data that arrives simultaneously from multiple sources. Stream Analytics also supports integration with Power BI.
- To compare this solution with alternatives, see the following resources:
Scenario details
The automated pipeline uses the following services to analyze the data:
- Azure AI services uses AI for question answering, sentiment analysis, and text translation.
- Azure Machine Learning supplies machine-learning tools for predictive analytics.
The solution automates the delivery of the data analysis. A connector links Azure Database for MySQL with visualization tools like Power BI.
The architecture uses an Azure Functions App to ingest data from multiple data sources. It's a serverless solution that offers the following benefits:
- Infrastructure maintenance: Azure Functions is a managed service that allows developers to focus on innovative work that delivers value to the business.
- Scalability: Azure Functions provides compute resources on demand, so function instances scale as needed. As requests fall, resources and application instances drop off automatically.
Potential use cases
This solution is ideal for organizations that run predictive analytics on data from various sources. Examples include organizations in the following industries:
- Finance
- Education
- Telecommunications
Considerations
For most features, the Azure AI Language API has a maximum size of 5120 characters for a single document. For all features, the maximum request size is 1 MB. For more information about data and rate limits, see Service limits for Azure Cognitive Service for Language.
Previous versions of this solution used the Azure AI services Text Analytics API. Azure AI Language now unifies three individual language services in Azure AI services: Text Analytics, QnA Maker, and Language Understanding (LUIS). You can easily migrate from the Text Analytics API to the Azure AI Language API. For instructions, see Migrate to the latest version of Azure Cognitive Service for Language.
Contributors
This article is maintained by Microsoft. It was originally written by the following contributor.
Principal author:
- Matt Cowen | Senior Cloud Solution Architect
To see non-public LinkedIn profiles, sign in to LinkedIn.
Next steps
- Functions
- Azure Function activity in Azure Data Factory
- Data Factory
- Event Hubs
- Blob Storage
- Cognitive Services
- Azure Cognitive Service for Language
- Azure Database for MySQL
- Azure Machine Learning
- Power BI