The following architecture expands on the analytics end-to-end with Azure Synapse Analytics scenario. Use this architecture to train a custom machine learning model in Azure Machine Learning and implement it with a custom application that you build by using Microsoft Power Platform.
Architecture
Download a Visio file of this architecture.
Workflow
The following dataflow corresponds to the previous diagram.
- Ingest
- Store
- Train and deploy a model
- Consume
Ingest
Use Azure Synapse Analytics pipelines to pull batch data from various sources, both on-premises and in the cloud. This lambda architecture has two data ingestion flows: streaming and batch.
Streaming: Streaming data flows, such as big data streams and IoT devices, are in the upper half of the preceding architecture diagram.
You can use Azure Event Hubs or Azure IoT Hub to ingest data streams generated by client applications or IoT devices. Event Hubs or IoT Hub ingests and stores streaming data and preserves the sequence of events that it receives. Consumers can connect to hub endpoints to retrieve messages for processing.
Batch: In the lower half of the architecture diagram, data is ingested and processed in batches, including:
- Unstructured data, like video, images, audio, and free text.
- Semi-structured data, like JSON, XML, CSV, and logs.
- Structured data, like relational databases and Azure data services.
Azure Synapse Link seamlessly integrates Azure Cosmos DB and Azure Synapse Analytics.
Azure Synapse Analytics pipelines can be triggered based on a predefined schedule or in response to an event. They can also be invoked by calling REST APIs.
Store
Ingested data can land directly in raw format and then be transformed on Azure Data Lake Storage. After the data is curated and transformed to relational structures, it can be presented for consumption in Azure Synapse Analytics.
Train and deploy a model
Machine Learning provides an enterprise-grade machine learning service that you can use to quickly build and deploy models. It provides a low-code designer, automated machine learning, and a hosted Jupyter notebook environment to users at any skill level. Models can be deployed either as real-time endpoints on Azure Kubernetes Service (AKS) or as a Machine Learning managed endpoint. For batch inferencing of machine learning models, you can use Machine Learning pipelines.
Consume
A batch or real-time model that's published in Machine Learning can generate a REST endpoint that can be consumed in a custom application that's built by using the low-code Power Apps platform. You can also call a real-time Machine Learning endpoint from a Power BI report to present predictions in business reports.
Note
Both Machine Learning and Microsoft Power Platform stacks have a range of built-in connectors to help ingest data directly. These connectors might be useful for a one-off minimum viable product (MVP). However, the Ingest and Store sections of the architecture describe the role of standardized data pipelines to source and store data from different sources at scale. The enterprise data platform teams typically implement and maintain these patterns.
Components
You can use the following components.
Microsoft Power Platform services
- Power Platform is a set of tools to analyze data, build solutions, automate processes, and create virtual agents. It includes Power Apps, Power Automate, Power BI, and Microsoft Copilot Studio (formerly Power Virtual Agents). Use these low-code tools to develop the user experience and integration with data sources.
- Power Apps is a suite of apps, services, connectors, and a data platform. It provides a rapid application development environment to build custom apps for your business needs. Use the low-code design environment to develop the user interface, collect inputs from the user, and present prediction outputs.
- Power Automate is a service that helps you create automated workflows between your favorite apps and services. Use it to synchronize files, get notifications, and collect data. The workflow uses inputs from the user interface to run machine learning activities like predictions and return results that display in Power Apps.
- Power BI is a collection of software services, apps, and connectors that work together to turn your unrelated sources of data into coherent, visually immersive, and interactive insights. This environment is where you design reports and dashboards and host them for users.
Azure services
- Machine Learning is an enterprise-grade machine learning service that you can use to build and deploy models quickly. Use its low-code designer, automated machine learning, and a hosted Jupyter notebook environment to develop machine learning models.
- Machine Learning managed endpoints are online endpoints that enable you to deploy your model without having to create and manage the underlying infrastructure. Applications like Power Apps use the hosted endpoint for interactive predictions.
- Data Lake Storage is a Hadoop-compatible file system. It has an integrated hierarchical namespace and the massive scale and economy of Azure Blob Storage. It provides the storage layer for data use in the machine learning model training process.
- Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics. This integrated capability stores and transforms data that's used in machine learning and report creation activities.
- Event Hubs and IoT Hub ingest data streams that client applications or IoT devices generate. Then they ingest and store streaming data while preserving the sequence of events received. Consumers can connect to the hub endpoints to retrieve messages for processing.
Platform services
To improve the quality of your Azure solutions, follow the recommendations and guidelines in the Azure Well-Architected Framework. The framework consists of five pillars of architectural excellence:
- Reliability
- Security
- Cost Optimization
- Operational Excellence
- Performance Efficiency
To create a design that follows these recommendations, consider the following services:
- Microsoft Entra ID provides identity services, single sign-on, and multifactor authentication across Azure workloads.
- Microsoft Cost Management provides financial governance over your Azure workloads.
- Azure Key Vault helps you manage credentials and certificates more securely.
- Azure Monitor collects, analyzes, and displays telemetry from your Azure resources. Use Azure Monitor to proactively identify problems to maximize performance and reliability.
- Microsoft Defender for Cloud helps strengthen and monitor the security posture of your Azure workloads.
- Azure DevOps and GitHub help you enforce automation and compliance of your workload development and deployment pipelines for Azure Synapse Analytics and Machine Learning.
- Azure Policy helps you implement organizational standards and governance for resource consistency, regulatory compliance, security, cost, and management.
Alternatives
A machine learning MVP benefits from speed to outcome. In some cases, pretrained Azure AI services can meet the needs of a custom model. In other cases, Power Apps AI Builder might provide a fit-for-a-purpose model.
Scenario details
A general trend in technology is the increasing popularity of citizen AI roles. These roles include business practitioners who want to improve business processes through the application of machine learning and AI technologies. A key factor that drives this trend is the growing maturity and availability of low-code tools to develop machine learning models.
Because of a well-known high failure rate to such initiatives, the ability to rapidly prototype and validate an AI application in a real-world setting becomes a key enabler to a fail-fast approach. The two key tools for developing models that modernize processes and drive transformative outcomes are a machine learning toolkit for all skill levels and a low-code application development toolkit.
A machine learning toolkit for all skill levels:
- Supports no-code to fully coded machine learning development.
- Has a flexible, low-code graphical user interface (GUI).
- Enables users to rapidly source and prepare data.
- Enables users to rapidly build and deploy models.
- Has advanced, automated machine learning capabilities for machine learning algorithm development.
A low-code application development toolkit:
- Enables users to build custom applications and automation workflows.
- Creates workflows so that consumers and business processes can interact with a machine learning model.
Machine Learning fulfills the role of a low-code GUI for machine learning development. It has automated machine learning and deployment to batch or real-time endpoints. Microsoft Power Platform, which includes Power Apps and Power Automate, provides the toolkits to rapidly build a custom application and workflow that implements your machine learning algorithm. Business users can build production-grade machine learning applications to transform legacy business processes.
Potential use cases
These toolkits minimize the time and effort needed to prototype the benefits of a machine learning model on a business process. You can easily extend a prototype to a production-grade application. The uses for these techniques include:
Manufacturing operations with legacy applications that use outdated deterministic predictions. These operations can benefit from the improved accuracy of a machine learning model. Improving accuracy requires both a model and a development effort to integrate with legacy systems on-premises.
Call center operations with legacy applications that don't adjust when data drifts. Models that automatically retrain might provide a significant uplift in churn prediction or risk profiling accuracy. Validation requires integration with existing customer relationship management and ticket management systems. Integration can be expensive.
Considerations
These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that you can use to improve the quality of a workload. For more information, see Well-Architected Framework.
Reliability
Reliability helps ensure that your application can meet the commitments that you make to your customers. For more information, see Design review checklist for Reliability.
Most of the components used in this example scenario are managed services that scale automatically. The availability of the services used in this example varies by region.
Apps based on machine learning typically require one set of resources for training and another set for serving. Resources required for training generally don't need high availability because live production requests don't directly use these resources. Resources required for serving requests need high availability.
Cost Optimization
Cost Optimization focuses on ways to reduce unnecessary expenses and improve operational efficiencies. For more information, see Design review checklist for Cost Optimization.
Azure pricing: Microsoft infrastructure as a service and platform as a service services on Azure use a consumption-based pricing model. They don't require a license or subscription fee. In general, use the Azure pricing calculator to estimate costs. For more information, see Cost Optimization.
Microsoft Power Platform pricing: Power Apps, Power Automate and Power BI are software as a service applications and have their own pricing models, including per-app plans and per-user plans.
Operational Excellence
Operational Excellence covers the operations processes that deploy an application and keep it running in production. For more information, see Design review checklist for Operational Excellence.
This example uses DevOps practices to orchestrate its end-to-end approach. The machine learning operations guide describes best practices for adopting machine learning operations.
You can apply DevOps automation to the Microsoft Power Platform solution provided in this example. For more information, see Microsoft Power Platform build tools for Azure DevOps.
Example scenario
Consider the following business scenario. A field agent uses an app that estimates a car's market price. You can use Machine Learning to quickly prototype a machine learning model of this app. Use a low-code designer and machine learning features to create the model, and then deploy it as a real-time REST endpoint.
The model might prove the concept, but a user has no easy way to consume a model that's implemented as a REST API. The following diagram shows how Microsoft Power Platform can help close this gap.
The following image shows a user interface for the app, which was created in Power Apps by using the low-code interface that Power Apps provides.
You can use Power Automate to build a low-code workflow to parse the user's input, pass it to the Machine Learning endpoint, and retrieve the prediction. You can also use Power BI to interact with the Machine Learning model and create custom business reports and dashboards.
More scenarios
Consider the following scenarios.
Deploy to Microsoft Teams
You can also deploy the Power Apps developed user interface in the preceding example to Microsoft Teams. Teams provides a great distribution channel for your apps and gives your users a collaborative app experience. For more information about how to deploy an app to Teams by using Power Apps, see Publish your app in Teams.
Consume the API from multiple apps and automations
In this example, we configure a Power Automate cloud flow to consume the REST endpoint as an HTTP action. We can instead set up a custom connector for the REST endpoint and consume it directly from Power Apps or from Power Automate. This approach is useful when we want multiple apps to consume the same endpoint. It also provides governance by using the connector data loss prevention policy in the Microsoft Power Platform admin center. To create a custom connector, see Use a custom connector from a Power Apps app. For more information, see Data loss prevention policies.
Contributors
Microsoft maintains this article. The following contributors wrote this article.
Principal author:
- Vyas Dev Venugopalan | Sr. Specialist - Azure Data & AI
To see nonpublic LinkedIn profiles, sign in to LinkedIn.