Share via

Power App and Power Automate: Extracting Text from Images

  • Article

Introduction

AI Builder is a Power Platform capability that provides you the AI models to take leverage of artificial intelligence to optimize your business as well as automate process. We can either use the already existing prebuilt models that suits our needs to use the Power of AI in Power Apps or Power Automate or create our own custom AI models to suit some specific business needs. You can read more about AI Models in the Official documentation here. The out of the box prebuilt available models in AI Builder are:

  • Business card reader    
  • Category classification     
  • Entity extraction                   
  • ID reader
  • Invoice Processing          
  • Key phrase extraction    
  • Language detection              
  • Receipt processing            
  • Sentiment analysis                
  • Text recognition                    
  • Text translation      

In this article we will see how to use the AI Builder Model – Text recognition to extract text from an image in unison with a Power App.

Scenario

We will try to create a Power App that can used on the fields for tracking speeding cars, so that we can use the app to capture the Registration Number of the vehicle. We will then use the Cognitive Analysis ability of AI Builder to extract the number from the image and store it in the back end for identifying the offender and sending them notifications.

Implementation

As the front end, we will create a Power App to capture the speeding image and any other needed information regarding the speeding offence. In the middleware, we will use Power Automate and AI Builder to extract information from the captured speeding image. We will then store all the information back to a SharePoint back end list.

Create the SharePoint List

We will create a SharePoint list with the below minimal columns to save the information 

Create the Power Automate

Let’s create a flow that will accept the captured speeding image and extract the text from it. We will create the flow with the Power Apps trigger and add the Recognize text in an image AI Builder action to the flow.

Click on Ask in PowerApps which will create a variable by the name “RecognizetextinanimageorPDFdocument_Image” which can be used to pass the image from Power App to Flow.The image variable passed from the Power App which contains the base 64 encoded image will be used as the input for the AI builder action which will process it and output the results body which contains the extracted text.

The output of the AI Builder action returns the result in the below structure where the Extracted Text is embedded inside the “Lines” array

So as to extract the image text, we will parse the body using the below expression which will get the extracted text and assign to the variable named SpeedingNumber.

outputs('Recognize_text_in_an_image_or_a_PDF_document')?['body/responsev2/predictionOutput/results'][0]?['lines'][0]?['text']

Finally, we will add a Respond to a Power App action to send back the extracted Speeding Number

Build the Power App

In the Power App, we will add the Camera Control, which on clicking will take the snap of the speeding vehicle. The captured snapshot is shown in the screen using an image control. Followed by that, we have a text area that shows the extracted speeding vehicles number which we get back from the previously created flow. Finally, we also have an field to capture the current time.

Capturing the Image

We will set the Image property of the Image Control to the Photo attribute of the Camera control so that on click of the camera, the image is shown in the image control.

 

In the OnSelect of the Camera Control,we will Run the Flow using <FlowName>.Run function and pass the Snapped image as the parameter to the flow which will be accepted by the variable that we had registered inside the flow. Since the flow outputs the Speeding number, we will get that and assign it to a variable using the Set function.

Set(SpeedingNumber,PowerAppSpeedingCamera.Run(Camera_SpeedingCamera.Photo).speedingnumber)

Finally, the extracted image text in the variable is assigned to the Text property of the text box to display it in the screen.

 

We have already made a data connection to the SharePoint List which we can see in the left pane. Let’s save the offence data to SharePoint list using Patch function

Patch('Speeding Tickets',{'Vehicle Number':Label_SpeedingNumber.Text,'Offence Date':Label_OffenceDate.Text,'Speeding Snapshot':Camera_SpeedingCamera.Photo})

This will save the data to the back end and for time being we are saving the Image as base64 text in the SharePoint List. We can use Power Automate to convert it into image and store as well.

Summary

Thus we saw how to create a Speeding Ticket Power App that takes snap of the speeding vehicles number plate and uses AI Builder’s cognitive ability to extract text from the image and save the offence to the SharePoint List.