Share via

AI Builder and Power Automate: Save data to SharePoint Online by Extracting Form Data from Document

  • Article

Introduction

The AI Builder is the new Power Platform feature that allows us to use Artificial Intelligence to easily automate the processes within a business and predict the outcome.

AI Models

AI Builder currently supports four types of models. Lets have a look at the 4 AI Models

  • Form processing model

Form processing allows you to create and use models that use machine learning technology to extracts data from form which helps reduce the manual effort involved in extracting data from the form

  • Text Classification model

 It allows us to tag elements of text in a company documents so as to use it for spam detection, sentiment analysis and so on. Thus, it enables classifying the text based on the tags assigned to it.

  • Prediction classification model

It predicts yes/no business outcomes by learning to associate historical data patterns with historical outcomes. Based on those results, the prediction model detects learned patterns in new data to predict future outcomes.

  • Object detection model

Object detection, allows us train the model with the images so that it can detect the object based on trained images

Work with Object Detection Model

With AI Builder, we can build and train these AI model suited to our needs, In this article we will take the Form Processing model and train it to extract data from forms. It uses existing forms/documents to learn how to analyse the document based on which the form data would be extracted from the input form. So the accuracy of the data extraction depends on the forms that we have used to train the model.  To get started with it, let’s head to the flow.microsoft.com site. AI Builder can be accessed from the left pane.

In case you are stumbled upon with the below message, go ahead and create a new environment with US as the region .


Click on create a new environment which will open up be the below dialog box

Once the environment is created, head over and create a database.

Now the model options will appear in the Build section of AI Builder.

Form Processing Model

In this demo we will create a form processing model and try to extract the data(Sum Total &) from the invoice document that is uploaded into SharePoint Library. We will then store the extracted data against the document in the same library.

To get started select Form Processing Model and specify a name for it which in this case we will call Invoice Model

Add Documents

You need some sample documents to train your model for the type of forms you'll be working with. AI Builder analyzes the samples you upload to see if you can use them for training.

Once the documents are added, upload them.

Analyse documents

Once the upload is completed, next step is the analysis of the documents.

Depending on the number of documents provided, the analysis could take longer but in most cases it should only take a few minutes.

Select Form Fields

Now we need to select the fields from which we can extract the data.

Select the fields which would be highlighted in green as shown below.

Once we have selected the fields, click on Train so that the model will get used to the sample documents and which fields to be selected from within the form.

Once the training is completed it will show the below success message.

Publish the model so that we can use this Power Apps or Power Automate to build applications.

Create Flow

So as to integrate the model with our implementation, lets create a solution in Flow.

Once the solution is created, Click on New and create a flow within it.

We will use the File is created trigger so that the flow gets invoked when a new document is added

Use the Get File Content action to get document content in base64 format

We will then use the predict action and use the newly created model. Pass the document content as base 64 value :

base64(outputs('Get_file_content')?['body'])

The output of the predict action will be the extracted form fields which we have trained the model to extract.

Test the application

Now lets test the implementation by uploading a document to SharePoint Library

This will kick start the flow and it will use the Predict action and the Invoice Model to extract the Total Amount and Submission date which will be saved back to the library as shown below :

Summary

Thus we saw how we can use AI Builder and Power Automate to extract the form data from the document uploaded in SharePoint Document library and update the data back to the item.