Automate document processing with Form Recognizer and Logic Apps (Part 1)

Processing of forms and documents is part of several scenarios both in business and in everyday life. Manual data extraction from documents, either in electronic or printed format, is time-consuming, costly, and error-prone.

Azure Form Recognizer is an Applied AI Service that enables you to extract text, table data, key-value pairs, and layout information from forms and documents. In this two-part series, you will learn how to build an end-to-end document processing automation solution utilizing Azure Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI. In the first part, you will be introduced to the Form Recognizer’s features and train a custom model for extracting key-value pairs from customer service review forms.

You will learn how to:

• Provision a Form Recognizer resource.
• Train a custom model in Form Recognizer Studio.
• Compose two custom template models together into a single model.

To build this solution, you will need an Azure subscription. If you don’t have one, you can sign up for an Azure free account.

What is Azure Form Recognizer?

Azure Form Recognizer is a cloud-based service that applies machine learning-based Optical Character Recognition (OCR) and document understanding technologies to extract data from forms and documents. It provides numerous pre-built models for analyzing common documents, such as invoices, receipts, and business cards, while you can also build custom models to analyze documents specific to your business.

Study the following sketch note to learn more about the Form Recognizer’s features.

Scenario details

Consider that you work in a company that provides customer service to a variety of customers. Every day, customers leave reviews about their experiences with the customer service they received. You need to analyze these reviews in order to identify areas of improvement and to track customer satisfaction. These reviews are submitted as paper forms or PDFs and may include up to two different form types. To make this process easier, we need to use a single service to analyze all the reviews.

In the following image, you can see the workflow that we will build to automate the process of extracting and analyzing customer service reviews. In this post, we will focus on the first step of the workflow: Train a model in Azure Form Recognizer.

Create a Form Recognizer resource

To use Form Recognizer, you need to create a Cognitive Services or a Form Recognizer resource. In this article, I’ll show you how to create a single-service resource in the Azure portal.

1. Sign-in to the Azure portal and select Create a resource.

2. Search for Form Recognizer and then click Create.

3. Create a Form Recognizer resource with the following settings:

   • Subscription: Your Azure subscription.
   • Resource group: Select an existing resource group or create a new one.
   • Region: Choose any available region, for example, West Europe.
   • Name: This would be your custom domain name in your endpoint. Enter a unique name.
   • Pricing tier: Free F0.
   

4. Select Review + Create and then select the Create button and wait for the deployment to complete.

5. Navigate to the Form Recognizer Studio and sign in.

Additional resources for custom projects

To train a custom model, in addition to the Form Recognizer resource, you need to create an Azure Storage Account and a container to upload your training dataset.

1. In the Azure Portal, search for Storage accounts and then select Create.

2. Create a storage account with the following settings:

   • Subscription: Your Azure subscription.
   • Resource group: Select the resource group that you created in the previous step.
   • Storage account name: Enter a unique name.
   • Region: Choose any available region, for example, West Europe.
   • Performance: Standard.
   • Redundancy: Locally-redundant storage (LRS).
   

3. Select Review + Create and then select the Create button and wait for the deployment to complete. Once the deployment is complete, navigate to your storage account.

4. In the left pane, under Data storage, select Containers.

5. Create a new container. Select a name and set the public access level to Container.

   

6. In the left pane, under Settings, select Resource sharing (CORS). Create a new CORS entry and set the Allowed origins to https://formrecognizer.appliedai.azure.com.

   

Now you can upload your sample documents set to the container that you created.

Create a custom project

1. In the Form Recognizer Studio, select the Custom extraction models card.

2. Create a new project and specify the following details:

   • Project name: Customer Service Reviews - Set 1
   • Configure service resource: Select your subscription, resource group, and Form Recognizer resource. In the API version, you can choose the 2022-08-31 (General Availability) API.
   • Connect training data source: Select your subscription, resource group, storage account, and blob container. If you’ve organized your data in folders, specify the folder path in the corresponding field.
   

3. Select Create project.

Label data and train model

1. In the Label data view, define the labels that you want to extract from your documents and their types (field, selection mark, signature, or table). For labels of type Field, you can additionally specify their sub-type, such as string or number.

2. Select a document from your dataset. Form Recognizer will run the document through the Layout API to extract text and layout information.

3. Then, you can start labeling your documents by selecting the text or selection mark in the document and choosing the appropriate label from the drop-down list.

   When labeling selection marks, select only the selection mark, without the surrounding text.

4. Once you have labeled at least five documents, select the Train option.

5. In the Train a new model window, ender a Model ID and set the Build Mode to Template.

Compose models

Once you’ve trained a template model for each document layout variation, you can compose them together into a single model. Then, you can use the composed model in your applications.

Summary and next steps

In this article, you learned how to train a custom template model in Azure Form Recognizer and create a composed model. In the next post, you will use the composed model in an Azure Logic Apps workflow to build an automated form processing solution.

If you want to improve the accuracy of the template model that you’ve trained, you can check out the following resources:

• Best practices: Generating Form Recognizer labeled dataset - Microsoft Docs
• Tips for labeling custom model datasets - Microsoft Docs

Clean-up

If you have finished learning, you can delete the resource group from your Azure subscription:

1. In the Azure portal, select Resource groups on the right menu and then select the resource group that you have created.
2. Click Delete resource group.