Automate customer reviews processing with Form Recognizer and Azure OpenAI

Organizations of all types rely on automated document processing and data extraction to streamline their operations. This article outlines a potential solution for automating PDF form processing, utilizing Azure Form Recognizer for data extraction and Azure OpenAI for “intelligent” data enrichment.

Data enrichment uses Artificial Intelligence (AI) to extract information, uncover patterns, and gain a deeper understanding of the data. This can be achieved through techniques such as key phrase extraction, named entity recognition, sentiment analysis, opinion mining, and custom models to identify essential information. To enrich the data, you can use the pretrained models of the Azure Cognitive Services for Language or train and deploy a custom model in Azure Machine Learning. In this post, I’ll demonstrate how to utilize the Davinci model of the Azure OpenAI Service to perform sentiment analysis and extract key phrases from customer service review forms.

In the two-part series “Automate document processing with Form Recognizer and Logic Apps” you learned how to train custom models in Azure Form Recognizer for extracting key-value pairs from documents and build an end-to-end form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI.

• Part 1: Create a custom template model in Azure Form Recognizer
• Part 2: End-to-end document processing automation solution

In this article, we will extend this pipeline by incorporating the Azure OpenAI service to enrich the extracted data. You will learn how to:

• Create an Azure OpenAI resource and deploy a model.
• Build a pipeline in Logic Apps to post a request to the Azure OpenAI endpoint, receive and evaluate the response.

Architecture

The following architecture diagram illustrates the main components involved in the “intelligent” form processing solution that we are building and the information flow. In this post, we will focus on how to integrate the Azure OpenAI service into the pipeline that was built in the previous articles.

Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

1. PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
2. Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
3. The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
4. The Logic App posts a request to the Azure OpenAI service to analyze the sentiment and extract key phrases from the customer’s review, and then receives the response data.
5. The extracted data are saved into a NoSQL database in Azure Cosmos DB.
6. Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.

Create an Azure OpenAI resource

You will create an Azure OpenAI resource through the Azure portal.

1. Sign in to the Azure portal and search for OpenAI.

2. In the Create Azure OpenAI page provide the following information:

   • Subscription: Your Azure subscription.
   • Resource group: Select an existing resource group or create a new one.
   • Region: Choose any available region, for example, West Europe.
   • Name: Enter a unique name.
   • Pricing tier: Standard (S0).
   

3. Select Review + Create and then Create.

4. Once the deployment is complete, navigate to the Azure OpenAI Studio.

Deploy a model

Before you can use the Azure OpenAI service to generate text, you need to deploy a model. There are several available models in the Azure OpenAI Studio, each of which is tailored to a specific use case. In this post, you will deploy the text-davinci-003 model.

1. In the Azure OpenAI Studio, select Deployments under Management.

2. Click on + Create new deployment, and deploy a new model with the following settings:

   • Model name: text-davinci-003
   • Deployment name: Choose a memorable name for your deployment.

3. Once the model is deployed, you can test it in the Completions playground.

   

4. In the Completions playground, click View code. In the Sample Code window, save the endpoint of your deployed model. You will need this URL in the next step.

   

Build the workflow

In a previous article, you’ve built the workflow below. This pipeline receives a form that is uploaded in an Azure Storage container, extracts the fields from the form and saves the extracted data in Azure Cosmos DB.

You are going to extend this pipeline by adding an HTTP request action to send a request to the Azure OpenAI service. The workflow is illustrated in the following image.

You will build the form processing workflow using the Logic App Designer.

1. After the When a resource event occurs trigger, add two Initialize variable actions. Create two variables named openai-api-key and openai-url to store the Key and the URL of your Azure OpenAI resource, respectively.

   
   The key of your Azure OpenAI service can be found by going to the Keys and Endpoint page of your resource in the Azure portal.

2. Under the Parse Fields block, select the plus (+) sign to add a new action. Select the HTTP action and enter the following information:

   • Method: POST

   • URI: openai-url

   • Headers:

     Content-Type	application/json
     api-key	openai-api-key

   • Body:

     1 2 3 4 5 6 7 8 9 10

     { "prompt": "You must extract the following information from the review below:\n1. Sentiment (key: Sentiment) (possible values: positive, negative, neutral, mixed)\n2. Opinion for the customer service in an array (key: KeyPhrases)\nAnswer the fields briefly and provide the results in JSON format using the keys above. For the second field, summarize the opinions if needed. If the review is empty, use \"NA\" for the first field and an empty array for the second field.\nReview: \"IF EXPRESSION\"", "temperature": 1, "top_p": 0.5, "frequency_penalty": 0, "presence_penalty": 0, "max_tokens": 100, "best_of": 1, "stop": null }

   Replace IF EXPRESSION with the following expression, which extracts the value associated with the field Comments.

   1	if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], '')

   

3. The output from the completions API will look as follows:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

   { "id": "ID of your call", "object": "text_completion", "created": 1680202256, "model": "text-davinci-003", "choices": [ { "text": "text generated by the OpenAI API", "index": 0, "finish_reason": "stop", "logprobs": null } ], "usage": { "completion_tokens": 41, "prompt_tokens": 121, "total_tokens": 162 } }

4. Add a Parse JSON action to extract the response of your OpenAI model. Specify the following details:

   • Content: Select Add dynamic content and find the block called Body.
   • Schema: To generate the schema, use a sample JSON response of the OpenAI API.
   

5. Then, use a second Parse JSON action to extract the values generated by the OpenAI API for the keys named Sentiment and KeyPhrases. Enter the following information:

   • Content: Select Add dynamic content and find the block called text under Parse OpenAI response.

   • Schema: To generate the schema, use a sample JSON generated by the OpenAI API or the following schema:

     1 2 3 4 5 6 7 8 9 10 11 12 13 14

     { "properties": { "KeyPhrases": { "items": { "type": "string" }, "type": "array" }, "Sentiment": { "type": "string" } }, "type": "object" }

   
   The Logic App automatically adds a For each block around the Parse JSON block.

6. Modify the existing Compose action to include the information generated by Azure OpenAI. To extract the values associated with the fields Sentiment and KeyPhrases, select Add dynamic content and choose the respective blocks.

   

7. Modify the Create or update document (V3) action as needed.

   

Save the workflow and then click Run Trigger > Run. Upload a file to your Azure Storage container to test the Logic App.

Visualize the data in Power BI

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

1. Open Power BI Desktop and in the Home tab select Get data > More…
2. Choose the Azure Cosmos DB v1 connection and click Connect.
3. In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click OK.
4. Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the data generated by the Azure OpenAI service.

Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, and Azure OpenAI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by integrating Azure Cognitive Services for Language into your Logic App workflow.

You can learn more about Azure OpenAI in the resources below:

• What is Azure OpenAI Service? – Microsoft Docs
• Revolutionize your Enterprise Data with ChatGPT: Next-gen Apps w/ Azure OpenAI and Cognitive Search