Text Analysis in PDF Documents with Azure AI Services

SIMON Witali 1

Hello everyone,

I am working on a project where I regularly handle extensive collections of PDF documents filled with specific technical terms and abbreviations. These documents also vary greatly in layout and formulation, presenting a unique challenge for text analysis.

After attempting a Retrieval Augmented Generation (RAG) solution and facing challenges with Azure Search, I’ve started exploring Azure Document Intelligence, which seems promising. However, I am exploring broader options and seeking more generalized advice.

Tutorial Link: https://learn.microsoft.com/en-us/training/modules/use-own-data-azure-openai/

I am looking for ideas on which Azure functions could assist me further and what a potential target architecture might look like for such a system. Specifically, I aim to:

Cluster documents by technology using a predefined list of terms and abbreviations.
Search documents for specific terms effectively.
Develop a Q&A system to answer general questions about the document contents.

If anyone has experience with similar challenges or could suggest specific Azure functions that could be useful for these needs, I would greatly appreciate your insights or any references to helpful resources.

Thank you in advance for your support!

Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-01-29T10:23:33.08+00:00
Hi @SIMON Witali,
Welcome to the Microsoft Q&A Platform!
It seems like you're facing challenges in handling and analyzing large collections of technical PDF documents with complex layouts and specific terminologies.

Azure Cognitive Search can assist you in incorporating strong full-text search functionality.
You can:
Apply the AI-enhanced cognitive capabilities integrated into the service, including text extraction, language identification, and entity recognition.
Index specific fields like terms and abbreviations that are relevant to your technical documents. Azure Search is also integrated with Azure OpenAI Service in order to enhance your search with natural language queries.

For different document types, Azure Form Recognizer is a great tool. It can extract information like tables and key-value pairs from documents, even complex forms. This helps make the extracted data consistent before further analysis.

After getting the text, Azure Text Analytics can help. It can identify key words and acronyms, and even group similar documents together based on technical terms.

For more advanced clustering, Azure Machine Learning is an option. You can train a model to recognize patterns in the text and group similar documents. You can use techniques like K-means or DBSCAN and even use pre-trained models from Azure OpenAI for a deeper understanding of the text.

For your Q&A system, Azure OpenAI's language models are ideal. These models can understand and answer questions about the document content. Combining these models with Azure Cognitive Search will give you very good results, especially when dealing with natural language questions about the documents.

Reference:
Azure Cognitive Search Documentation

Azure Form Recognizer Overview

Azure Machine Learning

Azure OpenAI Service
Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-01-30T08:28:26.72+00:00

Hi @SIMON Witali,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-01-31T06:46:00.7233333+00:00

Hi @SIMON Witali,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
SIMON Witali 1 Reputation point

2025-01-31T07:52:36.4066667+00:00

Thanks for your answer, I'll continue to experiment and look for a solution, but so far I'm not happy with the results.

1 answer

SIMON Witali 1 Reputation point

2025-01-31T07:51:08.4133333+00:00

Hello Bhargavi Naragani,

Thank you for your comprehensive answer to my previous query! It has been very helpful in guiding my project's direction. As I delve deeper into the capabilities of Azure OpenAI and Cognitive Search, I have a couple of follow-up questions:

Limitations on the Number of Documents in Azure OpenAI Search: I am encountering limitations on how many documents Azure OpenAI Search can process at once. Is there a specific limit to the number of documents that can be indexed and searched? If so, what strategies would you recommend for effectively managing large datasets?

Optimizing Indexing and Search Operations: What are the best practices for indexing a large number of documents in Azure Cognitive Search to optimize integration with Azure OpenAI? Are there particular techniques or settings that could help enhance performance, especially when dealing with a high volume of documents?

Thank you once again for your support!
Please sign in to rate this answer.
Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-02-03T08:54:20.9366667+00:00

Hi @SIMON Witali,
Thank you for reaching out to us with follow up questions!

In Microsoft Azure OpenAI Search, the limits of documents indexable vary as per your tier of service and are highest up to 24 billion on Basic, S1-S3, and L1-L2 levels, while a lower limit at 2 billion documents is associated with the S3 HD. Each separate item within a complex collection count in this limit; individual documents also cannot exceed approx. 16 megabytes.
Strategies for managing large datasets:

It's possible to speed up things by aggregating multiple documents into one batch and then sending the batch in a single request to Azure AI Search. It is possible to include up to 1,000 documents in each such batch, provided that the total size of all documents does not exceed 16 megabytes. This kind of batching can make quite a difference for the indexing of your documents.

To make the most of push APIs, manage how your application handles multiple tasks at the same time carefully, increase the number of tasks it can handle concurrently to make the best use of available resources, and implement a system that gracefully retries failed requests with increasing delays to overcome temporary issues like server unavailability.

Only include the fields that are really necessary for your searches when designing your search index. Avoid too many unnecessary field properties, which will slow down the search process and use up more storage space than necessary. For instance, mark fields as 'searchable' only if they really need to be searched for specific words or phrases.

Choose a service plan that suits the size and complexity of your data. More premium plans have greater storage and more performance. If you're dealing with a large dataset, you can also just increase the number of partitions; this spreads out the workload, so things go on much more smoothly.

Overview:
https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity
https://learn.microsoft.com/en-us/azure/search/search-how-to-large-index

Hope the above provided information help in better understanding and help you resolve the issue, if you have any further concerns or queries, please feel free to reach out to us.

Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-02-04T09:25:03.6566667+00:00

Hi @SIMON Witali,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Bhargavi Naragani 410 Reputation points Microsoft Vendor

2025-02-05T09:45:10.0566667+00:00

Hi @SIMON Witali,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Text Analysis in PDF Documents with Azure AI Services

1 answer

Your answer