Question on fairly complicated rag code base from Azure-Samples that doesnt't use Azure document skillset api

Question

I'm interested in creating a pdf document ingestion pipeline. I followed the tutorial here

https://github.com/Azure/azure-search-vector-samples/blob/main/demo-python/code/integrated-vectorization/azure-search-integrated-vectorization-sample.ipynb

and managed to create a simple splitting --> embedding skillset, and used it to create an index.

Now I want to be able to make the pipeline more adaptive, since some pdf doesn't contain any text (probably scanned), and must be processed with vision techniques (gpt4v). Otherwise, the pdf is assumed to contain text, then it should be chunked after image description is generated (GPT) and stored away for retrieval later (in another image index, for instance). This gets complicated quickly, and I can't find an example of implementing these ideas using Azure doc skillset. Some Azure-Samples repo do implement these ideas, but not using the skillset api at all. Instead they create their own sophisticated and dedicated application.

https://github.com/Azure-Samples/rag-as-a-service-with-vision/tree/main

https://github.com/Azure-Samples/multimodal-rag-code-execution

My question is: is it realistic to expect to implement these processing ideas using just the skillset? Or is skillset just for creating quick demos with happy case scenario? How flexible can it be to accommodate for complexities in business requirements?

Thanks

Answer 1

Tung Nguyen Xuan. Following-up on this,

1.We already have a new sample of how to call GPT-4o mini (and can be used with any LLM) from a custom skill, so you may do your own queries within a skillset calling your own custom skill.

AzureOpenAICustomInferenceSkill

Based on your requirement/ask- Just to highlight, the skillset functionality has been available for several years and is currently utilized by thousands of users in production environments.
You may perform stress tests to determine the capabilities you require. The built-in indexers are designed to easily transfer data to an index. When scheduled, they run incrementally and continue processing items until completion.

It is recommended to test with your own data to determine your specific performance requirements for indexing and to assess whether the built-in indexers are suitable for your scenario.

2.AI Search has built-in skills already, you may check the ones that are available:

Skills for extra processing during indexing (Azure AI Search) - however, as shown in #1, with a custom skill (Custom Web API skill in an Azure AI Search enrichment pipeline), typically you can call any Azure service you need that is not directly/natively supported (and perform any custom transformation from it with custom code).

(I'll convert my comment to answer -If my answer helped (pointed, you in the right direction) > please click Accept Answer - it will benefit the community/users to find the answer quickly.