How can I retrieve images in a chunked Azure search index?

Question

My issue is relatively simple.

I am using Azure Ai Search to index a set of documents. These documents are mostly PDFs but other formats as well. Most of them have embedded images. I know that in the document cracking phase these images are split from the document and stored in

/document/normalized_images/*/data

However, now I would like to store the base64 encoding of the image in the index. So I would like to get the chunk, its respective vector, and the base64 encoding of respective images (if they exist) on the index. For that I need to be able to process the normalized images and connect them with the right chunk.

From what I can see, this is not easy. First, I was expecting for the data field in

/document/normalized_images/*/data

to have the base64 encoding but no, I only see the name of the file like image3.jpg and in the chunk itself, I don't see any encoding either.

I have created a knowledge store to host the extracted images but I don't get any way of encoding them. I only get these fields:

None of these fields is an encoding of the image. ImageId is only an encoding of the name of the file.

Is there a better way to do this? I am willing to store the images in a knowledge store and retrieve them just as long as I can connect them in the index with the right chunk. The final objective is to be able to retrieve meaningful text and images when querying the index.

Answer 1

Hi @Nuno Rodrigues what are the typical sizes and formats of the images you'll be indexing? Are there any specific image formats you need to support (e.g., JPEG, PNG, TIFF)?

One way you can achieve your scenario is by creating a skillset with a custom skill that extracts the base64 encoded data for each image chunk during indexing. Here's a general breakdown:

• Define a skillset that includes your custom skill.
• Develop a custom skill that takes a chunk of the image data as input and outputs the base64 encoded data for that chunk.
• Associate the skillset with your Azure Search index during creation or update. This makes sure the custom skill runs on each chunk during indexing.

Another option is to leverage Azure AI Search's built-in vectorization capabilities. Here's a high-level process:

• Configure your indexer to use integrated vectorization.
• Consider using the built-in "Text Split" skill to split large documents by content boundaries before vectorization.
• Choose an appropriate embedding model for image vectorization.
• The indexing process automatically chunks large images, extracts image features using the embedding model, and indexes them for efficient retrieval.

Best,

-Grace

Answer 2

I finally managed to accomplish this. The key is to split the mappings between the ones that are coming from the document/pages source and the /normalized_images source. You need to create the separate mappings and then you can populate each image/chunk object in the index how you want.