Chunk index AI Studio BYOD

Question

It seems that creation of Indexes in AI Studio changed in August, when you create a Index now its created with 2 Skillsets Microsoft.Skills.Text.SplitSkill and Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill.

This follows the documentation https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization#using-integrated-vectorization-during-indexing, but no index or indexer is created for the chunks

"An indexer, which retrieves raw data from a supported data source and serves as the pipeline engine.
A vector index to receive the chunked and vectorized content."

If I look at older index created prior to any updates in AI Studio, we see that its made with Microsoft.Skills.Custom.WebApiSkill, and in this custom skillset parameters for the Chunk index is presence and the Chunk index and indexer is created so 2 index and indexers is created.

I did a comment on it here: https://learn.microsoft.com/en-us/answers/questions/2032550/automatic-indexer-creation-is-not-working-in-azure

So how do I get the chunking index and indexer back, without using the "old" custom skillset.

Answer 1

Hi @Matis Kofoed Petersen,

Thank you for your patience.

Azure AI Search has undergone updates that now include integrated data chunking and vectorization directly within the indexing process using standard skillsets like Microsoft.Skills.Text.SplitSkill and Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill. This shift may have replaced the need for custom skillsets like Microsoft.Skills.Custom.WebApiSkill that previously handled chunking. I tried reproducing the issue by adding new data, and I was able to retrieve the information successfully, indicating that the system is functioning as intended.

If the new setup doesn't meet your specific needs, you can consider customizing the process by manually configuring chunking through a custom skillset. This approach should help you achieve similar functionality to what was available before the update.

I hope this helps! Thank you.