How to create my own Web data sources like Azure AI Studio for Azure AI Search

Question

I can rapidly create a RAG solution using a web site as a data source via Azure AI Studio from within the Chat Playground.

Here is a data source created in chat playground that indexes a web site:

When I take the next step to create the underlying index, indexer, datasource, etc within Azure AI search, I can see a few breadcrumbs, like the created index, but the datasource, indexer, etc. are hidden.

If I try to use the index from within the Azure portal for cognitive search the search fails with 401.

When I look at possible data sources within Azure AI(cognitive) Search, Web or web site is not an option at all. So it seems:

-Azure AI studio is using the plumbing of Azure AI Search for RAG. Is creating some breadcrumbs in my account that are visible, like the index, but also creating some non-accesible stuff, like I suspect a data source for web crawling.

Does anyone have any guidance on how to "recreate" the web crawling data source that I am able to use so easily from the Chat playground? OR how I can just re-use the index it creates and avoid the 401 error? Does Microsoft intend to productize/share the Web data source plug in from Azure AI Studio.

TIA.

Answer 1

David Fuller,

Based on my understanding of your scenario.

1.There is no direct web-crawling native mechanism in AI Search at this time.

2.That error message is most likely because the vectorizer in the index has managed identity enabled and the Azure portal needs the keys.
You may try navigating to> the index vector configuration and switch for keys > to leverage from the portal.

3.On making the web data source plugin from Azure AI Studio available for broader use - As I understand, currently, this functionality may not be planned directly in AI Search.
I'll post back if I have more updates.

I have relayed this feedback internally to our product team.
 

If you wish you may share your feedback on Uservoice - All of the feedback, you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure. Additionally, users with a similar request can up-vote your post and add their comments. Azure Search feedback

---

If the answer helped (pointed, you in the right direction) > please click Accept Answer