Could not build the model: Total number of pages in the dataset is 1013. The maximum number of pages allowed in a dataset is 1000

Question

Error: (Depicted Below)

Could not build the model: Total number of pages in the dataset is 1013. The maximum number of pages allowed in a dataset is 1000

Question:

why this happing ?

Answer 1

Hello Prafful Agarwal,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you're having a pop-up error that displayed "Could not build the model: Total number of pages in the dataset is 1013. The maximum number of pages allowed in a dataset is 1000."

The error you're encountering is due to the dataset exceeding the maximum allowed number of pages, which is 1000 - You can read more here: https://learn.microsoft.com/en-us/azure/ai-studio/reference/reference-model-inference-api

To resolve the issue, follow these steps:

1. Try to reduce the number of pages in your dataset to 1000 or fewer. You can do this by splitting your dataset into smaller subsets and processing them separately.
2. Review your dataset to ensure that only the necessary pages are included. Remove any redundant or irrelevant pages that might be inflating the total count.
3. If reducing the dataset size is not feasible, consider reaching out to Azure support via Azure Portal.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Answer 2

Dear Prafful Agarwal,

Thank you for your feedback saying "you're using the Generative AI"

The solution provided will works because the error occurs because your dataset exceeds the maximum allowed number of pages, which is 1000. You can read more about this limitation here: https://learn.microsoft.com/en-us/azure/ai-studio/reference/reference-model-inference-api

You can:

1. Split your dataset into smaller subsets, each containing 1000 pages or fewer, and process them separately.
2. Ensure only necessary pages are included in your dataset. Remove any redundant or irrelevant pages to reduce the total count.
3. Contact Azure Support

And for a more accurate solution, consider the following additional steps:

• Before splitting the dataset, preprocess the data to remove any unnecessary content, such as blank pages or pages with minimal information.
• Use automated tools or scripts to help identify and remove redundant pages efficiently.
• Regularly review the latest Azure documentation for any updates or changes to the dataset size limitations and best practices.

Cheers