"This iteration failed to train" in Custom Vision. Any more info than this?

Ethan Fischer 20

Has anyone else gotten the "This iteration failed to train" error message in custom vision while training a model? We just got this while training a larger model. Can't find any more information than this, which makes it hard to tell what went wrong and how to address this.

CleanShot 2024-11-12 at 13.16.37@2x

CleanShot 2024-11-12 at 13.18.35@2x

1 answer

Saideep Anchuri 335 Reputation points Microsoft Vendor

2024-11-13T05:31:46.8033333+00:00
Hi Ethan Fische

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

I am able to see the repro in custom vision studio.

Kindly refer the below screenshot:

Here are some troubleshooting steps:

Make sure you have a minimum of 5 images for each label. verify that your images are high quality and accurately tagged.

Ensure your training and prediction endpoints are set up correctly, using an incorrect endpoint might lead to issues.

If you're working with the Custom Vision SDK, make sure your packages are updated.

It might be intermediate issue kindly try after some time.

if the issue still persists kindly, please do let us know.

Thank You.
Please sign in to rate this answer.
Ethan Fischer 20 Reputation points

2024-11-13T15:59:57.9766667+00:00

Hello,

You said you repro'd the issue but the screenshot you provided does not show this.

Suggestions 1-3 are not relevant: we've had successful training runs with the same setup.

Suggestions 4 is basically "just retry" which is a little frustrating. Would like more information on why the original error is occurring. Having to randomly retry is going to be a huge waste of time when these models take several hours to train

Saideep Anchuri 335 Reputation points Microsoft Vendor

2024-11-14T10:07:08.6066667+00:00

Hi Ethan Fischer

Sorry for the trouble you are facing. It seems that, despite following all the recommended setup steps and having ample data, the “This iteration failed to train” error persists. Given that you have 94,525 training images, let’s explore a few advanced possibilities that could be causing the issue.

Consider Custom Vision’s quota and project limits. Although you have uploaded a substantial dataset, Custom Vision may have undocumented constraints on image count, processing power, or project size that could affect training. Even if previous runs were successful, scaling up to this size might introduce issues.

Large datasets like yours can sometimes overload the platform’s batch processing capabilities. As a workaround, you might try dividing your dataset into smaller batches, to see if training succeeds with these smaller subsets. By isolating sections of your dataset, you can determine if certain images or configurations trigger the error and avoid needing to retry the full dataset each time.

I hope this helps. Could you please take a moment to retake the survey on the above response? Your feedback is greatly appreciated.

Thank you

Saideep Anchuri 335 Reputation points Microsoft Vendor

2024-11-15T00:42:18.46+00:00

Hi Ethan Fischer

Following up to see if the above answer was helpful.

Thank You.

Saideep Anchuri 335 Reputation points Microsoft Vendor

2024-11-18T00:26:08.56+00:00

Hi Ethan Fischer

We haven’t heard from you on the last response and was just checking back to see if the give response was helpful.

Thank You.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

"This iteration failed to train" in Custom Vision. Any more info than this?

1 answer

Your answer