Are test images also used for training or are they held aside during training?

Schmidt-Lebuhn, Alexander (NCMI, Black Mountain) 0

After training, custom vision tests the model on 20% of the images of each class. Were these images held aside and not used for training, or were they also used as part of the training set? Thanks in advance.

(Note that the AI-provided answer is that the 20% test images are being held aside and not used during training. But I am sad to say that cannot see where it 'thinks' it gets that information from in the two links it provided as the basis of its conclusion. As far as I can tell, the two linked documentation pages do not contain any information on that question.)

Saideep Anchuri 1,285 Reputation points Microsoft Vendor

2025-01-17T00:16:18.6766667+00:00

Hi Schmidt-Lebuhn, Alexander (NCMI, Black Mountain)

Following up to see if the given response was helpful.

Thank You.
Saideep Anchuri 1,285 Reputation points Microsoft Vendor

2025-01-20T01:04:07.6566667+00:00

Hi Schmidt-Lebuhn, Alexander (NCMI, Black Mountain)

We haven’t heard from you on the last response and was just checking back to see if the give response was helpful.

Thank You.

1 answer

Saideep Anchuri 1,285 Reputation points Microsoft Vendor

2025-01-16T04:36:08.54+00:00

Hi Schmidt-Lebuhn, Alexander (NCMI, Black Mountain)

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

It is a standard procedure in machine learning to split the dataset in Train, Validation and Test. The evaluation metrics including F1 score, precision, accuracy etc. are derived from comparison of predicted data against validation data and is displayed on summary page once model has been trained. We can provide external source of images for testing new model, ideally which were not used in model training to check its accuracy.

Kindly refer below link: evaluate-the-trained-model

Thank You.
Please sign in to rate this answer.
Schmidt-Lebuhn, Alexander (NCMI, Black Mountain) 0 Reputation points

2025-01-20T07:42:01.2766667+00:00

Hi Saideep,

Thanks for the quick answer. I did not check over the weekend, so I am seeing it only now.

I apologise if I am slow to understand, but "It is a standard procedure in machine learning to split the dataset in Train, Validation and Test" means that Custom Vision also follows that standard procedure, i.e., it does not Train on the 20% of images that it Tests on?

This is distinct from whether or not the user subsequently tests external images with "Try it out".

Thanks again,

Alexander

Saideep Anchuri 1,285 Reputation points Microsoft Vendor

2025-01-20T08:36:52.09+00:00

Hi Schmidt-Lebuhn, Alexander (NCMI, Black Mountain)

Thank you for following up. It is mentioned in the below document that "If an evaluation set isn't provided when training the model, the reported performance is estimated based on part of the training set. We strongly recommend you use an evaluation dataset (using the same process as above) to have a reliable estimation of your model performance.", Yes Try out feature is different from Test data practice we follow in machine learning. Here it is train and validation only if validation (evaluation dataset) is not provided from customer, In Machine learning, we ideally split entire original dataset to train-validation-test dataset (70-20-10) from the original dataset.

Do click Accept Answer and Yes for was this answer helpful.

kindly refer below documentation: evaluate-the-trained-model

Thank You.

Schmidt-Lebuhn, Alexander (NCMI, Black Mountain) 0 Reputation points

2025-01-20T23:28:45.29+00:00

Okay, so this means that the testing is done on images that are also used for training? I was not aware that there was a way for me to mark some of the training libraries as the evaluation set. I will have to look into how to do that. I would also say, however, that it would be more logical to me if Custom Vision did randomly select 20% of the libraries to not train on them but use them as the evaluation set, because there is a risk of bias if I as the user have to select the evaluation set manually.

Saideep Anchuri 1,285 Reputation points Microsoft Vendor

2025-01-21T05:29:56.06+00:00

Hi Schmidt-Lebuhn, Alexander (NCMI, Black Mountain)

Ok, thank you for providing clarity on inclusion of evaluation dataset from training dataset only in absence of evaluation. Work around is split the original dataset itself randomly to train/evaluation/test using python splitter library and save them in different folders and use the later for evaluation and test. train-test-validation I would suggest you post this an idea of keeping a portion of input data separate in Portal side for validation or test in Azure feedback if you want it be considered as feature request. https://feedback.azure.com/d365community

Thank You.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Are test images also used for training or are they held aside during training?

1 answer

Your answer