Evaluating Model Accuracy with Real-World Data and Online Inference Results

Question

I have modified the code referenced from the document Azure Machine Learning: Collect production data from models deployed for real-time inferencing .
I include a column containing real-world results in the input data.

The column containing real-world results is exported with the prediction results in the output analysis data.

Is this monitoring method accurate for checking if the model's accuracy has deteriorated?

Answer 1

Hi @anh quan,

Thank you for reaching out to Microsoft Q&A forum!

Your method of including real-world results alongside model predictions for monitoring accuracy is a valid approach. By comparing predicted outcomes with actual ground truth data, you can effectively track model performance over time. However, for accurate monitoring, ensure that the ground truth data is consistently labeled and reflects real-world results. Additionally, consider evaluating not just accuracy but other metrics like precision, recall, or F1 score, depending on the business context. Monitoring over a longer period can help identify true performance degradation, and periodic retraining with fresh data can prevent deterioration. Overall, this approach can help track accuracy but requires regular validation and updates to maintain reliable performance assessments.

I hope you understand! Thank you.