GenAI Evaluation - Monitor API is broken

Question

As i described in issue in azureml-ai sdk (Azure/azure-sdk-for-python#39075) : the monitoring in AzureML for LLMs is broken due to marshmallow updates.

Therefore, I saw that you updated the component generation_safety_quality_signal_monitor in this repo to v0.5.23.

But it's not updated with AzureML Monitor REST API because it creates a monitoring signal with old version of generation_safety_quality_signal_monitor (v0.4.16).

Please take a look to my issue i opened in your other repo. Do you update sometimes the monitor API to update the component version ?

Answer 1

Hello François V,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having GenAI Evaluation - Monitor API is broken.

Thank you for bringing this to our attention. I understand that the AzureML Monitor API is currently using an outdated version of the generation_safety_quality_signal_monitor component (v0.4.16) instead of the updated version (v0.5.23), and this is causing issues due to marshmallow updates.

To address this:

1. Check for Updates by ensuring that your AzureML SDK and related components are up to date. You can do this by running bash command: pip install --upgrade azureml-sdk
2. If the API does not automatically update the component version, you may need to manually update the component in your AzureML workspace. Refer to the AzureML REST API documentation - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-rest?view=azureml-api-2 for detailed steps on how to update components.
3. Contact Support via your Azure Portal if the issue persists.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.