Issues Deploying Endpoints for Custom HF and Preset Azure Mistral Models in Azure ML

Question

Hi everyone,

I'm fine‑tuning a language model for financial Q&A using a custom loss that emphasizes numerical tokens (tokens between <NUM> and </NUM>). I run the entire process in Azure ML using pipelines and register my models after training – one is a custom Hugging Face model and the other is an Azure Mistral 3B Finetuned model (which is a preset asset).

The Problem:

Custom HF Model: When I deploy my registered custom HF model as an endpoint, the container crashes. The logs mention issues like a missing azureml-inference-server-http (which I’ve already added to my environment). My scoring script is configured to load the model from the AZUREML_MODEL_DIR environment variable, but I'm unsure how to properly connect the deployment to this registered model.

Azure Mistral Model (Preset): I'm also encountering issues with the Azure Mistral 3B Finetuned model, which is registered as a preset asset. Since it isn’t packaged as a fully downloadable artifact, I run into mounting/loading problems when I try to use it as an input in my pipeline or deploy it as an endpoint.

Has anyone experienced these issues with deploying endpoints for both custom and preset model assets in Azure ML? How can I reliably connect my deployment to the registered models without encountering these artifact or mounting issues?

Answer 1

Hi Sebastian Buzdugan

Greetings & Welcome to the Microsoft Q&A forum! Thank you for sharing your query.

I understand that you're encountering an issue with Issues Deploying Endpoints for Custom HF and Preset Azure Mistral Models in Azure ML

When deploying your custom Hugging Face model, the container crash could be related to several issues. First, ensure that all required libraries, including azureml-inference-server-httpare correctly added to your environment. If the logs indicate that this library is missing, double-check your environment configuration.

For loading your model from the AZUREML_MODEL_DIR, ensure that your scoring script is correctly set up to access this environment variable. The model should be registered in the same workspace where you are deploying the endpoint. You can verify this by checking the model registration and ensuring that the correct version is being referenced in your deployment.

Regarding the Azure Mistral 3B Finetuned model, since it is a preset asset and not a fully downloadable artifact, you may face mounting/loading issues. In this case, ensure that you are following the correct procedures for accessing preset models in Azure ML, as they may require specific configurations or permissions that differ from custom models.

Can you please refer these https://github.com/agademic/custom-models-on-azure/blob/master/deploy_model_sdk_v2.ipynb

Debug scoring scripts with Azure Machine Learning inference HTTP server

I hope these helps you. Thank you!