Assistance Needed: "numpy.core.multiarray failed to import" Error During the Monitoring job at Model Performance Metrics Computation step in Azure ML Workspace

Vivek Kumar 0

I am encountering an issue while running a monitoring pipeline job in Azure Machine Learning. During the "Model Performance - Compute Metrics" step, I receive the following error:

"numpy.core.multiarray failed to import"

I do not have control over the packages that it uses while running this job

I am using this schema for creating the yaml file for setting up the monitoring job

$schema: https://azuremlschemas.azureedge.net/latest/monitorSchedule.schema.json

In documentation also i am not able to find if we can set up a custom environment there. The monitoring job automatically does the steps i only provide what metrices it needs to calculate and the data it requires. Has anyone else experienced a similar problem, and if so, how did you address it?

Bhaskar Turkar 0 Reputation points

2025-02-11T14:46:34.7366667+00:00

I am facing the same issue,
Manas Mohanty (Quadrant Resource LLC) 295 Reputation points Microsoft Vendor

2025-02-11T15:19:10.8833333+00:00

Hi Vivek Kumar!

Welcome to Azure AI Q and A forum. Thank you for posting your query here.

It is rising due to incompatible numpy libraries in default environment

You can update the environment used in endpoint to add or modify packages version.

Modifying the pip version to stable /latest version or unpinning few packages version works.

Reference

I see below schema in the official document of model monitoring.

$schema: http://azureml/sdk-2-0/Schedule.json

You can check the tested SDK or CLI mentioned in this doc.

Please let us know if these pointers resolve your issue.

Thank you.
Vivek Kumar 0 Reputation points

2025-02-11T16:18:07.1866667+00:00

Hi Manas,
I have set the yaml file using documentation only that you have mentioned and the schema url mentioned there does not works. Thats why I am using this url:

$schema: https://azuremlschemas.azureedge.net/latest/monitorSchedule.schema.json

I am using the yaml file for creating this job and using azure cli v2 to execute that command in my devops pipeline.
Can you help me with the steps i have to follow to update the pip version in the instance by the Azure CLI(using yaml file) method only?

Manas Mohanty (Quadrant Resource LLC) 295 Microsoft Vendor

Hi Vivek Kumar!

You can update your endpoint with yaml file with mentions of collectors in inputs and outputs, scoring script, env etc.

%%writefile deployment.YAML
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: my_endpoint
model: azureml:iris_mlflow_model@latest
environment:
  image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
  conda_file: model/conda.yaml #you can update the pip version to latest stable or modify other packages too.
code_configuration:
  code: scripts
  scoring_script: score.py
instance_type: Standard_F2s_v2
instance_count: 1
data_collector:
  collections:
    model_inputs:
      enabled: 'True'
    model_outputs:
      enabled: 'True'

$ az ml online-deployment create -f deployment.YAML

Update the environment only.

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: my_endpoint
model: azureml:iris_mlflow_model@latest
environment:
  image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
  conda_file: model/conda.yaml

az ml online-endpoint update --file updated_endpoint.yml --resource-group my-resource-group --workspace-name my-workspace

Reference on updating endpoints 1, 2, 3

Thank you.

Vivek Kumar 0

Hi Manas,
The steps that you have shared are for endpoint deployment. That i have already done
For scheduling monitoring we use this command
az ml schedule create -f ./model-performance-monitoring.yaml
as mentioned in the documentation. I am creating the yaml like given below:

# advanced-model-monitoring.yaml
$schema: https://azuremlschemas.azureedge.net/latest/monitorSchedule.schema.json

name: insurance_model_monitoring_performance
display_name: insurance model monitoring_performance
description: insurance_model_monitoring_performance

trigger:
  # perform model monitoring activity daily at 3:15am
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 # #every day
  time_zone: "India Standard Time"
  schedule: 
    hours: 10 # at 10am
    minutes: 15 # at 15 mins after 10am

create_monitor:

  compute: 
    instance_type: standard_e4s_v3
    runtime_version: "3.3"

  monitoring_target:
    ml_task: classification
    endpoint_deployment_id: azureml:insurance-service:staging
  
  monitoring_signals:
    data_drift_signal: # monitoring signal name, any user defined name works
      type: data_drift
      production_data:
        input_data:
```Now how do i give custom environment here? I am not able to see that in the documentation, we can only mention the instance according to that.

Manas Mohanty (Quadrant Resource LLC) 295 Reputation points Microsoft Vendor

2025-02-11T17:02:15.4866667+00:00

Hi Vivek Kumar!

Monitoring Yaml/SDK does not include environment param. (endpoint, model, cron job and signals mostly)

We have to update the endpoint only for environment related issues.

Thank you.
Vivek Kumar 0 Reputation points

2025-02-11T17:14:06.34+00:00

Hi Manas,
The monitoring is using its own compute i.e.
compute: instance_type: standard_e4s_v3

The endpoint is using another compute, so endpoint and monitoring have separate environments. I do not have control of what can install in monitoring environment.
I am providing the job flow for reference

Share via

Assistance Needed: "numpy.core.multiarray failed to import" Error During the Monitoring job at Model Performance Metrics Computation step in Azure ML Workspace

Your answer