Machine Learning Studio - prepare_image step fails when deploying real time inference endpoint

Question

I am using Azure Machine Learning Studio.

In the UI, I define a pipeline in a designer. I train model by pressing Configure and Submit, I only change compute instance to mine and I set a name. Then in the pipeline's menu, I choose Create Inference pipeline, then I choose Real-time inference pipeline.

New pipeline design window shows up. I press Configure and Submit, again, I set compute instance to the one I ran and then I set name.

Real-Time inference Pipeline finishes. I press deploy. New endpoint is being created along with prepare_image step with some random name. On Deployment configuration window, I choose Azure container instance as compute type.

Endpoint is never online as prepare_image job fails:

Execution failed. User process 'python' exited with status code 1. Please check log file 'user_logs/std_log.txt' for error details. Error: Traceback (most recent call last):
  File "script.py", line 110, in <module>
    docker_utilities._docker_build_or_error(
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 156, in _docker_build_or_error
    _write_error_and_exit(error_msg, error_file_path=error_file_path)
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 217, in _write_error_and_exit
    sys.exit(1)
SystemExit: 1

2024-11-09T15:22:59: ERROR: No matching distribution found for azureml-designer-serving==0.0.15
2024-11-09T15:22:59: ERROR: Could not find a version that satisfies the requirement azureml-designer-serving==0.0.15 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.12)
2024-11-09T15:22:59: ERROR: No matching distribution found for azureml-designer-serving==0.0.15
2024-11-09T15:22:59: ERROR: No matching distribution found for azureml-designer-serving==0.0.15

The same thing happens when I follow sample from Microsoft: https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-designer-automobile-price-deploy?view=azureml-api-1

I click Designer button, then I choose example "Regression - Automobile price prediction". Then I do the same steps as described. When I deploy, I receive the same error.

Full log from prepare_image job:

2024-11-09T15:21:43: Logging into Docker registry: contregstd.azurecr.io
2024-11-09T15:21:43: WARNING! Using --password via the CLI is insecure. Use --password-stdin.
2024-11-09T15:21:43: WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
2024-11-09T15:21:43: Configure a credential helper to remove this warning. See
2024-11-09T15:21:43: https://docs.docker.com/engine/reference/commandline/login/#credentials-store

2024-11-09T15:21:43: Login Succeeded


2024-11-09T15:21:43: Running: ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478:1']
2024-11-09T15:21:43: DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
2024-11-09T15:21:43:             Install the buildx component to build images with BuildKit:
2024-11-09T15:21:43:             https://docs.docker.com/go/buildx/

2024-11-09T15:21:43: Sending build context to Docker daemon  71.17kB
2024-11-09T15:21:43: Step 1/17 : FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04@sha256:71ca645852d85f8de2e33ca6dc244d114e65dad996788bed27b55339aa54adc8
2024-11-09T15:21:43: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04@sha256:71ca645852d85f8de2e33ca6dc244d114e65dad996788bed27b55339aa54adc8: Pulling from azureml/openmpi4.1.0-ubuntu20.04
2024-11-09T15:21:43: 86e5016c2693: Pulling fs layer
2024-11-09T15:21:43: d1a1d5b6646e: Pulling fs layer
2024-11-09T15:21:43: 5973bc5bc768: Pulling fs layer
2024-11-09T15:21:43: 747c78a11127: Pulling fs layer
2024-11-09T15:21:43: f2ed6cc7f70e: Pulling fs layer
2024-11-09T15:21:43: a0c2be36cd23: Pulling fs layer
2024-11-09T15:21:43: e6fe1b75ea98: Pulling fs layer
2024-11-09T15:21:43: 26159770588d: Pulling fs layer
2024-11-09T15:21:43: 8118942af5bd: Pulling fs layer
2024-11-09T15:21:43: 664ebb45d947: Pulling fs layer
2024-11-09T15:21:43: e6fe1b75ea98: Waiting
2024-11-09T15:21:43: 747c78a11127: Waiting
2024-11-09T15:21:43: f2ed6cc7f70e: Waiting
2024-11-09T15:21:43: a0c2be36cd23: Waiting
2024-11-09T15:21:43: 26159770588d: Waiting
2024-11-09T15:21:43: 8118942af5bd: Waiting
2024-11-09T15:21:43: 664ebb45d947: Waiting
2024-11-09T15:21:43: 5973bc5bc768: Verifying Checksum
2024-11-09T15:21:43: 5973bc5bc768: Download complete
2024-11-09T15:21:43: 747c78a11127: Verifying Checksum
2024-11-09T15:21:43: 747c78a11127: Download complete
2024-11-09T15:21:43: f2ed6cc7f70e: Verifying Checksum
2024-11-09T15:21:43: f2ed6cc7f70e: Download complete
2024-11-09T15:21:43: 86e5016c2693: Verifying Checksum
2024-11-09T15:21:43: 86e5016c2693: Download complete
2024-11-09T15:21:43: e6fe1b75ea98: Verifying Checksum
2024-11-09T15:21:43: e6fe1b75ea98: Download complete
2024-11-09T15:21:44: 26159770588d: Verifying Checksum
2024-11-09T15:21:44: 26159770588d: Download complete
2024-11-09T15:21:44: 8118942af5bd: Verifying Checksum
2024-11-09T15:21:44: 8118942af5bd: Download complete
2024-11-09T15:21:44: 664ebb45d947: Verifying Checksum
2024-11-09T15:21:44: 664ebb45d947: Download complete
2024-11-09T15:21:44: a0c2be36cd23: Verifying Checksum
2024-11-09T15:21:44: a0c2be36cd23: Download complete
2024-11-09T15:21:44: d1a1d5b6646e: Verifying Checksum
2024-11-09T15:21:44: d1a1d5b6646e: Download complete
2024-11-09T15:21:45: 86e5016c2693: Pull complete
2024-11-09T15:21:58: d1a1d5b6646e: Pull complete
2024-11-09T15:21:58: 5973bc5bc768: Pull complete
2024-11-09T15:21:59: 747c78a11127: Pull complete
2024-11-09T15:21:59: f2ed6cc7f70e: Pull complete
2024-11-09T15:22:05: a0c2be36cd23: Pull complete
2024-11-09T15:22:05: e6fe1b75ea98: Pull complete
2024-11-09T15:22:06: 26159770588d: Pull complete
2024-11-09T15:22:07: 8118942af5bd: Pull complete
2024-11-09T15:22:07: 664ebb45d947: Pull complete
2024-11-09T15:22:07: Digest: sha256:71ca645852d85f8de2e33ca6dc244d114e65dad996788bed27b55339aa54adc8
2024-11-09T15:22:07: Status: Downloaded newer image for mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04@sha256:71ca645852d85f8de2e33ca6dc244d114e65dad996788bed27b55339aa54adc8
2024-11-09T15:22:07:  ---> 17e753bc1c6d
2024-11-09T15:22:07: Step 2/17 : USER root
2024-11-09T15:22:07:  ---> Running in b6c17faaadff
2024-11-09T15:22:08: Removing intermediate container b6c17faaadff
2024-11-09T15:22:08:  ---> 902d76bea2c4
2024-11-09T15:22:08: Step 3/17 : RUN mkdir -p $HOME/.cache
2024-11-09T15:22:08:  ---> Running in 8042b22ded27
2024-11-09T15:22:10: Removing intermediate container 8042b22ded27
2024-11-09T15:22:10:  ---> c3c7b63bcada
2024-11-09T15:22:10: Step 4/17 : WORKDIR /
2024-11-09T15:22:10:  ---> Running in 5556df32a641
2024-11-09T15:22:11: Removing intermediate container 5556df32a641
2024-11-09T15:22:11:  ---> 1f6aebf1e086
2024-11-09T15:22:11: Step 5/17 : COPY azureml-environment-setup/99brokenproxy /etc/apt/apt.conf.d/
2024-11-09T15:22:12:  ---> 60caf59db876
2024-11-09T15:22:12: Step 6/17 : RUN if dpkg --compare-versions `conda --version | grep -oE '[^ ]+$'` lt 4.4.11; then conda install conda==4.4.11; fi
2024-11-09T15:22:12:  ---> Running in 36cf90187852
2024-11-09T15:22:16: Removing intermediate container 36cf90187852
2024-11-09T15:22:16:  ---> 0059cc283ff3
2024-11-09T15:22:16: Step 7/17 : COPY azureml-environment-setup/mutated_conda_dependencies.yml azureml-environment-setup/mutated_conda_dependencies.yml
2024-11-09T15:22:17:  ---> 3e38bed90e4e
2024-11-09T15:22:17: Step 8/17 : RUN ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_d94996a922f33988d50b0d1ef907c9d2 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig
2024-11-09T15:22:17:  ---> Running in 9c6235c28f0b
2024-11-09T15:22:18: /opt/miniconda/lib/python3.10/argparse.py:1943: FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. Use `conda env create --file=URL` instead.
2024-11-09T15:22:18:   action(self, namespace, argument_values, option_string)
2024-11-09T15:22:18: /opt/miniconda/lib/python3.10/site-packages/conda/base/context.py:198: FutureWarning: Adding 'defaults' to channel list implicitly is deprecated and will be removed in 25.3. 

2024-11-09T15:22:18: To remove this warning, please choose a default channel explicitly with conda's regular configuration system, e.g. by adding 'defaults' to the list of channels:

2024-11-09T15:22:18:   conda config --add channels defaults

2024-11-09T15:22:18: For more information see https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/use-condarc.html

2024-11-09T15:22:18:   deprecated.topic(
2024-11-09T15:22:18: Retrieving notices: ...working... done
2024-11-09T15:22:18: /opt/miniconda/lib/python3.10/site-packages/conda/base/context.py:198: FutureWarning: Adding 'defaults' to channel list implicitly is deprecated and will be removed in 25.3. 

2024-11-09T15:22:18: To remove this warning, please choose a default channel explicitly with conda's regular configuration system, e.g. by adding 'defaults' to the list of channels:

2024-11-09T15:22:18:   conda config --add channels defaults

2024-11-09T15:22:18: For more information see https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/use-condarc.html

2024-11-09T15:22:18:   deprecated.topic(
2024-11-09T15:22:18: Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you.
2024-11-09T15:22:18: Channels:
2024-11-09T15:22:18:  - anaconda
2024-11-09T15:22:18:  - conda-forge
2024-11-09T15:22:18:  - defaults
2024-11-09T15:22:18: Platform: linux-64
2024-11-09T15:22:43: Collecting package metadata (repodata.json): ...working... done
2024-11-09T15:22:44: Solving environment: ...working... done

2024-11-09T15:22:52: Downloading and Extracting Packages: ...working... done
2024-11-09T15:22:52: Preparing transaction: ...working... done
2024-11-09T15:22:54: Verifying transaction: ...working... done
2024-11-09T15:22:56: Executing transaction: ...working... done
2024-11-09T15:22:59: Installing pip dependencies: ...working... Pip subprocess error:
2024-11-09T15:22:59: ERROR: Could not find a version that satisfies the requirement azureml-designer-serving==0.0.15 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.12)
2024-11-09T15:22:59: ERROR: No matching distribution found for azureml-designer-serving==0.0.15

2024-11-09T15:22:59: Ran pip subprocess with arguments:
2024-11-09T15:22:59: ['/azureml-envs/azureml_d94996a922f33988d50b0d1ef907c9d2/bin/python', '-m', 'pip', 'install', '-U', '-r', '/azureml-environment-setup/condaenv.o2n_3m90.requirements.txt', '--exists-action=b']
2024-11-09T15:22:59: Pip subprocess output:
2024-11-09T15:22:59: Collecting azureml-defaults
2024-11-09T15:22:59:   Downloading azureml_defaults-1.58.0-py3-none-any.whl (2.0 kB)
2024-11-09T15:22:59: Collecting azureml-model-management-sdk
2024-11-09T15:22:59:   Downloading azureml_model_management_sdk-1.0.1b6.post1-py2.py3-none-any.whl (130 kB)

2024-11-09T15:22:59: failed
2024-11-09T15:22:59: 
2024-11-09T15:22:59: CondaEnvException: Pip failed

2024-11-09T15:23:02: The command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_d94996a922f33988d50b0d1ef907c9d2 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig' returned a non-zero code: 1
2024-11-09T15:23:02: 


2024-11-09T15:23:02: CalledProcessError(1, ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478:1'])

2024-11-09T15:23:02: Building docker image failed with exit code: 1

2024-11-09T15:23:02: Logging out of Docker registry: contregstd.azurecr.io
2024-11-09T15:23:02: Removing login credentials for https://index.docker.io/v1/


2024-11-09T15:23:02: Traceback (most recent call last):
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 152, in _docker_build_or_error
    docker_execute_function(docker_command, build_command, print_command_args=True)
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 23, in docker_execute_function
    return killable_subprocess.check_call(command_args, *popen_args,
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/killable_subprocess.py", line 261, in check_call
    raise subprocess.CalledProcessError(process.returncode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478', '-t', 'contregstd.azurecr.io/azureml/azureml_9117e0c9672679c2ada59d81d9f40478:1']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 110, in <module>
    docker_utilities._docker_build_or_error(
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 156, in _docker_build_or_error
    _write_error_and_exit(error_msg, error_file_path=error_file_path)
  File "/mnt/azureml/cr/j/f99b2aa177bc47d4a49962ac17f8ccc6/exe/wd/docker_utilities.py", line 217, in _write_error_and_exit
    sys.exit(1)
SystemExit: 1

Environment prepare_image uses by prepare_image: docker-tools/version/1

Answer 1

I think that this error appears to be coming from a missing version (0.0.15) of azureml-designer-serving. Azure Machine Learning Studio tries to install this version, which isn’t available in the current Azure package registry.

In the pipeline environment setup, manually specify the latest available version of azureml-designer-serving (e.g., 0.0.12). You can edit the environment file (mutated_conda_dependencies.yml) or manually add the following in the pipeline environment configuration:

- pip:

   - azureml-designer-serving==0.0.12

Instead of relying on the default setup, create a custom Azure Machine Learning environment that excludes the incompatible version. You can create an environment in Azure ML Studio under "Environments" and specify compatible versions.

Sometimes, cached Docker images can lead to issues. If you have access, try clearing Docker caches or use the --no-cache option for Docker builds. You may need assistance from an Azure administrator if this option isn't directly accessible.

Answer 2

For those who use only Azure ML Studio, I found a way to bypass this issue:

• For those who use only Azure ML Studio, I found a way to bypass this issue:
• For those who use only Azure ML Studio, I found a way to bypass this issue:
• Access the trained model data at "Data" section (the name always starts with "MD-");
• Go to "Data sources", copy the relative path and open the datastore link;
• Open the "Blob container" link; this will take you to the blob storage;
• Paste the relative path copied before at the Search bar and open the "Trained_model" folder;
• Look for the "conda_env.yaml" file, click on the "..." and select the "View/edit";
• Look for the "azureml-designer-serving" line, and change the version to the last valid version (for now, it's 0.0.12):
  (azureml-designer-serving==0.0.12)
• Save the file;
• Now you can rerun the real-time inference pipeline job and deploy the endpoint update;

After that, the prepare_image job will run and complete as expected.