ADF and Azure ML Integration

Question

Hello,

In Azure Data Factory (ADF), we have Development (DEV), Staging (Stage), and Production (PROD) workspaces. Similarly, Azure Machine Learning (Azure ML) also has corresponding DEV, Stage, and PROD workspaces.

In ADF DEV, I have an Azure Machine Learning (Azure ML) activity that connects using an Azure ML linked service. Currently, the ML linked service contains hardcoded values for subscriptionId, resourceGroupName, and WorkspaceName. With this setup, I am able to connect to the Machine Learning pipeline and successfully execute it through ADF.

However, when migrating the Machine Learning linked service to a higher environment (Stage/PROD), I am encountering issues due to the hardcoded values.

• In Stage, it attempts to fetch Stage ML details.
• In PROD, it fetches PROD ML details.

To resolve this, I am passing the required values in a parameterized JSON format. The connections test successfully, but unfortunately, I am unable to fetch the Machine Learning pipeline name details.

Interestingly, if I configure the linked service with hardcoded values, I am able to fetch the ML pipeline list and execute the pipeline through ADF without issues.

below json request
{

"properties": {

    "type": "AzureMLService",

    "parameters": {

        "subscriptionId": {

            "type": "String"

        },

        "resourceGroupName": {

            "type": "String"

        },

        "mlWorkspaceName": {

            "type": "String"

        }

    },

    "annotations": [],

    "typeProperties": {

        "subscriptionId": "@{linkedService().parameters.subscriptionId}",

        "resourceGroupName": "@{linkedService().parameters.resourceGroupName}",

        "WorkspaceName": "@{linkedService().parameters.WorkspaceName}",

        "authentication": "MSI"

    }

}

}

Could you please provide insights or recommendations on resolving this issue?

Thank you.

Best regards,
praveen

Answer 1

Hello Sama Praveen Kumar Reddy,

Thank you for reaching out on the Microsoft Q&A.

When you parameterize the Azure ML linked service in ADF, fetching the ML pipeline list doesn't work like it does with hardcoded values. This happens because ADF needs static values to fetch the pipelines at design time.

To fix this:

1. For development, use hardcoded values to fetch the pipeline list.
2. Once you're ready to deploy to Stage or PROD, parameterize the linked service with the correct values for subscriptionId, resourceGroupName, and mlWorkspaceName.

This way, the connection will work across environments, but you won't be able to fetch pipelines at design time with parameterized values. For more details, check out ADF Parameterization and Execute ML Pipelines.

I hope this helps! If you have any further questions, feel free to ask.

If the information is useful, please accept the answer and upvote it to assist other community members.