Move data from one component to the next in Azure Machine Learning

Question

I have got 2 components in Azure Machine Learning. I have got 2 dataframes in the first component (called prep) which I want to pass into the next component (called middle) for further processing.

In the prep code, I have tried to save the dataframe into the component's output section, into a datastore and into the args location passed in as input parameters. As shown below:

print((Path(args.Y_df) / "Y_df.csv"))
df1.to_csv("./outputs/Y_df.csv")
 df1.to_csv(args.Y_df.path)
 df1.to_csv("azureml://subscriptions/subscription_id/resourcegroups/rg_group/workspaces/workspace_name/datastores/datastore_name/paths/azureml/forecast/testing/y_df.csv")

Out of these only the first method works. Now I want to pass this into the next component. So in the pipeline definition code, I have mentioned this:

def data_pipeline(
    compute_train_node: str,
):

    prep_node = prep()
    transform_node = middle(Y_df=prep_node.outputs.Y_df,
                            S_df=prep_node.outputs.S_df)

I am trying to run a basic code in the middle component but it just does not get started. It fails with the following error:

What am I doing wrong? How do I pass file from one component to the other?

Answer 1

This question got answered here