Issue with Concatenating Two Azure ML Tables in Azure Data Assets for AutoML

Engineering IOT 20

I am trying to use Azure ML designer to concatenate two Azure ML tables in data assets that share the same schema using the Add Rows component. I have used an Azure compute instance with azureml-core version 1.59. However, I encountered the error mentioned in the log file attached to this message. The two ML tables are from two different data stores but share the same blob storage account. I have given the compute instance Blob Storage Reader and Reader access to this storage account.

Previously, I attempted to make it work using the Python SDK but faced issues importing it into Azure AutoML for object detection. These two data assets were the result of exporting two different Azure labeling projects for object detection. And I am trying to concatenate these two and use it in azure auto ml. Please let me know if anything is missing here.

kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-03T21:24:09.11+00:00
Hi Engineering IOT

I understand that you're encountering an issue with concatenating two Azure ML tables in Azure Data Assets for Auto ML. Here are some steps to follow.

Ensure both datasets have exactly the same schema, meaning the same number of columns and data types in each column.

Drag the Add Rows component into your pipeline under Data Transformation.

Connect the datasets to the two input ports, with the dataset you want to append connected to the second (right) port.

Submit the pipeline and verify that the number of rows in the output dataset equals the sum of the rows of both input datasets.

Ensure that the roles are correctly assigned and that there are no delays in the propagation of role assignments.

I hope these helps you. Thank you!

Engineering IOT 20

Hi Snigdha,

I am following the exact same steps as you mentioned. I am also attaching the screen shot of the data preview. Both datasets have the same schema and data types Screenshot from 2025-03-04 11-17-10

Here is the error message I get :

Execution failed. User process '/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/bin/python' exited with status code 1. Please check log file 'user_logs/std_log.txt' for error details. Error:   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/core/logger.py", line 214, in wrapper
    ret = func(*args, **kwargs)
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_host_executor.py", line 52, in execute_with_cli
    do_execute_with_env(parser, FolderRuntimeEnv())
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_host_executor.py", line 63, in do_execute_with_env
    ModuleReflector(parser.module_entry, env).exec(
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 397, in exec
    self._handle_exception(bex)
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 471, in _handle_exception
    raise exception
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 371, in exec
    reflected_input_ports = self._reflect_input_ports(input_ports)
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 418, in _reflect_input_ports
    value = self._env.handle_input_port(annotation, input_value)
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/env.py", line 123, in handle_input_port
    ErrorMapping.rethrow(e, InvalidDatasetError(
  File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/common/error.py", line 842, in rethrow
    raise err from e
azureml.studio.common.error.InvalidDatasetError: Dataset1 contains invalid data, failed to load dataset from /mnt/azureml/cr/j/faa96b01e9954ec1bcba61f88ad198e8/cap/data-capability/wd/INPUT_Dataset1.

1 answer

kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-04T19:11:43.3466667+00:00
Hi Engineering IOT

Can you please try the below steps.

Ensure the dataset is located at the correct path /mnt/azureml/cr/j/faa96b01e9954ec1bcba61f88ad198e8/cap/data-capability/wd/INPUT_Dataset1.

Confirm that the dataset is not corrupted and is in the correct format.

Use Azure ML Studio to manually validate the dataset.

If necessary, update the dataset path or recreate the dataset in Azure ML.

After resolving the issue, rerun the pipeline to ensure it executes successfully.
Please sign in to rate this answer.
kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-06T00:58:16.89+00:00

Hi Engineering IOT

Following up to see if the above response was helpful.

kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-06T16:26:43.2666667+00:00

Hi Engineering IOT

Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Issue with Concatenating Two Azure ML Tables in Azure Data Assets for AutoML

1 answer

Your answer