Unable to create HDInsight On Demand Cluster in Azure Data Factory

Tan Check Leam 10

I am getting the issue to creating the On Demand HDInsights in Azure Data Factory. The status is stuck at "PreparingCluster".

User's image

Here's my LinkedService configuration.

OS type : Linux

User's image

PRADEEPCHEEKATLA 90,461 Reputation points

2024-11-13T05:26:24.14+00:00

Tan Check Leam - Thanks for the question and using MS Q&A platform.

Creating an HDInsight cluster takes usually around 15-20 minutes.

Could you please try to connect after few minutes and see if that works. In case, if you experiencing the same issue, I would request you to create a new on-demand HDInsight cluster and see if you experiencing the same issue?
Meanwhile, checkout: Tutorial: Create on-demand Apache Hadoop clusters in HDInsight using Azure Data Factory.

Hope this helps. Do let us know if you have any questions.
Tan Check Leam 10 Reputation points

2024-11-13T06:05:32.6933333+00:00

Hi,

The status is still the same even I had leave it for more than 2 hours.

Is there any where I can check at which stage the provisioning have issues?
Tan Check Leam 10 Reputation points

2024-11-13T06:38:57.5366667+00:00

Hi,

It's stuck at the progress "PreparingCluster" after 3 hours.

Is there any where I can check which stage actually have issues?
PRADEEPCHEEKATLA 90,461 Reputation points

2024-11-13T10:37:37.8866667+00:00

@Tan Check Leam - This issue looks strange. As I suggested earlier, could you to create a new on-demand HDInsight cluster and see if you experiencing the same issue?
Tan Check Leam 10 Reputation points

2024-11-13T10:47:37.88+00:00

Hi ,

I have deleted and recreated the linked service for many times and same thing happened.

If not mistaken, I cannot create the cluster because it's supposed to be auto created when I execute my task.
PRADEEPCHEEKATLA 90,461 Reputation points

2024-11-15T11:59:26.4966667+00:00

@Tan Check Leam - This issue looks strange. Can you please open a support ticket to investigate further?
Chandra Boorla 5,755 Reputation points Microsoft Vendor

2024-11-15T12:08:53.5766667+00:00
Hi @Tan Check Leam

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

In addition to @PRADEEPCHEEKATLA

It appears you are facing a challenge where the On-Demand HDInsight cluster creation within Azure Data Factory is not progressing beyond the "PreparingCluster" stage. This situation could be due to a variety of factors including configuration errors, limitations on resources, or connectivity problems. Here are few troubleshooting actions and advice that might help in resolving the issue:

Check Quotas and Limits - Ensure that your Azure subscription has sufficient quota for creating new HDInsight clusters. You can check and request quota increases through the Azure Portal.

Verify Configuration - Ensure that your LinkedService configuration is correct. Common issues include incorrect resource group names, cluster names, or configurations that do not comply with HDInsight requirements.

Network Configuration - If your HDInsight cluster is in a virtual network, ensure that the network configuration is correct. Verify that the necessary network security group rules, route tables, and DNS settings are properly configured.

Regional Availability - Ensure that HDInsight is available in the region where you're trying to create the cluster. Some regions may have limitations or may not support certain configurations.

Azure Role-Based Access Control (RBAC) - Ensure that the account you are using has the necessary permissions to create HDInsight clusters. You may need to be assigned a role such as "HDInsight Cluster Operator" or have sufficient permissions in the resource group.

Check Activity Log - Navigate to the Activity Log in the Azure Portal for the resource group where you are trying to create the HDInsight cluster. Look for any failure or error messages that might provide more insight into why the cluster creation is stuck.

Logs and Diagnostics - Enable diagnostics logging for your HDInsight cluster. Check the logs for any errors or warnings that could indicate why the cluster is not progressing beyond the "PreparingCluster" stage.

Please refer to the below Microsoft documentations as it may provide you with some useful insights.

Troubleshoot a slow or failing job on a HDInsight cluster

HDInsight cluster management best practices

Capacity planning for HDInsight clusters

Azure HDInsight: Cluster creation errors

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Tan Check Leam 10 Reputation points

2024-11-16T15:30:52.6066667+00:00

Can you show me where can I check any logs generated when the cluster is creating? I have no idea what is happening because the status is just showing "PreparingCluster" stage
Chandra Boorla 5,755 Reputation points Microsoft Vendor

2024-11-18T12:18:14.91+00:00
@Tan Check Leam

Apologies for the inconvenience.

Could you please attempt to set up a new on-demand HDInsight cluster in a different region to check if the same issue persists?

If the issue still persists, please try to upgrade the HDInsight version to the latest one in case if you are using an older version. For details, please refer: Supported HDInsight versions.

If the issue continues:

I understand that the issue seems unusual, and I wasn't able to reproduce this issue. If you have a support plan, could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan, please let us know here.

For activity logs, please refer: https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-log-management

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Chandra Boorla 5,755 Reputation points Microsoft Vendor

2024-11-19T06:35:39.65+00:00

@Tan Check Leam

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thank you.
Tan Check Leam 10 Reputation points

2024-11-20T07:19:36.5433333+00:00

@Chandra Boorla

My issue is referring to the ON DEMAND HDInsights which I try to provision using the linked service. Are the recommendation above referring to the same or to the HDInsights clusters created in Azure Portal ?

FYI, I have no issue on creating the HDInsights from Azure Portal but the issue is with the ADF ON DEMAND HDInsights.

Hope this clarify the confusion
Chandra Boorla 5,755 Reputation points Microsoft Vendor

2024-11-21T05:58:14.3433333+00:00

@Tan Check Leam

I agree that this issue looks strange, and I wasn't able to reproduce this issue. If you have a support plan, could you please file a support ticket for deeper investigation and do share the SR# with us?
Chandra Boorla 5,755 Reputation points Microsoft Vendor

2024-11-22T01:25:18.93+00:00

@Tan Check Leam

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Share via

Unable to create HDInsight On Demand Cluster in Azure Data Factory

Your answer