Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.

Vishesh Agarwal 0

I am trying to create a linked service of HDInsight hadoop Clusters. With Cluster size 4, 1 hour as time to live, Provided service principal Id, Azure Key vault Secret Key, Azure Storage account and a SQL Server in Hcatalog. But I am unable to move forward from the Preparing clusters stage and the Activity gets Timed out. Please help.

Chandra Boorla 9,835 Reputation points Microsoft External Staff

2025-03-07T18:38:26.69+00:00
@Vishesh Agarwal

It seems like the issue could be related to access from the HDInsight cluster’s IP range to the SQL Server for HCatalog. Since everything else has been verified, here’s how you can ensure that the SQL Server allows access from the cluster:

Allow Azure Services to Access SQL Server - Go to your SQL Server resource in the Azure portal. In the Networking section, ensure that the option "Allow Azure services and resources to access this server" is enabled. This will allow access from any Azure service, including your HDInsight cluster.

Manually Add IP Range (if necessary) - If you have stricter firewall rules, you may need to manually add the IP range of your HDInsight cluster. While the exact IP range for HDInsight clusters isn’t always exposed, enabling the "Allow Azure services" option should usually resolve this.

Check for Virtual Network (VNET) or Private Endpoint - If your SQL Server is inside a Virtual Network (VNET) or using a Private Endpoint, ensure that the HDInsight cluster is either in the same VNET or that the necessary network routing is in place to allow communication between the cluster and the SQL Server.

Once you've confirmed these settings, try creating the HDInsight cluster again.

For more details, please refer the following documentation.

Azure SQL Database and Azure Synapse Analytics network access controls

HDInsight management IP addresses

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

1 answer

Amira Bedhiafi 29,711 Reputation points

2025-03-05T13:30:23.91+00:00

Start by verifying that the Service Principal has the necessary permissions (for example Contributor role) to create and manage HDInsight clusters.

Verify also the Azure Key Vault secret is accessible, the storage account is in the same region as the cluster, and the SQL Server for HCatalog allows access from the cluster IP range.

If the issue persists, try increasing the Time to Live or reaching out to Azure Support for further assistance.

https://stackoverflow.com/questions/36114644/azure-data-factory-hdinsight-on-demand-cluster-unable-to-instantiate-sessionhiv

https://learn.microsoft.com/en-us/answers/questions/418412/hdinsight-cluster-deployment-failure
Please sign in to rate this answer.
Vishesh Agarwal 0 Reputation points

2025-03-06T14:52:49.9366667+00:00

Rest all are verified, except the Access to Server for Cluster IP range, How do I ensure that. Also I just receive this:
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.

1 answer

Your answer