Issue Creating HDInsight Cluster with Virtual Network

Question

I am trying to create an HDInsight cluster using Terraform. However, I encounter the following error when connecting it to a virtual network:

│ Error: creating Hadoop HDInsight Cluster (Subscription: "XXX"

│ Resource Group Name: "XXX"

│ Cluster Name: "HadoopCluster371113"): polling after Create: polling failed: the Azure API returned the following error:

│

│ Status: "InvalidNetworkConfigurationErrorCode"

│ Code: ""

│ Message: "Virtual Network configuration is incompatible with HDInsight requirements. Error: 'Cluster Validation timed out.', Please follow https://go.microsoft.com/fwlink/?linkid=853974 to fix it. "

│ Activity Id: ""

│

│ ---

│

│ API Response:

│

│ ----[start]----

│ {"status":"Failed","error":{"code":"InvalidNetworkConfigurationErrorCode","message":"Virtual Network configuration is incompatible with HDInsight requirements. Error: 'Cluster Validation timed out.', Please follow https://go.microsoft.com/fwlink/?linkid=853974 to fix it. "}}

│ -----[end]-----

│

│

│ with module.hadoop[0].azurerm_hdinsight_hadoop_cluster.hadoop_cluster,

│ on hadoopModule\main.tf line 1, in resource "azurerm_hdinsight_hadoop_cluster" "hadoop_cluster":

│ 1: resource "azurerm_hdinsight_hadoop_cluster" "hadoop_cluster" {

The solutions provided on the referenced webpage (https://go.microsoft.com/fwlink/?linkid=853974) do not resolve the issue, and this specific error is not described there.

VNet Setup:

resource "azurerm_virtual_network" "vnet" {

name = "vnet"

address_space = ["10.0.0.0/16"]

location = var.location

resource_group_name = var.resource_group_name

}

resource "azurerm_subnet" "subnet" {

name = "default"

resource_group_name = var.resource_group_name

virtual_network_name = azurerm_virtual_network.vnet.name

address_prefixes = ["10.0.0.0/24"]

}

The error also occurs when you set up the vnet with default settings via the Azure portal.

Additionally:

The cluster is not accessible via SSH.

I have also tested creating the cluster via the Azure portal using different cluster types (Hadoop, HBase, Spark, and Kafka), but the same issue persists. (I have not tested the Interactive Query cluster type.)

Could you please provide guidance or investigate this issue further?

Thank you for your assistance.

Answer 1

I am getting the same issue. Something is broken on azure side ? we have 2 other clusters which are working fine only new cluster creation is giving issue. From UI or terraform doesnt matter it will fail

Answer 2

I have an issue in a similar context, I am trying to deploy kafka and spark using tutorial https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-kafka-spark-structured-streaming but the ARM template (json) is outdated. How to fix it. Please update the template