Azure HDInsight
An Azure managed cluster service for open-source analytics.
220 questions
0 answers
HDInsight Spark-type cluster failing to be deployed due to failed start up of SPARK3_THRIFTSERVER service
In Microsoft Azure the Spark-type HDInsight cluster (with 5.1 stack version) is failing to be deployed with an AmbariClusterCreationFailedErrorCode error-code due to an internal error of Spark3 hriftserver service on Ambari: Execute['!…
asked 2025-02-24T15:42:41.6233333+00:00
Tibor, Ecseki
0
Reputation points
edited the question 2025-02-24T16:21:25.26+00:00
Tibor, Ecseki
0
Reputation points
1 answer
Creating Azure HDInsight process is stuck at "Status: HDInsight Configuration"
Hey, During the creation of Azure HDinsight (Hadoop Cluster) The process seems to be stuck at "Status: HDInsight Configuration" For more than 20 minutes; after it did manage to somewhat create it (note the dashboard itself is up and running),…
asked 2025-02-13T09:40:47.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 42%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
lidorbenshitrit
0
Reputation points
commented 2025-02-17T11:53:28.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,340
Reputation points Microsoft Vendor
2 answers
Issue Creating HDInsight Cluster with Virtual Network
I am trying to create an HDInsight cluster using Terraform. However, I encounter the following error when connecting it to a virtual network: │ Error: creating Hadoop HDInsight Cluster (Subscription: "XXX" │ Resource Group Name:…
asked 2025-01-28T18:41:16.3233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 13%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEJ%3C/text%3E%3C/svg%3E)
Emily Jil Sauer
0
Reputation points
commented 2025-02-17T09:40:38.7333333+00:00
AnnuKumari-MSFT
34,021
Reputation points Microsoft Employee
1 answer
Sku family standardev3family quota is not enough
I'm attempting to go through this tutorial: https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal?source=docs I cannot create a HDInsight cluster because of this error that I'm seeing on the Configuration +…
asked 2024-12-19T18:33:00.0833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 9%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Matt
0
Reputation points
commented 2024-12-23T02:43:17.32+00:00
Ganesh Gurram
4,340
Reputation points Microsoft Vendor
1 answer
The resource api version is unavailable, how to delete resource
the service corresponding to the resource located under sub has been taken offline or the API version is unavailable, which makes it impossible to delete this resource. How to delete the reources in portal? The VM network and other resources…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 80%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXW%3C/text%3E%3C/svg%3E)
1 answer
In Data Lake Storage Account Gen 2 unexpected reduction of calls API "GetPathStatus" and "ListFilesystemDir"
Hi, I am reaching out with an inquiry regarding the transactions quantity in an data lake storage account gen 2 on Azure. We have noticed that starting on 28th october, the transactions quantity associated with storage account have significantly…
asked 2024-11-12T11:14:53.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 81%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Javier Coll Doñetz
0
Reputation points
commented 2024-11-25T05:56:16.9266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
8,171
Reputation points Microsoft Vendor
0 answers
Unable to create HDInsight On Demand Cluster in Azure Data Factory
I am getting the issue to creating the On Demand HDInsights in Azure Data Factory. The status is stuck at "PreparingCluster". Here's my LinkedService configuration. OS type : Linux
asked 2024-11-13T03:52:15.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 19%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Tan Check Leam
10
Reputation points
commented 2024-11-22T01:25:18.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,870
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Compatibility of Azure HDInsight Kafka
Can you provide clarity on whether Azure HDInsight Kafka works with Azure Event Registry? Additionally, I’m interested in any compatibility details or constraints that might apply. Could you share more on this integration?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Difference between synapse and hdinsight
What are major difference between synapse and HD insight and under what circumstances one should use synapse and vice versa
asked 2021-10-12T12:43:25.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 34%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shivam Ramola
71
Reputation points
commented 2024-11-06T12:09:12.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 92%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Michael Schmidt
340
Reputation points
1 answer
Spark HDInsight cluster deployment fails with an internal server error
I've tried deploying a 5.1.3 HDInsight cluster twice, and both times it fails with an internal server error. JSON below. I chose all the defaults except for using an existing storage account and virtual network. Any suggestions appreciated. { …
asked 2024-10-27T20:54:05.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 42%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Michael P Spertus
0
Reputation points
commented 2024-10-30T13:16:19.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,400
Reputation points Microsoft Vendor
0 answers
Azure HDInsight: Managing Unhealthy Nodes and Sessions
Hi Team, We are currently facing an issue with Unhealthy Nodes on Azure HDInsight. We have one centralized HDInsight cluster shared between Data Teams, where multiple Spark and Hive jobs are being submitted continuously. However, due to either overload…
asked 2024-10-21T10:38:47.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 91%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mahesh Sanga
0
Reputation points Microsoft Employee
commented 2024-10-24T19:54:07.18+00:00
Chandra Boorla
8,870
Reputation points Microsoft Vendor
1 answer
spark notebooks with terraform
i want to create notebooks using automation for automation i am using terraform, can we invoke spark notebooks in terraform
asked 2024-08-28T17:00:07.9466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 92%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gnana Sai krishna
80
Reputation points
commented 2024-09-02T08:59:22.7133333+00:00
PRADEEPCHEEKATLA
90,556
Reputation points
1 answer
Stucking in Creating HDinsight Clusters
Hi, My regional quota for vCPUs is 20, but I'm still unable to create the HDInsight clusters. Can anyone help me?
asked 2024-08-14T20:09:43.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
Ziwei Zeng
10
Reputation points
answered 2024-08-16T11:40:40.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
13,980
Reputation points Microsoft Vendor
1 answer
Unable to run Spark activity through data factory pipeline
After configuring the network and ensuring that the test connection between data factory and the respective spark cluster is successful, I am unable to submit nor view any spark jobs that are submitted through the data pipeline using the spark activity.…
asked 2024-07-31T10:23:06.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 77%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Aleksandr Maxermis
0
Reputation points
edited a comment 2024-08-06T15:14:11.93+00:00
Smaran Thoomu
20,400
Reputation points Microsoft Vendor
0 answers
Kafka REST proxy endpoint is corrupting and wrongly parsing data
Hello Team, I recently activated REST proxy endpoint for HdInsight kafka. The endpoint is used to produce data to a topic, and upon consuming from the topic I observed that the data is wrongly encoded and parsed. If I try to produce data from the client…
asked 2024-07-11T20:33:51.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 87%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AwesomePaul
20
Reputation points
commented 2024-07-22T05:02:13.68+00:00
PRADEEPCHEEKATLA
90,556
Reputation points
1 answer
How can we integrate hadoop logs in insights
Hi , Need to integrage spark application logs in azure insights or api insgihts. And also need to build monitoring dashboard for azure spark hid ingishts Any inputs please
asked 2024-07-17T11:11:18.2966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 41%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Panneeru, Vikram
0
Reputation points
answered 2024-07-17T17:34:56.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
31,206
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Unable to use HdinsightKafka REST proxy API after authenticating with access token.
Hello Team, I'm having issues connecting to HdInsight kafka using REST proxy endpoint. I followed the steps here (https://learn.microsoft.com/en-us/azure/hdinsight/kafka/rest-proxy), and was able to get token and authenticate the endpoint. However,…
asked 2024-07-10T12:25:43.3633333+00:00
AwesomePaul
20
Reputation points
commented 2024-07-12T08:35:26.7166667+00:00
AwesomePaul
20
Reputation points
1 answer
Kafka REST Proxy Authentication
We would like to develop a HDInsight Kafka cluster to share real time data with a subcontractor. The REST proxy documentation indicates that "Kafka clients that need access to the REST proxy should be registered to a group by the group owner."…
asked 2022-02-18T18:30:46.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 43%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mike McNulty
1
Reputation point
commented 2024-07-10T10:15:39.4366667+00:00
AwesomePaul
20
Reputation points
1 answer
Unclear what generates costs
After requesting a quota increase I managed to create a HDInsight cluster. The support told me that services generate costs only when they are in use. According to this page https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-delete-cluster I…
asked 2024-07-05T14:13:40.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 62%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB5%3C/text%3E%3C/svg%3E)
blubb-5305
20
Reputation points
commented 2024-07-08T22:11:19.6433333+00:00
Bhargava-MSFT
31,206
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight Version 3.6 is Deprecated,User SubscriptionId 'XXX' does not have cores left to create resource 'hdi-kafka'. Required: 38, Available: 0.
I am going through this tutorial and getting this error when trying to create a HDInsight cluster: HDInsight Version 3.6 is Deprecated,User SubscriptionId 'XXX' does not have cores left to create resource 'hdi-kafka'. Required: 38, Available: 0. This…
asked 2024-07-02T09:00:09.9866667+00:00
blubb-5305
20
Reputation points
accepted 2024-07-05T13:38:15.2733333+00:00
blubb-5305
20
Reputation points