HDInsight: - CoC (Calculation of Cores)

If you ever notice “99% of XX HDINSIGHT CORES” in your HDInsight dash board on wonder what it means? Or during cluster creation if you get error/warning “The cluster can have at most 0 data nodes” and wonder what
it means?

In this short article I’ll just explain what above means and show core calculation.

So let’s take an example if you have 740 HDInsight Core on your subscription and 740 HDInsight Core being used by 8 clusters (167 nodes). Each worker node consumes 4 cores and each head node consumes 8 cores.

Here is the calculation:-

Clusters	Data Nodes x Cores	Head Nodes x Cores	Total Cores used
First cluster	16 x 4 = 64	1 x 8 = 8	72
Second cluster	16 x 4 = 64	1 x 8 = 8	72
Third cluster	16 x 4 = 64	1 x 8 = 8	72
Forth cluster	16 x 4 =64	1 x 8 = 8	72
Fifth cluster	32 x 4= 128	1 x 8 = 8	136
Six cluster	32 x 4= 128	1 x 8 = 8	136
Seven cluster	32 x 4= 128	1 x 8 = 8	136
Eight cluster	7 x 4 =28	1 x 8 = 8	36
TOTAL	668 cores	64 cores	732

 

So if you try to create cluster at least with 1 node it requires 12 cores which is not available to subscription. That’s why you get error “The cluster can have at most 0 data nodes” and message “99% of 740 HDInsight cores” in the dashboard.

In case it’s require to increase the core, one of the options is to raise ticket from the portal.

 

The other option is reuse existing cluster in case you are not using it.

 

HTH.