HDInsight Spark-type cluster failing to be deployed due to failed start up of SPARK3_THRIFTSERVER service

Question

In Microsoft Azure the Spark-type HDInsight cluster (with 5.1 stack version) is failing to be deployed with an AmbariClusterCreationFailedErrorCode error-code due to an internal error of Spark3 hriftserver service on Ambari:

Execute['! /usr/hdp/current/spark3-thriftserver/bin/beeline -u 'jdbc:hive2://hn0-spark.2cnbgqf2x5rudcy2kit0pqodfh.parx.internal.cloudapp.net:10002/default;transportMode=http' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL

This prevents the hdiwatchdog service to check OK the health of Spark3 service, and the deployment process will eventually fail.

What was tried unsiccessfully during multiple failed deployment attempts:

• restarting the failing thriftserver, and the entire cluster
• ssh local port forwarding on the two failing thristserver node
• and delete the service as it is (our product is not using hive, hence it is not really required)
• turning on maintenance mode on spark thriftserver
• changing its port to 10000 on ambari.

If anyone has encountered a similar issue while deploying a Spark HDInsight cluster, it would be a great help if you could share your solution to the above-mentioned error!