Spark HDInsight cluster deployment fails with an internal server error

Question

I've tried deploying a 5.1.3 HDInsight cluster twice, and both times it fails with an internal server error. JSON below. I chose all the defaults except for using an existing storage account and virtual network. Any suggestions appreciated.

{
    "authorization": {
        "action": "Microsoft.HDInsight/clusters/write",
        "scope": "/subscriptions/b2c3e770-2991-4cba-a0ff-2a4dea242464/resourcegroups/MPCS53014RG/providers/Microsoft.HDInsight/clusters/spark3-mpcs53014-2024"
    },
    "caller": "spertus@UCHICAGO.EDU",
    "channels": "Operation",
    "claims": {
        "aud": "https://management.core.windows.net/",
        "iss": "https://sts.windows.net/83b02c92-5f26-48ed-9e5b-6c2fca46a8e6/",
        "iat": "1730060096",
        "nbf": "1730060096",
        "exp": "1730064124",
        "http://schemas.microsoft.com/claims/authnclassreference": "1",
        "aio": "ATQAy/8YAAAAUKLFhA4ZasItJuy26CL1TGtV+E04Chs8X5lazPznHhvM8sVXvBQxhFUjTpSAj1GQ",
        "http://schemas.microsoft.com/claims/authnmethodsreferences": "pwd",
        "appid": "c44b4083-3bb0-49c1-b47d-974e53cbdf3c",
        "appidacr": "2",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "Spertus",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "Michael",
        "groups": "fba39e04-af97-4aba-8b98-f7894e0e8b22,ffa5e605-b678-4861-8bdf-c94f1921a4c5,d82f0111-d9fa-4edf-8bce-dd11ac102c87,41a30c2d-0c49-4b8e-b23c-74f64864256c,c6e50850-fe2f-4bf9-8b34-a8195152ce6f,496d4d73-f18d-43ae-a0d5-6188e69efac3,6d210f7d-0fd8-4866-8bb4-110fb41d2124,1b40a380-1d81-41a2-abb4-c34ead2485ea,b4c3fd81-7a1e-4ca4-9818-706b5b3a019f,23f2a48e-0dd9-4d27-9016-3347b88eacd1,82a3e095-049b-48f3-b7dd-06f806c2827d,1e7ba596-05e0-43a4-997e-72c213f2b4fc,86d0dbae-9ea2-4fcf-a981-eda64c2feab4,78ee38b4-f158-443a-900b-58eb5b7e8f0f,ba43cbc0-7527-4519-ab11-2ec4bc4dd1b7,8ff4dac9-ae12-4d2e-b5d3-ff6ac6b93406,980c47ff-613a-4805-8b2d-fa53c15a4ed7",
        "idtyp": "user",
        "ipaddr": "24.148.80.78",
        "name": "Michael P Spertus",
        "http://schemas.microsoft.com/identity/claims/objectidentifier": "491208eb-696b-4f59-b504-a33bcb645f48",
        "onprem_sid": "S-1-5-21-1644491937-1604221776-725345543-92611",
        "puid": "1003BFFD9FB092E3",
        "rh": "0.AXYAkiywgyZf7UieW2wvykao5kZIf3kAutdPukPawfj2MBO0ABk.",
        "http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "eOYRz-GY2k8IuNOxrw3vrFP88htF443pvMIbJ1octlE",
        "http://schemas.microsoft.com/identity/claims/tenantid": "83b02c92-5f26-48ed-9e5b-6c2fca46a8e6",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "spertus@UCHICAGO.EDU",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "spertus@UCHICAGO.EDU",
        "uti": "6QZjzVLVQEastazsP88IAA",
        "ver": "1.0",
        "xms_idrel": "1 16",
        "xms_tcdt": "1450452263"
    },
    "correlationId": "4eb1c99d-3823-4135-bc6d-430da2231163",
    "description": "",
    "eventDataId": "b931372c-cf63-4d41-8f20-363e595e9928",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2024-10-27T20:40:27.3411586Z",
    "id": "/subscriptions/b2c3e770-2991-4cba-a0ff-2a4dea242464/resourcegroups/MPCS53014RG/providers/Microsoft.HDInsight/clusters/spark3-mpcs53014-2024/events/b931372c-cf63-4d41-8f20-363e595e9928/ticks/638656584273411586",
    "level": "Error",
    "operationId": "2ecb6285-6265-4515-b1f0-e16c2d7c5c2b",
    "operationName": {
        "value": "Microsoft.HDInsight/clusters/write",
        "localizedValue": "Create or Update Cluster"
    },
    "resourceGroupName": "MPCS53014RG",
    "resourceProviderName": {
        "value": "Microsoft.HDInsight",
        "localizedValue": "Microsoft.HDInsight"
    },
    "resourceType": {
        "value": "Microsoft.HDInsight/clusters",
        "localizedValue": "Microsoft.HDInsight/clusters"
    },
    "resourceId": "/subscriptions/b2c3e770-2991-4cba-a0ff-2a4dea242464/resourcegroups/MPCS53014RG/providers/Microsoft.HDInsight/clusters/spark3-mpcs53014-2024",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "",
        "localizedValue": ""
    },
    "submissionTimestamp": "2024-10-27T20:41:32Z",
    "subscriptionId": "b2c3e770-2991-4cba-a0ff-2a4dea242464",
    "tenantId": "83b02c92-5f26-48ed-9e5b-6c2fca46a8e6",
    "properties": {
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"InternalServerError\",\"message\":\"Internal server error occurred while processing the request. Please retry the request or contact support.\"}]}}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/b2c3e770-2991-4cba-a0ff-2a4dea242464/resourcegroups/MPCS53014RG/providers/Microsoft.HDInsight/clusters/spark3-mpcs53014-2024",
        "message": "Microsoft.HDInsight/clusters/write",
        "hierarchy": "83b02c92-5f26-48ed-9e5b-6c2fca46a8e6/Production-Management-Group/b2c3e770-2991-4cba-a0ff-2a4dea242464"
    },
    "relatedEvents": []
}

Answer 1

Hi @Michael P Spertus
Thank you for reaching out, and I understand the frustration of encountering repeated internal server errors when deploying the HDInsight cluster.

Given that you’re using a custom storage account and virtual network, I suggest the following steps to help troubleshoot:

1. Verify that the storage account and virtual network configurations are correctly set up with appropriate permissions for HDInsight. Sometimes access-related issues can lead to unexpected errors during deployment.
2. Occasionally, region-based quota limits or temporary issues in the Azure region might cause deployment failures. Ensure that the resource group’s region supports the HDInsight cluster configuration you’re attempting.
3. You could attempt deploying through the Azure CLI or PowerShell to see if a different method yields more insights or error details.

If the issue persists after these checks, it may be due to an underlying issue with the HDInsight service. In that case, I would suggest you raise the support ticket for further investigation.

I hope this helps. Please let me know if you have any questions.