Rediger

Del via

Basic Load balancer deprecation: Guidelines for Azure HDInsight

  • Artikkel

This article describes the details about the impact on HDInsight clusters and the necessary steps required as HDInsight service is transitioning to use standard load balancers for all its cluster configurations.

This transition is done in line with the announcement of retirement of Azure basic load balancer by 30 September 2025 and no support for new deployment by Mar 31, 2025. For more information, see Azure Basic Load Balancer retires on 30 September 2025. Upgrade to Standard Load Balancer.

As part of the migration from the basic load balancer to the standard load balancer, the public IP address will be upgraded from basic SKU to standard SKU. For more details, see Upgrade to Standard SKU public IP addresses in Azure by 30 September 2025 - Basic SKU will be retired. Additionally, default outbound access will be deprecated. For further information, see Default outbound access for VMs in Azure will be retired - transition to a new method of internet access.

Note

Please be advised not to alter any components created by HDInsight within your VNet, including load balancers, IP addresses, network interfaces, etc. Any modifications you make to these components may be reverted during cluster maintenance.

Impact on HDInsight clusters

Long running clusters (when you don’t create and delete cluster frequently for your use case)

  • Scenario 1: Existing HDInsight clusters without custom virtual network (No virtual network used during cluster creation).

    • No immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster before Mar 31, 2025 to avoid potential disruption.

  • Scenario 2: Existing HDInsight clusters with custom virtual network (Used your own VNet for cluster creation).

    • No immediate disruption until Mar 31, 2025. However, we strongly recommend re-creating the cluster before Mar 31, 2025 to avoid potential disruption.

New cluster creation

Due to the deprecation of the default outbound access, a new outbound connectivity method is required by the HDInsight cluster. There are several ways provided in the document Source Network Address Translation (SNAT) for outbound connections that could provide outbound connectivity for a cluster. The only compatible way with HDInsight is to associate a NAT gateway to the subnet, which supports auto-scaling features of HDInsight clusters. NAT gateway provides outbound network connectivity for the cluster. NSG controls both the inbound and outbound traffic, which is required by the standard load balancer.

  • Scenario 1: HDInsight clusters without custom virtual network (Creating cluster without any virtual network).

    • In this case, no impact. You can recreate the cluster directly.

  • Scenario 2: HDInsight clusters with custom virtual network (Using your own VNet during cluster creation).

    • In this case, there are two options to create a cluster

      Approach 1: Create the cluster with a new subnet

      1. Create a new NAT gateway and a new Network Security Group (NSG) or use the existing ones. NAT gateway provides outbound network connectivity for the cluster. NSG controls both the inbound and outbound traffic, which is required by the standard load balancer.

        Note

        You can use an existing NAT gateway and NSG.

        Follow the docs Quickstart: Create a NAT gateway - Azure portal, and Create, change, or delete an Azure network security group to create a new NAT gateway and a new network security group. You can refer to this document Control network traffic in Azure HDInsight to set up correct NSG rules.

      2. Create a new subnet and associate the subnet with the NAT gateway and network security group.

        Screenshot showing how to create a subnet.

        Note

        If you are unable to find the NAT gateway, see, FAQ of the NAT gateway Azure NAT Gateway frequently asked questions.

      3. Create a new cluster with the subnet.

        Screenshot showing how to create a HDInsight cluster.

      Approach 2: Create the cluster using the existing subnet

      Your existing virtual network may be incompatible with Azure Standard Load Balancer, to upgrade your existing custom virtual network to integrate with Azure standard load balancer. You need to attach a network security group and a NAT gateway to your existing subnet. Since the existing subnet which has HDInsight clusters with Azure basic load balancers can't be associated with an NAT gateway due to incompatibility with basic load balancer, there are two scenarios:

Note

If you are using an ESP cluster with MFA disabled, ensure to check the MFA status once cluster is recreated using a NAT gateway.

Next steps

Plan a virtual network for Azure HDInsight