Muokkaa

Jaa

Use firewall to restrict outbound traffic using Azure portal

  • Artikkeli

Note

We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.

Only basic support will be available until the retirement date.

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

When an enterprise wants to use their own virtual network for the cluster deployments, securing the traffic of the virtual network becomes important. This article provides the steps to secure outbound traffic from your HDInsight on AKS cluster via Azure Firewall using Azure portal.

The following diagram illustrates the example used in this article to simulate an enterprise scenario:

Diagram showing the network flow.

Create a virtual network and subnets

  1. Create a virtual network and two subnets.

    In this step, set up a virtual network and two subnets for configuring the egress specifically.

    Diagram showing creating a virtual network in the resource group using Azure portal step number 2.

    Diagram showing creating a virtual network and setting IP address using Azure portal step 3.

    Diagram showing creating a virtual network and setting IP address using Azure portal in step number four.

    Important

    • If you add NSG in subnet , you need to add certain outbound and inbound rules manually. Follow use NSG to restrict the traffic.
    • Don't associate subnet hdiaks-egress-subnet with a route table because HDInsight on AKS creates cluster pool with default outbound type and can't create the cluster pool in a subnet already associated with a route table.

Create HDInsight on AKS cluster pool using Azure portal

  1. Create a cluster pool.

    Diagram showing creating a HDInsight on AKS cluster pool using Azure portal in step number five.

    Diagram showing creating a HDInsight on AKS cluster pool networking using Azure portal step 6.

  2. When HDInsight on AKS cluster pool is created, you can find a route table in subnet hdiaks-egress-subnet.

    Diagram showing creating a HDInsight on AKS cluster pool networking using Azure portal step 7.

Get AKS cluster details created behind the cluster pool

You can search your cluster pool name in portal, and go to AKS cluster. For example,

Diagram showing creating a HDInsight on AKS cluster pool kubernetes networking using Azure portal step 8.

Get AKS API Server details.

Diagram showing creating a HDInsight on AKS cluster pool kubernetes networking using Azure portal step 9.

Create firewall

  1. Create firewall using Azure portal.

    Diagram showing creating a firewall using Azure portal step 10.

  2. Enable DNS proxy server of firewall.

    Diagram showing creating a firewall and DNS proxy using Azure portal step 11.

  3. Once the firewall is created, find the firewall internal IP and public IP.

    Diagram showing creating a firewall and DNS proxy internal and public IP using Azure portal step 12.

Add network and application rules to the firewall

  1. Create the network rule collection with following rules.

    Diagram showing adding firewall rules using Azure portal step 13.

  2. Create the application rule collection with following rules.

    Diagram showing adding firewall rules using Azure portal step 14.

Create route in the route table to redirect the traffic to firewall

Add new routes to route table to redirect the traffic to the firewall.

Diagram showing adding route table entries using Azure portal step 15.

Diagram showing how to add route table entries using Azure portal step 15.

Create cluster

In the previous steps, we have routed the traffic to firewall.

The following steps provide details about the specific network and application rules needed by each cluster type. You can refer to the cluster creation pages for creating Apache Flink, Trino, and Apache Spark clusters based on your need.

Important

Before creating the cluster, make sure to add the following cluster specific rules to allow the traffic.

Trino

  1. Add the following rules to application rule collection aksfwar.

    Diagram showing adding application rules for Trino Cluster using Azure portal step 16.

  2. Add the following rule to network rule collection aksfwnr.

    Diagram showing how to add application rules to network rule collection for Trino Cluster using Azure portal step 16.

    Note

    Change the Sql.<Region> to your region as per your requirement. For example: Sql.WestEurope

  1. Add the following rule to application rule collection aksfwar.

    Diagram showing adding application rules for Apache Flink Cluster using Azure portal step 17.

Apache Spark

  1. Add the following rules to application rule collection aksfwar.

    Diagram showing adding application rules for Apache Flink Cluster using Azure portal step 18.

  2. Add the following rules to network rule collection aksfwnr.

    Diagram showing how to add application rules for Apache Flink Cluster using Azure portal step 18.

    Note

    1. Change the Sql.<Region> to your region as per your requirement. For example: Sql.WestEurope
    2. Change the Storage.<Region> to your region as per your requirement. For example: Storage.WestEurope

Solving symmetric routing issue

The following steps allow us to request cluster by cluster load balancer ingress service and ensure the network response traffic doesn't flow to firewall.

Add a route to the route table to redirect the response traffic to your client IP to Internet and then, you can reach the cluster directly.

Diagram showing how to solve symmetric routing issue with adding a route table entry in step number 19.

If you can't reach the cluster and have configured NSG, follow use NSG to restrict the traffic to allow the traffic.

Tip

If you want to permit more traffic, you can configure it over the firewall.

How to Debug

If you find the cluster works unexpectedly, you can check the firewall logs to find which traffic is blocked.