Rediger

Del via

Network isolated Azure Kubernetes Service (AKS) clusters (Preview)

  • Artikkel

Organizations typically have strict security and compliance requirements to regulate egress (outbound) network traffic from a cluster to eliminate risks of data exfiltration. By default, Azure Kubernetes Service (AKS) clusters have unrestricted outbound internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. The conceptual document on outbound network and FQDN rules for AKS clusters provides a list of required endpoints for the AKS cluster and its optional add-ons and features.

One solution to restricting outbound traffic from the cluster is to use a firewall device to restrict traffic based on domain names. Configuring a firewall manually with required egress rules and FQDNs is a cumbersome and complicated process.

Another solution, a network isolated AKS cluster (preview), simplifies setting up outbound restrictions for a cluster out of the box. The cluster operator can then incrementally set up allowed outbound traffic for each scenario they want to enable. A network isolated AKS cluster thus reduces the risk of data exfiltration.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

How a network isolated cluster works

The following diagram shows the network communication between dependencies for an AKS network isolated cluster.

Traffic diagram of network isolated AKS cluster.

AKS clusters pull images required for the cluster and its features or add-ons from the Microsoft Artifact Registry (MAR). This image pull allows AKS to provide newer versions of the cluster components and to also address critical security vulnerabilities. A network isolated cluster attempts to pull those images from a private Azure Container Registry (ACR) instance connected to the cluster instead of pulling from MAR. If the images aren't present, the private ACR pulls them from MAR and serves them via its private endpoint, eliminating the need to enable egress from the cluster to the public MAR endpoint.

The following options are supported for a private ACR with network isolated clusters:

  • AKS-managed ACR - AKS creates, manages, and reconciles an ACR resource in this option. You don't need to assign any permissions or manage the ACR. AKS manages the cache rules, private link, and private endpoint used in the network isolated cluster. An AKS-managed ACR follows the same behavior as other resources (route table, Azure Virtual Machine Scale Sets, etc.) in the infrastructure resource group. To avoid the risk of cluster components or new node bootstrap failing, do not update or delete the ACR, its cache rules, or its system images.. The AKS-managed ACR is continuously reconciled so that cluster components and new nodes work as expected.

    Note

    After you delete an AKS network isolated cluster, related resources such as the AKS-managed ACR, private link, and private endpoint are automatically deleted.

  • Bring your own (BYO) ACR - The BYO ACR option requires creating an ACR with a private link between the ACR resource and the AKS cluster. See Connect privately to an Azure container registry using Azure Private Link to understand how to configure a private endpoint for your registry.

    Note

    When you delete the AKS cluster, the BYO ACR, private link, and private endpoint aren't deleted automatically. If you add customized images and cache rules to the BYO ACR, they persist after cluster reconciliation, after you disable the feature, or after you delete the AKS cluster.

When creating a network isolated AKS cluster, you can choose one of the following private cluster modes:

  • Private link-based AKS cluster - The control plane or API server is in an AKS-managed Azure resource group, and your node pool is in your resource group. The server and the node pool can communicate with each other through the Azure Private Link service in the API server virtual network and a private endpoint which is exposed on the subnet of your AKS cluster.
  • API Server VNet Integration (Preview) - A cluster configured with API Server VNet Integration projects the API server endpoint directly into a delegated subnet in the virtual network where AKS is deployed. API Server VNet Integration enables network communication between the API server and the cluster nodes without requiring a private link or tunnel.

Limitations

Frequently asked questions

What's the difference between network isolated cluster and Azure Firewall?

A network isolated cluster doesn't require any egress traffic beyond the VNet throughout the cluster bootstrapping process. A network isolated cluster will have outbound type as either none or block. If the outbound type is set to none, then AKS doesn't set up any outbound connections for the cluster, allowing the user to configure them on their own. If the outbound type is set to block, then all outbound connections are blocked.

A firewall typically establishes a barrier between a trusted network and an untrusted network, such as the Internet. Azure Firewall, for example, can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination, giving you fine-grained egress traffic control, but at the same time allows you to provide access to the FQDNs encompassing an AKS cluster’s outbound dependencies (something that NSGs can't do). For example, you can set outbound type of the cluster to userDefinedRouting to force outbound traffic through the firewall and then configure FQDN restrictions on outbound traffic.

In summary, while Azure Firewall can be used to define egress restrictions on clusters with outbound requests, network isolated clusters go further on secure-by-default posture by eliminating or blocking the outbound requests altogether.

Do I need to set up any allowlist endpoints for the network isolated cluster to work?

The cluster creation and bootstrapping stages don't require any outbound traffic from the network isolated cluster. Images required for AKS components and addons are pulled from the private ACR connected to the cluster instead of pulling from Microsoft Artifact Registry (MAR) over public endpoints.

After setting up a network isolated cluster, if you want to enable features or add-ons that need to make outbound requests to their service endpoints, private endpoints can be set up to the services powered by Azure Private Link.

Can I manually upgrade packages to upgrade node pool image?

Manually upgrading packages based on egress to package repositories isn't supported. Instead, you can autoupgrade your node OS images. Only NodeImage node OS autoupgrade channel is supported for network isolated clusters.

Next steps