Bring your own Network Security Group (NSG) to an Azure Red Hat OpenShift (ARO) cluster

Typically, when setting up an ARO cluster, you must designate a resource group for deploying the ARO cluster object (referred to as the Base Resource Group in the following diagram). In such scenarios, you can use either the same resource group for both the virtual network (VNET) and the cluster, or you can opt for a separate resource group solely for the VNET. Neither of these resource groups directly corresponds to a single ARO cluster, granting you complete control over them. This means you can freely create, modify, or delete resources within these resource groups.

During the cluster creation process, the ARO Resource Provider (RP) establishes a dedicated resource group specific to the cluster's needs. This group houses various cluster-specific resources like node VMs, load balancers, and Network Security Groups (NSGs), as depicted by the Managed Resource Group in the following diagram. The Managed Resource Group is tightly secured, prohibiting any modifications to its contents, including the NSG linked to the VNET subnets specified during cluster creation. In some situations, the NSG generated by the ARO RP might not adhere to the security policies of certain organizations.

This article shows how to use the "bring your own" Network Security Group (NSG) feature to attach your own preconfigured NSG residing in the Base/VNET resource group (RG) (shown in the following diagram as BYO-NSG) to the ARO cluster subnets. Since you own this preconfigured NSG, you can add/remove rules during the lifetime of the ARO cluster.

General capabilities and limitations

• You need to attach your preconfigured NSGs to both master and worker subnets before you create the cluster. Failure to attach your preconfigured NSGs to both subnets results in an error.

• You can choose to use the same or different preconfigured NSGs for master and worker subnets.

• When using your own NSG, the ARO RP still creates an NSG in the Managed Resource Group (default NSG), but that NSG isn't attached to the worker or master subnets.

• You can't enable the preconfigured NSG feature on an existing ARO cluster. Currently, this feature can only be enabled at the time of cluster creation.

• The preconfigured NSG option isn't configurable from the Azure portal.

• If you used this feature during preview, your existing preconfigured clusters are now fully supported.

Note

If you're using the "bring your own" NSG feature and want to use NSG flow logs, please refer to Flow logging for network security groups in Azure Network Watcher documentation, rather than the ARO specific flow log documentation (which won't work with the bring your own NSG feature).

Using rules

Warning

Preconfigured NSGs aren't automatically updated with rules when you create Kubernetes LoadBalancer type services or OpenShift routes within the ARO cluster. Therefore, you must update these rules manually, as required. This behavior is different from the original ARO behavior wherein the default NSG is programmatically updated in such situations.

• The default ARO cluster NSG (not attached to any subnet while using this feature) will still be updated with rules when you create Kubernetes LoadBalancer type services or OpenShift routes within the ARO cluster.

• You can detach preconfigured NSGs from the subnets of the cluster created using this feature. It results in a cluster with subnets that have no NSGs. You can then attach a different set of preconfigured NSGs to the cluster. Alternatively, you can attach the ARO default NSG to the cluster subnets (at which point your cluster becomes like any other cluster that's not using this feature).

• Your preconfigured NSGs shouldn't have INBOUND/OUTBOUND DENY rules of the following types, as these can interfere with the operation of the cluster and/or hinder the ARO support/SRE teams from providing support/management. (Here, subnet indicates any or all IP addresses in the subnet and all ports corresponding to that subnet):

  • Master Subnet ←→ Master Subnet

  • Worker Subnet ←→ Worker Subnet

  • Master Subnet ←→ Worker Subnet

  • Misconfigured rules result in a signal used by Azure Monitor to help troubleshoot preconfigured NSGs.

• To allow incoming traffic to your ARO public cluster, set the following INBOUND ALLOW rules (or equivalent) in your NSG. Refer to the default NSG of the cluster for specific details and to the example NSG shown in Deployment. You can create a cluster even without such rules in the NSG.

  • For API server access → From Internet (or your preferred source IPs) to port 6443 on the master subnet.
  • For access to OpenShift router (and hence to OpenShift console and OpenShift routes) → From Internet (or your preferred source IPs) to ports 80 and 443 on the default-v4 public IP on the public Load-balancer of the cluster.
  • For access to any Load-balancer type Kubernetes service → From Internet (or your preferred source IPs) to service ports on public IP corresponding to the service on the public Load-balancer of the cluster.

Deployment

Create VNET and create and configure preconfigured NSG

1. Create a VNET, and then create master and worker subnets within it.

2. Create preconfigured NSGs with default rules (or no rules at all) and attach them to the master and worker subnets.

Create an ARO cluster and update preconfigured NSGs

1. Create the cluster.

   az aro create \
   --resource-group BASE_RESOURCE_GROUP_NAME \
   --name CLUSTER_NAME \
   --vnet VNET_NAME \
   --master-subnet MASTER_SUBNET_NAME \
   --worker-subnet WORKER_SUBNET_NAME \
   --client-id CLUSTER_SERVICE_PRINCIPAL_ID \
   --client-secret CLUSTER_SERVICE_PRINCIPAL_SECRET \
   --enable-preconfigured-nsg
   

2. Update the preconfigured NSGs with rules as per your requirements while also considering the points mentioned in Capabilities and limitations.

   The following example has the Cluster Public Load-balancer as shown in the screenshot/CLI output:

   

   $ oc get svc | grep tools
   tools LoadBalancer 172.30.182.7 20.141.176.3 80:30520/TCP 143m
   $ $ oc get svc -n openshift-ingress | grep Load
   router-default LoadBalancer 172.30.105.218 20.159.139.208 80:31157/TCP,443:31177/TCP 
   5d20