Deploy Microsoft Software Defined Networking (SDN)
Applies to: AKS on Azure Stack HCI 22H2, AKS on Windows Server
This article describes how to deploy AKS infrastructure and workload VMs to an SDN Virtual Network using our SDN Software Load Balancer for all AKS Arc load balancing scenarios. AKS enabled by Azure Arc offers a fully supported container platform that can run cloud-native applications on the Kubernetes container orchestration platform. The architecture supports running virtualized Windows and Linux workloads.
Limitations
The following features are out of scope and not supported in this GA release:
- Attaching pods and containers to an SDN virtual network.
- Pods use Flannel or Calico (default) as the network provider.
- Network policy enforcement using the SDN Network Security Groups.
- The SDN Network Security Groups can still be configured outside of AKS Arc using SDN tools (REST/PowerShell/Windows Admin Center/SCVMM), but Kubernetes NetworkPolicy objects don't configure them.
- Attaching AKS Arc VM NICs to SDN logical networks.
- Installation using Windows Admin Center.
- Physical host to AKS Arc VM connectivity: VM NICs are joined to an SDN virtual network and thus aren't accessible from the host by default. For now, you can enable this connectivity manually by attaching a public IP directly to the VM using the SDN Software Load Balancer.
Prerequisites
To deploy AKS enabled by Arc with SDN, make sure your environment satisfies the deployment criteria of both AKS Arc and SDN.
- AKS Arc requirements
- SDN requirements: Plan a Software Defined Network infrastructure
Note
SDN integration with AKS Arc only requires Network Controller and Software Load Balancer. Gateway VMs are optional.
Install and prepare SDN for AKS Arc
The first step is to install SDN. To install SDN, we recommend SDN Express or Windows Admin Center. A reference configuration file that deploys all the needed SDN infrastructure components can be found here: Software Load Balancer.psd1.
Once the SDN Express deployment completes, there should be a screen that reports the status as healthy.
If anything went wrong or is being reported as unhealthy, see Troubleshooting SDN.
It's important that SDN is healthy before proceeding. If you deploy SDN in a new environment, we also recommend creating test VMs and verifying connectivity to the load balancer VIPs. See how to create and attach VMs to an SDN virtual network using Windows Admin Center.
Steps to install AKS
Initialize and prepare all the physical host machines for AKS Arc. See Deploy an AKS host for the most up-to-date instructions.
Install the AKS-HCI PowerShell module
See Install the AksHci PowerShell module for information about installing the AKS-HCI PowerShell module.
Note
After you complete this step, refresh or reload any opened PowerShell sessions to reload the modules.
Register the resource provider to your subscription
For information about how to register the resource provider to your subscription, see Install the AksHci PowerShell module.
Prepare your machines for deployment
For information about how to prepare your machines for deployment, see Prepare your machines for deployment.
Configure AKS for installation
Choose one of your Azure Stack HCI servers to drive the creation of AKS Arc. There are three steps that need to be done prior to installation:
Configure the AKS network settings for SDN; for example, using:
- SDN Virtual network "10.20.0.0/24" (10.20.0.0 – 10.20.0.255). A virtualized network, and you can use any IP subnet. This subnet does not need to exist on your physical network.
- vSwitch name "External." The external vSwitch on the Azure Stack HCI servers. Ensure that you use the same vSwitch that was used for SDN deployment.
- Gateway "10.20.0.1." This address is the gateway for your virtual network.
- DNS Server "10.127.130.7." The DNS server for your virtual network.
$vnet = New-AksHciNetworkSetting –name "myvnet" –vswitchName "External" -k8sNodeIpPoolStart "10.20.0.2" -k8sNodeIpPoolEnd "10.20.0.255" -ipAddressPrefix "10.20.0.0/24" -gateway "10.20.0.1" -dnsServers "10.127.130.7"
Parameter Description -name
Name of virtual network in AKS enabled by Arc (must be lowercase). -vswitchName
Name of external vSwitch on the Azure Stack HCI servers. Use same vSwitch that was used for SDN deployment. -k8sNodeIpPoolStart
-k8sNodeIpPoolEnd
IP start/end range of SDN virtual network. -ipAddressPrefix
Virtual network subnet in CIDR notation. -gateway
-dnsServers
Gateway and DNS server of the SDN virtual network. For more information about these parameters, see New-AksHciNetworkSetting.
In the same PowerShell window you used in Step 1, create a VIP pool to inform AKS of our IPs that can be used from our SDN Load Balancing logical network:
$VipPool = New-AksHciVipPoolSetting -name "publicvip" -vipPoolStart "10.127.132.16" -vipPoolEnd "10.127.132.23
Parameter Description -name
The "PublicVIP" logical network that you provided when configuring SDN load balancers. Within the cmdlet, this name must be lowercase. -vipPoolStart
IP start range of logical network used for public load balancer VIP pool. You must use an address range from the "PublicVIP" SDN logical network. -vipPoolEnd
IP end range of logical network used for public load balancer VIP pool. You must use an address range from the "PublicVIP" SDN logical network. In the same PowerShell window used in Step 2, create the AKS configuration for SDN by providing references to the targeted SDN networks, and supply the network settings ($vnet, $vipPool) that we previously defined:
Set-AksHciConfig –imageDir "C:\ClusterStorage\Volume1\ImageStore" –workingDir "C:\ClusterStorage\Volume1\WorkDir" –cloudConfigLocation "C:\ClusterStorage\Volume1\Config" –vnet $vnet –useNetworkController –NetworkControllerFqdnOrIpAddress "nc.contoso.com" –networkControllerLbSubnetRef "/logicalnetworks/PublicVIP/subnets/my_vip_subnet" –networkControllerLnetRef "/logicalnetworks/HNVPA" -vipPool $vipPool
The HNVPA logical network is used as the underlying provider for the AKS Arc virtual network.
If you use a static IP address assignment for your Azure Stack HCI cluster nodes, you must also provide the
CloudServiceCidr
parameter. This parameter is the IP address of the MOC cloud service, and must be in the same subnet as Azure Stack HCI cluster nodes. For more information, see Microsoft On-premises Cloud service.Parameter Description –imageDir
The path to where AKS Arc stores its VHD images. This path must be a shared storage path, or an SMB share. –workingDir
The path to where small files for the module are stored. This path must be a shared storage path, or an SMB share. -cloudConfigLocation
The path to the directory where cloud agent configuration is stored. This path must be a shared storage path, or an SMB share. -vnet
Name of AksHciNetworkSetting
variable created in the previous step-useNetworkController
Enable integration with SDN. -networkControllerFqdnOrIpAddress
Network controller FQDN. You can get the FQDN by executing Get-NetworkController
on the Network Controller VM and using theRestName
parameter.-networkControllerLbSubnetRef
Reference to the public VIP logical network subnet configured in the Network Controller. You can get this subnet by running the Get-NetworkControllerLogicalSubnet
cmdlet. When using this cmdlet, usePublicVIP
as theLogicalNetworkId
. TheVipPoolStart
andvipPoolEnd
parameters in theNew-AksHciVipPoolSetting
cmdlet must be part of the subnet referenced here.-networkControllerLnetRef
Normally, this value is "/logicalnetworks/HNVPA." -vipPool
VIP pool used as the front end IPs for load balancing. For more information about these parameters, see Set-AksHciConfig.
Sign in to Azure and configure registration settings
Follow the instructions here to configure registration settings.
Note
If you don't have owner permissions, it's recommended that you use an Azure service principal.
Install AKS
Once the AKS configuration completes, you are ready to install AKS on Azure Stack HCI.
Install-AksHci
Once the installation succeeds, a control plane VM (management cluster) is created, and its VmNIC is attached to your SDN network.
Collect logs from an SDN and AKS on HCI environment
With SDN and AKS on HCI, we gain isolation of the AKS nodes on virtual networks. Since they are isolated, we must import a new SDN AKS-HCI log collection script and run a modified command that uses the load balancer to retrieve logs from the nodes:
Install-Module -Name AksHciSdnLogCollector -Repository PSGallery
Get-AksHciLogsSdn
Feedback/issues
See the self-help resources here for SDN and here for AKS-HCI.
Next steps
Next, you can create workload clusters and deploy your applications. All AKS VM NICs in AKS enabled by Arc are seamlessly attached to the SDN virtual network that was provided during installation. The SDN Software load balancer is also used as the external load balancer for all Kubernetes services, and acts as the load balancer for the API server on Kubernetes control-plane(s).