Remediate Multi-tenancy Port Collision -or- How I put Multiple Container Websites is a single VM / Scaleset
Many apps (legacy and modern) provide service interfaces on TCP ports. Many of these service interfaces expect to reside on a well-known port, for example HTTP (80) and HTTPS(443). The current implementation of Windows Containers employs a port mapping mechanism to expose ports running on the container to access from outside the container, and any other access from the network to the container is blocked. All access to the services on the container must be made via a mapped port on the container host. Obviously port numbers cannot be shared by multiple services, so each service must map to a unique port number. This is usually a blocker, especially with HTTP clients which usually expect to operate on the well-known ports. We will demonstrate how to use the Azure Load Balancer to resolve this issue.
Step by Step
In this step-by-step we will:
- create two containers running IIS on a container host in Azure
- configure an Azure Load Balancer to reference the web sites on the containers
Create two containers running on a single Host.
- start a Powershell admin session on the container host
- Create two containers
docker run -it --name Site1 -p 81:80 -v c:\shared:c:\shared microsoft/dotnet-framework:4.6.2docker run -it --name site2 -p 82:80 -v c:\shared:c:\shared microsoft/dotnet-framework:4.6.2
In this example we will assume the following IP Addresses
Container Host IP: 10.138.137.183
Internal IP: Site1 : 172.28.103.94
Internal IP: Site2 : 172.28.104.223
Load Balancer Configuration (Option 1: Azure Portal)
Front End IP Pool
Each Tenant should have a Virtual IP created, and DNS configured to resolve to that Front IP. In the example below we have site1 and site2 setup with their individual Public IP's assigned from the Virtual network to which the Load balancer is connected.
Backend Pools
In the below example we have a single VM in the backend pool. To achieve HA and Load balancing, we would create multiple VMs in the same Availability set. The backend pool would be set to this Availability set. If the same container is placed on the hosts in the Availability set. Requests will be loaded balanced across those machines.
Health Probes
Each site on the container host has a container listening on the hosts port. We need to setup a probe for those ports. In the example below, we have a container running on Container IP: port 80, mapped to the Host IP: Ports 81 & 82, and the host has a site on port 80. The health probes will mark those Hosts as "up" if they are responding on these ports.
Load balancing Rules
Each Site will need a Load balancing rule to map the Virtual IP:port tuple to the HostIP: Port. Inside the container host the Host IP: port is mapped to the Container IP: port.
In the example below we also route the Virtual IP:80 to the Host:80. We then route site 1 virtual IP and port 80 to the host: port 81, and site 2 virtual IP and port 80 to the host: port 82.
Load Balancer Configuration (Option 2: Azure Powershell)
Script:
/en-us/azure/load-balancer/load-balancer-get-started-ilb-arm-ps
---Create Script starts---
Connect to Subscription: Login-AzureRmAccount Select-AzureRmSubscription -Subscriptionid "GUID of subscription" New-AzureRmResourceGroup -Name NRP-RG -location "West US"
Create Virtual IP and Backend Pool: $frontendIP1 = New-AzureRmLoadBalancerFrontendIpConfig -Name LB-Frontend -PrivateIpAddress 10.0.2.5 -SubnetId $vnet.subnets[0].Id $frontendIP2 = New-AzureRmLoadBalancerFrontendIpConfig -Name LB-Frontend -PrivateIpAddress 10.0.2.5 -SubnetId $vnet.subnets[0].Id $beaddresspool= New-AzureRmLoadBalancerBackendAddressPoolConfig -Name "BackendPool"
For RDP access to hosts: $inboundNATRule1= New-AzureRmLoadBalancerInboundNatRuleConfig -Name "RDP1" -FrontendIpConfiguration $frontendIP -Protocol TCP -FrontendPort 3441 -BackendPort 3389 $inboundNATRule2= New-AzureRmLoadBalancerInboundNatRuleConfig -Name "RDP2" -FrontendIpConfiguration $frontendIP -Protocol TCP -FrontendPort 3442 -BackendPort 3389
Create Health Probes and IB rules: $healthProbe0 = New-AzureRmLoadBalancerProbeConfig -Name "Port80" -Protocol tcp -Port 80 -IntervalInSeconds 15 -ProbeCount 2 $healthProbe1 = New-AzureRmLoadBalancerProbeConfig -Name "Port81" -Protocol tcp -Port 81 -IntervalInSeconds 15 -ProbeCount 2 $healthProbe2 = New-AzureRmLoadBalancerProbeConfig -Name "Port82" -Protocol tcp -Port 82 -IntervalInSeconds 15 -ProbeCount 2 $lbrule = New-AzureRmLoadBalancerRuleConfig -Name "Host" -FrontendIpConfiguration $frontendIP1 -BackendAddressPool $beAddressPool -Probe $healthProbe1 -Protocol Tcp -FrontendPort 8080 -BackendPort 80 $lbrule = New-AzureRmLoadBalancerRuleConfig -Name "Site1" -FrontendIpConfiguration $frontendIP1 -BackendAddressPool $beAddressPool -Probe $healthProbe1 -Protocol Tcp -FrontendPort 80 -BackendPort 81 $lbrule = New-AzureRmLoadBalancerRuleConfig -Name "Site2" -FrontendIpConfiguration $frontendIP2 -BackendAddressPool $beAddressPool -Probe $healthProbe1 -Protocol Tcp -FrontendPort 80 -BackendPort 82
---Create Script ends---
---Add Script starts---
Connect to Subscription: Login-AzureRmAccount Select-AzureRmSubscription -Subscriptionid "GUID of subscription"
Create Virtual IP and Health Probe: $Subnet = Get-AzureRmVirtualNetwork -Name [vnet name] -ResourceGroupName ERNetwork | Get-AzureRmVirtualNetworkSubnetConfig –Name "subnet-1" Get-AzureRmLoadBalancer -Name [ilbname] -ResourceGroupName [resource group name] | Add-AzureRmLoadBalancerFrontendIpConfig -Name "Site3" -Subnet $Subnet | Set-AzureRmLoadBalancer $ilb = Get-AzureRmLoadBalancer -Name [ilb name] -ResourceGroupName [resource group name] $beaddresspool= get-AzureRmLoadBalancerBackendAddressPoolConfig -Name "BackendPool" -LoadBalancer $ilb add-AzureRmLoadBalancerProbeConfig -Name "Port83" -Protocol tcp -Port 83 -IntervalInSeconds 15 -ProbeCount 2 -LoadBalancer $ilb $ilb | Set-AzureRmLoadBalancer
Create the LB rule: $healthProbe3 = Get-AzureRmLoadBalancerProbeConfig -Name "Port83" -LoadBalancer $ilb $frontIP3 = Get-AzureRmLoadBalancerFrontendIpConfig -Name site3 -LoadBalancer $ilb $ilb | add-AzureRmLoadBalancerRuleConfig -Name "site3" -FrontendIPConfiguration $frontIP3 -Protocol "Tcp" -FrontendPort 80 -BackendPort 83 -BackendAddressPool $beaddresspool -Probe $healthProbe3 $ilb | Set-AzureRmLoadBalancer
---Add Script ends---
Test Site Access
All three sites should be available from the network on port 80
Host Site: https://10.138.137.183:80
Container Site1: https://10.138.137.166:80
Container Site2: https://10.138.137.170:80