AAD Connect, a dedicated resource forest, a custom connector, and a bunch of transform rules: a GalSync story (Part 1)

[アーティクル]
11/12/2018

A few years ago, I worked with one of my close consultant peers to build a GALSync-style solution for a big state government that was going through a divestiture from a single BPOS-D (yes, I am old) and a single managed hosted Exchange environment to multiple O365 multi-tenant instances. It was the equivalent of 100 agencies and 225,000 users going from two large hosted environments to 100 separate tenants.

Welcome to GALSync on a truly enterprise scale, since each instance was going to have about 250,000 contact objects for users and distribution groups. Throughout the course of our deployment, we were going to add nearly 25,000,000 objects to the global Azure AD footprint. Good thing Azure AD is pretty scalable. This design was the result of trying to overcome a significant number of challenges. We had considered a single large-scale MIM deployment, but with the number of connected data sources (over 100), we'd only be able to get a delta sync cycle out to every remote forest once every 2 or 3 days. Plus, it was really an unknown what kind of hardware we'd need to have 100+ connector spaces, each with hundreds of thousands of objects. At some point (like, probably 4 minutes after deployment), providing capacity and redundancy for the GALSync infrastructure would become a full-time job.

Our final solution was a centralized resource forest that would hold a contact object for every user in every organization in a tree structure. We decided to use AAD Connect's native Active Directory connector to be the conduit between the remote agency's local Active Directory forest and the resource forest. Agencies would export their users as contacts to OU=Agency <x>,OU=Shared GAL,DC=resource forest,DC=com, and then would import from OU=Shared GAL,DC=resource forest,DC=com. Due to deployment timeline issues, our solution brought along with it an enormous amount of challenges and baggage. I've revisited it, modified it, and stripped it down to provide a simple (yet effective) sync solution for organizations that need a shared GAL but don't have Microsoft Forefront Identity Manager or Microsoft Identity Manager deployed (nor do they want to).

To walk through how this is going to look, I'm going to build a brand new lab in Azure and spin up two new Office 365 subscriptions. The solution involves building a central resource forest to hold contacts, and then connecting each of the forests via the Active Directory connector to import and export contacts to N's Azure AD Connect's connector space, and then utilizing default rules to export them to the respective tenants.

Lost yet? Good.

Also, please don't call Premier asking for support on this. They will hunt me down and give me a stern talking to. As any custom solution is, this is also unsupported. While it does work and only utilizes built-in connectors, AAD Connect (like Office 365) is an evergreen product, so there is potential that the steps outlined here may someday cease to work or functionality may be deprecated. This works with the build version of AAD Connect current as of this writing: 1.1.882.0 (September 8, 2018).

Without further ado ... Onward!

[toc]

Overview

The lab will consist of 3 virtual machines running Windows Server 2016 (the version doesn't matter, as long as it's new enough for AAD Connect to install successfully). Each of the 3 servers will be configured as its own forest (something like GalSyncTenantA, GalSyncTenantB, and GalSyncShared, since I'm well-known for my creativity). Because we at Microsoft like to make acronyms out of everything, I'm calling them GSTA, GSTB, and GSS. GSTA and GSTB will each be connected to their own respective tenants, synchronizing their own identities. GalSyncShared (GSS) will be the central forest that we use to export and import the shared contacts. While this solution only has three forests (2 account forests and one resource forest), it can be expanded by repeating the steps for any of the account forests.

When we get to the sync configuration in part 2 of this series, we'll be leveraging the native Active Directory Connector, which requires SRV lookup capability to the resource forest.

And, since I'm not one to miss an opportunity for shameless promotion, I'm going to make use of a handful of tools that I've developed: CreateLabUsers, AAD Permissions, and AAD Network Test.

Prepare Office 365 Tenants

First thing's first. I need some tenants. So, I head over to https://www.office365.com and sign up for two E3 trial tenants. My creativity in naming them carried on to the tenant names--galsynctenanta.onmicrosoft.com and galsynctenantb.onmicrosoft.com.

Prepare Azure AD Virtual Infrastructure

If you don't have a lot of experiencing deploying virtual infrastructure in Azure, I'm going to go through the steps I used to create this environment. Specifically, I'm going to create:

Virtual Networks - One of the requirements is that all three of the environments be able to talk to each other. In the real world, you may have separate infrastructures separated by VPNs and physical networking. For purposes of the lab, all three of these machines will be in the different networks, since that's how you'll probably encounter it. If you go to do this for real, you'll have to ensure the each of the account forests (GalSyncTenantA and GalSyncTenantB) have line of sight and connectivity to the resource forest (GalSyncShared). We'll go over the specific networking requirements later.
Network Security Groups - Think of Network Security Groups as firewall rules or router access control lists in the cloud. NSGs are sets of rules that determine what traffic is allowed to move between networks and hosts.
Virtual Machines - In order to meet the requirements for installing AAD Connect, I'll need a machine that meets the minimum specifications. I'll be preparing the environments by extending them with the Exchange 2016 schema so they host all of the attributes that we're going to need. Then, I'll be stocking them with about 10,000 users each.

Create virtual networks

I want all of the virtual machines in my lab to be able to talk to each other. I'm going to create three virtual networks (one representing each forest). My virtual network settings:

GalSyncTenantA 10.0.0.0/24
GalSyncTenantB 10.0.1.0/24
GalSyncShared 10.0.2.0/24

Log into https://portal.azure.com. If you don't already have any subscriptions, you'll need to acquire one of those. We do offer some trial subscriptions, so if you want to follow along with me, you'll need some way to do this. You can also do this in your on-premises infrastructure or (gasp) with another provider.
Select +Create a resource, start typing virtual and select Virtual network from the list.
Ensure Resource manager is selected as the deployment model (since this is 2018) and click Create.
Select the options for your first virtual network and click Create. I'm going to name them to match the forests and tenants that we'll be using, so hopefully it will be obvious which ones we're acting against in the later parts of this lab. I created a new resource group, because I want to be able to identify all of the resources associated with this project. Note: You can create a virtual network and then divide it logically into smaller subnets--for example, you could create a network of 10.0.0.0/24, and then create subnets of 10.0.0.0/25 (10.0.0.0-10.0.0.127) and 10.0.0.128/25 (10.0.0.128-10.0.0.255). In order to route between subnets, you need to create a standard subnet and a Gateway subnet inside the same network. As a bonus, they can't overlap. To keep my math simple, I'm going to create two subnets per network: a standard subnet to be used for "devices" at 10.0.x.0/25, and a Gateway subnet configured in the same virtual network at 10.0.x.128/25.
Lather, rinse, and repeat steps 2-4 for your other two virtual networks.
After you've created your virtual networks, go check them out! Click All services, type Virtual Networks and then click the Virtual Networks link (not the Virtual Networks (Classic) link).
You should be greeted with something similar to this (a resource group and three virtual networks associated with it):
Click on a virtual network, and then select Subnets. As I described earlier, I created a "normal" subnet in the 10.0.x.0/25 space, and then a "gateway" subnet in the 10.0.x.128/25 space.
Good? Sweet! On to Network Security Groups!

Create Network Security Groups

As mentioned earlier, we need to ensure connectivity from each of the account forests to the resource forest. To help achieve this, we're going to define some networking to allow us to be able to reach the endpoints. We're going to create a NSG to allow GSTA and GSTB to communicate with GSS on the following ports:

53 - DNS
135 - RPC PortMapper
389 - LDAP
445 - SMB
636 - LDAP over SSL (optional, you can configure AAD Connect to connect securely)
3268 - Global Catalog
3389 - RDP (optional, but during the configuration, I'd like to be able to reach the DC in GSS from either of the account forests)

We're going to create a network security group for each virtual network. When you create a new network security group, it is automatically populated with the following rules:

Default security rules

Azure creates the following default rules in each network security group that you create:

Inbound

AllowVNetInBound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65000	VirtualNetwork	0-65535	VirtualNetwork	0-65535	All	Allow

AllowAzureLoadBalancerInBound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65001	AzureLoadBalancer	0-65535	0.0.0.0/0	0-65535	All	Allow

DenyAllInbound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65500	0.0.0.0/0	0-65535	0.0.0.0/0	0-65535	All	Deny

Outbound

AllowVnetOutBound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65000	VirtualNetwork	0-65535	VirtualNetwork	0-65535	All	Allow

AllowInternetOutBound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65001	0.0.0.0/0	0-65535	Internet	0-65535	All	Allow

DenyAllOutBound

Priority	Source	Source ports	Destination	Destination ports	Protocol	Access
65500	0.0.0.0/0	0-65535	0.0.0.0/0	0-65535	All	Deny

What that basically amounts to:

Allow inbound traffic from all resources in the same virtual network.
Block all inbound traffic from anywhere outside of the virtual network.
Allow all outbound traffic.

Based on our network requirements, we're going to create three network security groups and then configure the network security group associated with GalSync Shared to allow inbound traffic on the necessary ports.

From the Azure Portal (https://portal.azure.com), select +Create a resource, start typing Network Security Group, and then select it from the list.
Ensure Resource Manager is selected as the deployment type, and click Create.
Enter a name of the security group, select a resource group, and click Create.
Repeat steps 2 and 3, creating network security groups for the additional virtual networks.
After you've created them, click All services, type network security groups to filter the list, and select Network security groups (again, not the classic one).
You should have three brand-spanking new Network Security Groups.

Configure Network Security Groups

Now that we've got NSG objects, we need to configure the rules that will govern our virtual machines when they are created and inserted into the networks.

From the Network Security Groups blade, click on the NSG representing GalSync Shared (resource forest).
Click Subnets, and then click the Associate button.
Click Choose a virtual network and then select the GalSync Shared network.
Select Choose a subnet, and then select the subnet associated with the virtual network.
Click OK.
Click Inbound security rules and then click Add.
For my configuration, I just wanted to create a single rule to capture all of the GALSync traffic. Using an Advanced rule, you can configure multiple sources and ports. Ensure the Advanced input is toggled (click the wrench icon to switch between Basic and Advanced). Select IP Addresses under source, enter the CIDR blocks for GalSyncTenantA and GalSyncTenantB VNets (in my case, 10.0.0.0/24 and 10.0.1.0/24 for the internal addresses and 137.117.58.26 and 168.62.181.187 as the public IPs), set the destination as Virtual network (if you were doing this for real, you might bind it to a specific host. We haven't created any hosts yet, and we're only putting a single host in the network, so this will work fine). Under Destination port ranges, enter 53,135,389,445,3268,3389. Select Any under Protocol, and give the rule a name. Enter a description if you feel enterprising. Click Add when finished.
Click Network Security Groups in the breadcrumb trail to take you back to the screen in step 1, and then associate a virtual network and subnet with each of the remaining network security groups (steps 2-5).
For all three network security groups, configure an inbound rule for port 3389 from your workstation (which will allow you to log in via an RDP client). For example, I created an inbound rule in each network security group by selecting the following settings:

Create Virtual Network Gateways

Routing between VNets is, unfortunately, not automatic. In order to make it work, you need to create a routing topology using Gateways. We'll have to do this a few times.

Create the Virtual Gateways

From Virtual Networks, click the Resource Forest virtual network.
If you haven't already created the Gateway Subnets, click Subnets, and then click + Gateway subnet.
Enter the subnet you want to use for the gateway subnet. The Gateway Subnet must be inside the address space for the network. In this case, I've divided my /24 virtual network into two /25 subnets and used the first half for hosts and the second half for the gateway subnet.
Click Add.
Click +Create a resource, and start typing Virtual Network Gateway. Select Virtual Network Gateway from the list.
Click Create.
Enter a name for the gateway, select VPN for the gateway type, select Route-based for the VPN type, select VpnGw1 for the SKU type, and select the resource forest virtual network. Enter a name for the Gateway IP (I just copied the name I stuck in the top and appended -IP). Click Create when finished.
Repeat steps 5-8 for the account forest networks. Be prepared to wait a while. Each VPN Gateway can take up to 45 minutes to provision.

Create Virtual Network Gateway Connections

From this point, we'll be creating Vnet-to-Vnet gateways. The connections will be created on both sides (just like traditional VPN gateways).

Once the network gateways are created, click All services and navigate to Virtual Network Gateways.
Select a Gateway from the list. In this example, I'm going to start with the resource forest gateway GalSyncSharedVNG, and connect it to GalSyncTenantAVNG.
Select Connections, and then click Add.
Enter a connection name, and then select the virtual network gateways that will be included. Select VNet-to-VNet as the connection type. Enter a pre-shared key (you will need the same pre-shared key when you create the VNet on the other side. You can use a PowerShell one-liner such as this to generate a PSK that will be quite difficult to guess: [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes((Get-Random).ToString() + (Get-Random).ToString() + (Get-Random).ToString()))
Click OK when finished.
Click All services, start typing Virtual Network Gateway, and select Virtual network gateways. Select the account forest virtual network gateway used in step 4, and then create a connection using the resource forest gateway. Use the same pre-shared key used in step 4 (both sides of the VNet-to-VNet configuration must use the same key).
Verify connections show as Connected for each virtual network gateway.

Create Domain Controllers in Azure

During this process, we're going to deploy a single virtual machine to each of the virtual networks. Once we've deployed the VMs, we'll assign static IP addresses and then start the DC promotion process.

Provision virtual machines

From the navigation blade of the Azure portal, All services, start typing virtual machines, and select Virtual machines from the filtered list.
Click Add.
Select a subscription, resource group, and name for your machine. I've selected my existing GalSync resource group, specified a machine name (GSS-DC), selected a Windows Server 2016 image, and the VM size B2ms (the virtual machine sizes you have available will depend upon your subscription and region), and specified an administrator credential.
Click Next : Disks > to go to the next screen.
Select a disk configuration. Since this is just a lab environment, I'm going to select standard hard drives.
Click Next : Networking > to go to the networking configuration.
Select the appropriate virtual network, subnet, and a new public IP. Click Advanced radio button and select the appropriate network security group.
Click Next : Management > to proceed to the next screen.
Select any additional options (I didn't click any here) and click Next : Guest config to continue.
Add any extensions (I also didn't select any) and click Next : Tags > to proceed to the next screen.
Add a tag if you like to further group the resources. Click Next : Review + create > to proceed to the last screen.
Ensure your machine passes validation and click Create.
Repeat steps 1-12 for the other two virtual machines for the other forests (GalSyncTenantA and GalSyncTenantB) and assign them to the appropriate virtual networks and network security groups.
Click Virtual Machines in the navigation blade and monitor the progress.

Assign Static IPs to Domain Controllers

While it's totally possible to use dynamic IP addresses for your domain controllers, it's definitely not recommended (just like it's not recommended on-premises). We'll convert the existing dynamic IPs to static IPs, and then we can start the actual installations!

Click All services and start typing network interfaces. Click Network interfaces to open the blade.
Click the first network interface.
Click IP configurations and then click the IP configuration (likely named ipconfig1).
Under Assignment, select Static. Select an IP address (it's populated with the current address, so you can just leave it). Click the Save icon to save changes.
Click the X to close this configuration blade.
Select DNS servers, select Custom, and then add two DNS server entries (primary as this server's static address you assigned in step 4, and a secondary of either an Azure recursive DNS server, a DNS forwarder already configured in your network, or a public recursive DNS server). Click the Save icon. Note that your VM will restart.
Repeat steps 1-6 for the remaining servers in the lab.

Install AD DS Role

Using the guide at docs.microsoft.com, install AD DS. Since I want this to be done sooner rather than later, I'm just going to use PowerShell in an elevated prompt on each server.

Install required Windows Features.

 Install-WindowsFeature -name AD-Domain-Services,Dns -IncludeManagementTools

At the end of the installation, you should have both the AD DS and Dns Server binaries installed. If you're installing in Azure, you'll probably see this warning, which you can safely ignore (since the virtual machine is configured with a static IP in the VM configuration).
Add a credential.
```
 $cred = Get-Credential
```

Configure this server as the first domain controller in a new forest.

 Install-ADDSForest -DomainName <DNS domain name> -SafeModeAdministratorPassword $cred.Password -DomainMode Win2012R2 -DomainNetbiosName <NetBIOS domain name> -ForestMode Win2012R2 -InstallDns

The severs will reboot after installation.

Extend schema for Exchange 2016

In order to ensure the environment has all of the applicable attributes necessary during AAD Connect setup, we need to extend the schema using the Exchange 2016 setup.

On each of the following servers, perform each of the following actions:

Download the Exchange 2016 media (https://www.microsoft.com/en-us/download/confirmation.aspx?id=49161). You may need to disable the Internet Explorer Enhanced Security Configuration.

 Write-Host -ForegroundColor Green "Closing all Internet Explorer processes."
If (Get-Process -Name iexplore -ErrorAction SilentlyContinue) { Get-Process -Name iexplore | % { Stop-Process $_ -Force } }
Write-Host -ForegroundColor Green "Disabling IE Enhanced Security for Administrators."
$AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 0

Extract the download to a temp folder.
Launch an elevated command prompt (not a PowerShell prompt--things may not work correctly) and change to the directory containing the extracted Exchange 2016 setup files.

From the command prompt, run:

 setup /PrepareSchema /IAcceptExchangeServerLicenseTerms

Restart server.

Prepare Active Directory

Now that the base infrastructure is in place, we're going to create some users in both environments to work with.

On the account forest domain controllers (GSTA-DC and GSTB-DC, in my lab), I'm going to use the Create-LabUsers tool to create about 5,000 users each.

On GSTA-DC, download Create-LabUsers.
Unblock it if necessary (right-click | Properties | Unblock).

Launch an elevated PowerShell session, change to the directory where Create-LabUsers.ps1 was downloaded and then run the following command:

 .\Create-LabUsers.ps1 -Domain <verified domain in Office 365> -Company "GalSync Tenant A" -Count 5000 -Password Passw0rd123 -AddUpnSuffix -UpnSuffix <verified domain in Office 365>

In my case, since I'm not adding any new verified domains (since it's for a lab), so I'm going to tell the script to use my tenant domain as the UPN suffix for my users (and add it as a UPN suffix to the forest).

 .\Create-LabUsers.ps1 -Domain galsynctenanta.onmicrosoft.com -Company "GalSync Tenant A" -Count 5000 -Password Passw0rd123 -AddUpnSuffix -UpnSuffix galsynctenanta.onmicrosoft.com

Since we didn't install Exchange, we don't have all of the mail-enabled attributes on those user. You'll have to update and provision the mailNickname attribute (because we need it later):
```
 Get-ADUser -Filter * | % { $alias = $_.sAMAccountName; Set-ADUser $_ -Replace @{mailnickname = $alias } }
```
Close the PowerShell window.
Repeat on GSTB-DC.

Configure Directory Synchronization

This is the last set of steps to perform--making sure that our source tenant environments can communicate with Office 365, configure permissions for write-back, and proceed with the AAD Connect installation.

AAD Connect Prerequisite Test

On each of the account forest DCs (GSTA-DC and GSTB-DC) domain controllers, perform the following:

Download the AAD Connect Network Tool.
Unblock it if necessary (right-click | Properties | Unblock).
Launch an elevated command and change directories to where the script has been downloaded.
Run with the -InstallModules switch to install the MSOnline module and run all tests. Enter your Azure AD Global Admin credential when prompted.
```
 .\AADConnect-CommunicationsTest.ps1 -InstallModules
```
Verify that the checks complete successfully.

Configure delegated account for AAD Connect

Most of my customers require a least-privilege deployment for AAD Connect, meaning that they don't want to configure a domain or enterprise admin account as a service account. AAD Connect fully supports that installation model. More information can be found on the support page: /en-us/azure/active-directory/hybrid/reference-connect-accounts-permissions. You can use the permissions tool I wrote at https://aka.ms/aadpermissions to configure the specific delegate permissions.

Install AAD Connect

Once that the prerequisite testing has completed successfully, it's time to install AAD Connect. On both GTSA-DC and GTSB-DC, we're going to use the Express Setup to connect to our tenants.

Download AAD Connect (https://aka.ms/aadconnect).
Launch AAD Connect Setup and choose Express Setup.
Select Exchange Hybrid to enable hybrid-write-back for your user objects.
Follow the bouncing ball until setup is complete.

Whew! Now we're ready to start the hard stuff! On to part 2!

次の方法で共有