SharePoint 2013: User Profile Incremental Synchronization timer job fails with Access Denied

Problem:

Consider the following scenario:
The User Profile Service (the web service, NOT the Sync service) is running on two servers in the farm: App1, App2.
In that case, the <UPA name>- User Profile Incremental Synchronization timer job (internal name: ProfileImportJob) can run on either server.

The User Profile Synchronization Service is running on App2.
When the User Profile Incremental Synchronization timer job runs on the same server that is running the Sync service (App2), it is successful.
When the import timer job runs on the other server (App1), it fails with Access Denied.

 

Cause:

The SharePoint Farm account has been removed from the local Administrators group on the App2 server (the Sync server) per security best practice:
https://technet.microsoft.com/en-us/library/ee721049(v=office.15).aspx#Phase1  

When the import timer job runs on a “remote” server, remote WMI calls are made to the Sync server to initiate the Sync.  These remote WMI calls fail with “Access is Denied” because the Farm account does not have permission to make remote WMI calls when it is not in the Administrators group.

Even after granting the “Remote Enable” permission to ROOT\MicrosoftIdentityIntegrationServer via wmimgmt.msc as specified in the above Technet article, remote WMI calls still fail with “Access is Denied”

 

Resolution:

If you are willing to add the Farm service account to the local Administrators group on the Sync server and keep it there, then just do that.  That makes life easy since you must have it in Administrators any time you want to start the Sync service.

However, if you really want to do the whole "least privilege" thing, then do this instead:

-- Add the Farm account (the one running the SharePoint Timer service) to the Performance Log Users local group on the Sync server.

Note: I recommend you add it to that group on all the servers in the farm in case you decide to move the Sync service to another server at some point in the future.

-- Set the “Remote Enable” permission for the Farm account on ROOT\MicrosoftIdentityIntegrationServer via wmimgmt.msc per https://technet.microsoft.com/en-us/library/ee721049(v=office.15).aspx#Phase1   

 

Troubleshooting and error details:

-- You can test remote WMI calls to FIM completely outside of SharePoint by running this PowerShell:

Get-WmiObject -Class MIIS_RunHistory -Namespace root/MicrosoftIdentityIntegrationServer -ComputerName SyncServer -Credential Contoso\SPFarmAccount

Note: Replace “SyncServer” with the name of the Synchronization server and replace “Contoso\SPFarmAccount” with your Farm account.
When you run this, you’ll be prompted for the password for the Farm account.  A successful run will display the run history for the Sync service.  However, if you have the WMI permission problem as detailed above, it will fail with:
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

-- These are the errors I get in my ULS logs when I remove my Farm account from the local Administrators group on the Sync server and the User Profile Incremental Synchronization timer job runs on a different server in the farm:

OWSTIMER.EXE (0x2D34) 0x32D0 SharePoint Foundation Timer 6398 Critical The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob (ID 050f7640-5ba8-49d0-9ae8-a723aba6aade) threw an exception. More information is included below.  Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

OWSTIMER.EXE (0x2D34) 0x32D0 SharePoint Foundation Timer 72ae Unexpected Exception stack trace:
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Management.ManagementScope.InitializeGuts(Object o)
at System.Management.ManagementScope.Initialize()
at System.Management.ManagementObjectSearcher.Initialize()
at System.Management.ManagementObjectSearcher.Get()
at Microsoft.Office.Server.UserProfiles.Synchronization.ManagementAgent.GetInstances(String machineName)
at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.CreateManagementAgents(UserProfileApplication upa)
at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.CreateSteps()
at Microsoft.Office.Server.UserProfiles.UserProfileImportJob.Execute()
at Microsoft.Office.Server.Administration.UserProfileApplicationJob.Execute(SPJobState jobState)
at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)

-- And this is thrown in the application event log:
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:
User:          Contoso\FarmAccount
Computer:      APP1
Description:
The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob (ID 050f7640-5ba8-49d0-9ae8-a723aba6aade) threw an exception. More information is included below.
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

-- Also, in Central Administration, timer job history shows that the “<UPA name> - User Profile Incremental Synchronization” timer job has a “Failed” status.
If you click on the “Failed” link, the Error Message is:
  “Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))”