SharePoint 2010 – User Profile Replication Engine (UPRE)–Troubleshooting

 

Hello @all,

it’s a long time ago as a wrote my last blog post… since November 2010 I have changed my job role inside of Microsoft (from Support Escalation Engineer to a Premier Field Engineer). With the new role I had the chance to join into cool projects around the topic SharePoint and here come my next post.

In a global MySite deployment project for 70.000 users we have 3 farms: 1 in Europe, 1 in America and 1 in ASIA. All 3 Farms should host SharePoint MySite for each local user. This means: America users have their MySite in America farm, ASIA users have their MySite in ASIA farm and Europe users have their MySite in Europe. Based on performance, I believe in this design we reduce the IO traffic from one continent to another.

Caused of this design we have 3 User Profile Stores, one in each farm and for local search we need to replicate the user profiles with using the user profile replication engine (UPRE).
Comment: we could create a remote UPA connection and only one Farm host the global User Profile Store, but this create a lot of traffic for each MySite page rendering request. In case of Social networking functionality with outlook the amount of additional MySite request increase and we need much more hardware for one farm. Additionally we create a single point of failure.

So what we did: create 3 farms, use the replication:

1. install SharePoint Administration Toolkit to get the User Profile Replication Engine installed
2. as next step: we need to start the replication with using powershell cmdlets, first we need to collect all user profile properties which we want to replication and then we need to start the Full or incremental replication.

So we did this:

 get-spprofilepropertycollection –source https://Europe.demo.com | 
start-spprofileservicefullreplication -source https://Europe.demo.com 
-destination https://America.demo.com –EnableInstrumentation 
-MaxNumberOfThreads 15 -DoSocialReplication

An user profile has a size of around 1 MB and with a amount of 70.000 users a full replication mean we copy 70.000 MB = 70 GB data from Europe to America. Later we can easily start the increment replication.

 

With these action the whole magic of the replication is finished, but the learning of this project is not finished:

1. the replication was very slow and our calculation show us: a Full Replication of 70.000 user profiles took around: 20 days (measured over IIS w3c logs and UPRE logs). From replication engine we have 4 log file to troubleshoot the behavior:

2.  Inside the Error logs we see many time the following Event Entries and Exception:

 

Event Log Error:

Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---> System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---> System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

 

Detailed Error:

Log Name: Application

Source: Forefront Identity Manager

Date: 2/23/2011 11:04:04 AM

Event ID: 3

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: XXXXXXXXX

Description:

Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---> System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---> System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

   at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid& transactionIdentifier, OletxTransactionIsolationLevel& isolationLevel, ITransactionShim& transactionShim)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   --- End of inner exception stack trace ---

   at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)

   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)

   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)

   at System.Transactions.Transaction.Promote()

   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)

   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)

   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)

   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)

   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)

   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

   at System.Data.SqlClient.SqlConnection.Open()

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.OpenNewConnection(Boolean disallowEnlist)

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.Init(String connectionStr)

   --- End of inner exception stack trace ---

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHost(ResourceManagementWorkflowDefinition workflowDefinition)

   at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.StartWorkflowInstance(Guid workflowInstanceIdentifier, KeyValuePair`2[] additionalParameters)

Event Xml:

<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Forefront Identity Manager" />

    <EventID Qualifiers="0">3</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-02-23T10:04:04.000Z" />

    <EventRecordID>699449</EventRecordID>

    <Channel>Application</Channel>

    <Computer>XXXXXXXXXXXXXXX</Computer>

    <Security /> 
</System>

<EventData>

    <Data>Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---&gt; System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---&gt; System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

   at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid&amp; transactionIdentifier, OletxTransactionIsolationLevel&amp; isolationLevel, ITransactionShim&amp; transactionShim)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   --- End of inner exception stack trace ---

   at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)

   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)

   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)

   at System.Transactions.Transaction.Promote()

   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)

   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)

   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)

   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)

   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)

   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

   at System.Data.SqlClient.SqlConnection.Open()

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.OpenNewConnection(Boolean disallowEnlist)

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.Init(String connectionStr)

   --- End of inner exception stack trace ---

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHost(ResourceManagementWorkflowDefinition workflowDefinition)

   at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.StartWorkflowInstance(Guid workflowInstanceIdentifier, KeyValuePair`2[] additionalParameters)</Data>

  </EventData>

</Event>

 

Resolution:

As we can see from the callstack and from Event log the issue hit the component: Forefront Identity Manager. SharePoint 2010 User Profiles will be handled with FIM and the replication of user profile uses FIM too. The issue itself happens in the .net layer and below: System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions.

I’ve found an solution object in the Microsoft KB: https://support.microsoft.com/kb/817064/en-us. The KB was created for Windows Server 2003 but also works for our environment Windows Server 2008.

Action Plan: Enable Network DTC Access

https://technet.microsoft.com/en-us/library/cc753510(WS.10).aspx

Steps to enable network DTC access

1. Click Start, point to Control Panel, and then click Add or Remove Programs.

2. Click Add/Remove Windows Components.

3. Select Application Server, and then click Details.

4. Select Enable network DTC access, and then click OK.

5. Click Next.

6. Click Finish.

If you are running Windows Server 2003 Service Pack 1 (SP1), you must follow these additional steps:

1. Click Start, click Run, type comexp.msc, and then click OK to open Component Services.

2. Expand Component Services, expand Computers, right-click My Computer, and then click Properties.

3. On the MSDTC tab, click Secuity Configuration under Transaction Configuration, click to select the Network DTC Access check box under Secuity Settings, and then click to select the following check boxes under Transaction Manager Communication:

o Allow Inbound

o Allow Outbound

4. On Microsoft Cluster Server (MSCS) clusters, you cannot select Mutual Authentication Required. Therefore, click to select one of the following check boxes:

o Incoming Caller Authentication Required

o No Authentication Required

Note For more information about these options, click the following article number to view the article in the Microsoft Knowledge Base:

899191 (https://support.microsoft.com/kb/899191/ ) New functionality in the Distributed Transaction Coordinator service in Windows Server 2003 Service Pack 1 and in Windows XP Service Pack 2

5. Make sure that the Logon Account is set to NTAUTHORITY\NetworkService.

6. Click OK. A message box explains that the MS DTC Service will be stopped and restarted, and that all dependent services will also be stopped and restarted. Click Yes.

Note If this is a Majority Node Set (MNS) cluster, do not use the MNS resource as the storage device for MS DTC. MS DTC requires a storage resource such as a physical disk.

 

At the end of the story, enable DTC access improve the replication a lot, over 100 Profiles per minute. The Full replication finish successful in less than 12 h, so it’s even better than 20 days.

Now the replication runs much faster…

 

Now the second issue: By default SharePoint user profiles have the following setting configured:

- User Profile notification is enabled

Question: What happens during the first Full User Profile replication?

Answer: The replication operation is an update operation on each user profile from source user profile store to target user profile store. In case the source adds new colleagues into the target, then the related user receive an email from target environment, because a new colleague was added… In case of many users, we see a lot of emails…. So it’s very important to disable the 2nd check box: “Notify me when someone adds me as a colleague”, otherwise users will receive an email of colleague was added.

Based on many emails, I would ever suggest to disable all email notification before announcement.

How to get an overview of enabled/disabled User Profile Notifications? (using powershell)

 #Load the SharePoint snap-in
Add-PsSnapin Microsoft.SharePoint.PowerShell;

#Load the SharePoint assemblies
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server");
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.UserProfiles");

#Specify the MySite URL
$MySiteUrl = "https://mysite.demo.local";

#Get the server context for the profile manager
$site = Get-SPSite $MySiteUrl;
$ServerContext = Get-SPServiceContext $site;
$UPManager = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($ServerContext);

#Count variables
$ucount = 0;

write-host ""
write-host "Dump User Email Notification"
write-host "----------------------------"
write-host ""


$enumProfiles = $UPManager.GetEnumerator();
"Total User Profiles available:" + $UPManager.Count
$count=0;

# Email notification flag is handled via Property: "SPS-EmailOptin"
# SharePoint Notification emails is a integeter with 3 bit flags - SPS-EmailOptin (int)
# This field has 3 values one for each email type

# Bit flag:   0 =  enabled –> 0 (integer value)
# Bit flag: 111 = disabled –> 7 (integer value)

write-host ""
write-host ""

$disabled  = 0;
$enabled   = 0;
$undefined = 0;

foreach ($oUser in $enumProfiles)
{
    $count = $count + 1;
    $u = $oUser.Item("Accountname").ToString();
    $notification = $oUser["SPS-EmailOptin"].Value;

    Switch ($notification) 
    {
       "0"  { Write-host "enabled: "  $u; $enabled++;}
       "7"  { Write-host "disabled: " $u; $disabled++;}
    default { Write-host "undefined: "$u; $undefined++}
    }
}
Write-host "-----------------------------------------------"
Write-host "Total"
Write-host "-----"
Write-host "disabled User: "  $disabled
Write-host "enabled User: "   $enabled 
Write-host "undefined User: " $undefined 


#Dispose of site object
$site.Dispose();

I hope it helps.

Regards

Patrick

Comments

• Anonymous
  January 01, 2003
  Hi Brad, the incremental push is done as a running service. Take a look into Windows "Services.msc" You should see a Service called "Replication Engine". What i've seen is, the incremental replication runs in every 5 secounds (i found a thread.sleep() statement using Reflector/ILSpy) that's the default schedule plan for incremental replication. Each schedule runs the replication with: 1st: collect the chanageLog from MySite-Webapplication. 2nd:. try to push all changes from changeLog to the target UPA. In case 14 days of change log is not large enough you have the changethe size of the change log under Central administration / General setting of the mysite webapplication. regards patrick

• Anonymous
  January 01, 2003
  Hi RichM, based on logs i would say: [401.2 -> here you should see something in IIS http kernel log, or in SharePoint ULS log] POST /_vti_bin/userprofileservice.asmx - 80 - 10.7.2.223 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5448) - - mysite 401 2 5 258 752 0 -> is 10.7.2.223 your server IP or client IP from UPRE? [that's ok, the first call is annonymous, 401.1] POST /_vti_bin/userprofileservice.asmx - 80 - 10.7.2.223 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5448) - - mysite 401 1 2148074254 539 424 0 [that's the 2nd request with credential, and 200 is successful, -> no problem, UPRE run successful] POST /_vti_bin/userprofileservice.asmx - 80 <domain><username> <IP Address> HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5448) - - mysite 200 0 0 49618 1454 31 at the end, you should focus on: [401.2 -> here you should see something in IIS http kernel log, or in SharePoint ULS log] POST /_vti_bin/userprofileservice.asmx - 80 - 10.7.2.223 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5448) - - mysite 401 2 5 258 752 0 via: fiddler, IIS logs, IIS httpsys logs and ULS logs from sharepoint. in case you have a errror in sharepoint uls log then the callstack is very good to get more idear. regards Patrick

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  Hi Jonny, please check the notification setting on each user Profile on both farms.

• Anonymous
  January 01, 2003
  Hello RichM, http 401 is more then Unauthorized. As next step i would like see the substatus code from IIS W3C logs. What UPRE does is: connect to the source Url and call UserProfile webservice (.asmx) then UPRE connects to the target URL and call the next webservice. Check you IIS W3c Logs from Mysite webapplication. there you will find the 401.x substatus code. You manage the Webservice call permission on central admin / service applications / select you User Profile Application / press Manage Permission Your logon account starts cmd or powershell console. this account calls the UserProfile Webservice. on your IIS logs you will see the account too. regards Patrick

• Anonymous
  May 31, 2011
  Good post Patrick. Wish I have found it earlier and could have save some long hours. www.agileconcepts.com/.../Post.aspx Aamir Qureshi

• Anonymous
  September 06, 2011
  This is one of the best articles on this subject - I'm though experiencing some diffuclty synchronising some custom properties - here is the thread in the sp2010 forum... maybe somebody comes a cross a solution to this... social.msdn.microsoft.com/.../8481c647-fc1c-4c87-85d6-f33c0c2f14ba

• Anonymous
  December 19, 2011
  Hi Patrick, thanks for this post, with incremental replication i read on technet that it uses a change log, and you must do the incremental push within 14 days, to pick up changes, do you know if this limit is fixed, or can be extended etc? i.e. you may hit a scenario whereby your incremental stops working and only notice this 3 weeks later by which time you would have lost a weeks worth of changes? thanks Brad

• Anonymous
  January 29, 2012
  The comment has been removed

• Anonymous
  August 02, 2012
  Hi, great article. I disabled the email notifications and ran the full import but the emails were still sent!

• Anonymous
  August 23, 2012
  Hi Patrick, Great article, especially as there's not a lot of stuff out there about the UPRE. My issue is that I can get the UPRE to perform a Full Replication but in the Full_Err_Push log every user account in the source User Profile Service errors with: System.Net.WebException: The request failed with HTTP status 401: Unauthorized. If I then try to run a Full Replication again, the actual PowerShell window errors with: Start-SPProfileServiceFullReplication : The request failed with HTTP status 401 : Unauthorized. I've checked all the obvious stuff like permissions for the account running the service and the loopback check but try as I might I cannot get the UPRE to subsequently run.

• Anonymous
  August 23, 2012
  Hi Patrick, Thanks again for the speedy reply:

1. MySite IIS Logs - I have generated the error again and the following two lines get logged with 401 codes: 2012-08-23 14:28:28 ::1 GET / - 999 - ::1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729) 401 2 5 0 2012-08-23 14:28:28 ::1 GET / - 999 - ::1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729) 401 1 2148074254 0 Many thanks again

• Anonymous
  August 24, 2012
  Hi Patrick, Many thanks again for your help. At the moment I can answer 1, 2, 4 & 5

1. Both the source and destination MySite Web Applications are using Windows Authentication
2. They weren't, but they are now. It hasn't made a difference to the situation.
3. Yes - in both places. Just for extra info, both Farms are in the same domain and I'm using the same account on both farms. The account is a Farm Admin in both farms as well as having Full Control of the MySite Web Apps and the User Profile Service
4. see 1 For some reason today I'm not getting any IIS logs produced and at the moment I cannot figure out why. I've increased the log level on the soure MySite Web Application (ticked all the boxes in the select fields section) but alas no log file - I'll keep trying and as soon as I have one I'll post the details............... Still for the life of me cannot figure why it ran once (even though it didn't replicate any profiles) but will not run again when nothing has changed! Grrrrrr!

• Anonymous
  August 28, 2012
  The comment has been removed