SharePoint 2010: Unable to start UPA sync service ,stuck at “Starting”
As discussed in my previous blog, its been observed that starting User profile sync job is not an easy task at times. This post describes errors and the solution that I recently worked on. The issue was on a SharePoint 2010 farm which is migrated from 2007 farm.
Issue & Observations
Unable to start User Profile sync service from Central administration site ->system settings->Services on server. The service freezes at "starting" and never start. Below are the information from event viewer and verbose enabled ULS logs.
From ULS
Exception occured while connecting to WCF endpoint: Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application. at Microsoft.SharePoint.SPRoundRobinServiceLoadBalancer.BeginOperation() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.get_Channel() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)
UserProfileApplicationProxy.InitializePropertyCache: Microsoft.Office.Server.UserProfiles.UserProfileException: There are no addresses available for this application. ---> Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application. at Microsoft.SharePoint.SPRoundRobinServiceLoadBalancer.BeginOperation() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.get_Channel() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at
Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) --- End of inner exception stack trace --- at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.GetProfileProperties() at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.RefreshProperties(Guid applicationID) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValueNow(K key) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValue(K key, Boolean asynchronous) at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.InitializePropertyCache()
From Event viewer
Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Foundation
Date: 3/4/2011 12:30:23 PM
Event ID: 8077
Task Category: Topology
Level: Error
Keywords:
User: Domain\sp_farm
Computer: Computer FQDN
Description:
"There are no instances of the User Profile Service started on any server in this farm. Ensure that at least one instance is started on an application server in the farm using the Services on Server page in Central Administration."
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Microsoft-SharePoint Products-SharePoint Foundation" Guid="{6FB7E0CD-52E7-47DD-997A-241563931FC2}" />
<EventID>8077</EventID>
<Version>14</Version>
<Level>2</Level>
<Task>13</Task>
<Opcode>0</Opcode>
<Keywords>0x4000000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T18:30:23.718750000Z" />
<EventRecordID>18688</EventRecordID>
<Correlation />
<Execution ProcessID="3356" ThreadID="5948" />
<Channel>Application</Channel>
<Computer>Computer name</Computer>
<Security UserID="S-1-5-21-1834648779-3785927647-1620804944-5169" />
</System>
<EventData>
<Data Name="string0">User Profile Service</Data>
</EventData>
</Event>
-----------------------------------------------------------------------------------------------------------------------------------
Log Name: Application
Source: Microsoft Resource Management Service
Date: 3/4/2011 1:00:07 PM
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: Computer name
Description:
Service cannot be started. System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.
Login failed for user domain\sp_farm'.
at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Microsoft Resource Management Service" />
<EventID Qualifiers="0">0</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />
<EventRecordID>18718</EventRecordID>
<Channel>Application</Channel>
<Computer>sharepoint</Computer>
<Security />
</System>
<EventData>
<Data>Service cannot be started. System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.
Login failed for user 'Domain\sp_farm'.
at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)</Data>
</EventData>
</Event>
-----------------------------------------------------------------------------------------------------------------------------------
Log Name: Application
Source: Forefront Identity Manager
Date: 3/4/2011 1:00:07 PM
Event ID: 3
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sharepoint
Description:
.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.
Login failed for user 'domain\sp_farm'.
at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(SqlConnection connection)
at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(DataStore store)
at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction, IsolationLevel isolationLevel, DataStore dataStore)
at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction)
at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Forefront Identity Manager" />
<EventID Qualifiers="0">3</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />
<EventRecordID>18717</EventRecordID>
<Channel>Application</Channel>
<Computer>sharepoint</Computer>
<Security />
</System>
<EventData>
<Data>.Net SqlClient Data Provider: Sys
-----------------------------------------------------------------------------------------------------------------------------------
Log Name: Application
Source: Microsoft.ResourceManagement.ServiceHealthSource
Date: 3/4/2011 1:00:07 PM
Event ID: 2
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sharepoint
Description:
The Forefront Identity Manager Service could not bind to its endpoints. This failure prevents clients from communicating with the Web services.
A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint. Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.
Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints. Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
<EventID Qualifiers="0">2</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />
<EventRecordID>18716</EventRecordID>
<Channel>Application</Channel>
<Computer>sharepoint</Computer>
<Security />
</System>
<EventData>
<Data>The Forefront Identity Manager Service could not bind to its endpoints. This failure prevents clients from communicating with the Web services.
A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint. Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.
Ensure that no other processes have bound to that endpoint and that the service account has pe
-----------------------------------------------------------------------------------------------------------------------------------
Log Name: Application
Source: Microsoft.ResourceManagement.ServiceHealthSource
Date: 3/4/2011 1:00:07 PM
Event ID: 26
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sharepoint
Description:
The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.
Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution. If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.
Restart the Forefront Identity Manager Service.
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
<EventID Qualifiers="0">26</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />
<EventRecordID>18715</EventRecordID>
<Channel>Application</Channel>
<Computer>sharepoint</Computer>
<Security />
</System>
<EventData>
<Data>The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.
Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution. If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.
Restart the Forefront Identity Manager Service.</Data>
</EventData>
</Event>
-----------------------------------------------------------------------------------------------------------------------------------
Log Name: Application
Source: Microsoft.ResourceManagement.ServiceHealthSource
Date: 3/4/2011 1:00:06 PM
Event ID: 22
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sharepoint
Description:
The Forefront Identity Manager Service cannot connect to the SQL Database Server.
The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.
Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.
Event Xml:
<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >
<System>
<Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />
<EventID Qualifiers="0">22</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-03-04T19:00:06.000000000Z" />
<EventRecordID>18712</EventRecordID>
<Channel>Application</Channel>
<Computer>sharepoint</Computer>
<Security />
</System>
<EventData>
<Data>The Forefront Identity Manager Service cannot connect to the SQL Database Server.
The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.
Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, ver
Cause
In this case, the SSP was not migrated properly. Some of the database information was not properly registered in the SharePoint server registry.
Resolution
- Stop User profile service and User profile sync service on ALL SharePoint servers in the farm
- Clear timer config cache by following the "resolution" documented at https://support.microsoft.com/kb/939308
- Deleted the Pending timer job related to the UPA sync service provisioning using Central administration->Monitoring->check job status
- Confirmed that the security token service has only windows and anonymous enabled ( from IIS manager->SharePoint web services->Security token service->Authentication (under IIS))
- On the SharePoint server access the registry editor and confirm that the information related to FIM ( database servername , sync DB name etc ) are properly populated, if found wrong , back up the registry key and edit the values to reflect the right information.
HKLM\system\currentcontrolset001\services\FIMservice
HKLM\system\currentcontrolset002\services\FIMservice
HKLM\system\currentcontrolset\services\FIMservice
- Delete all FIM related certificates from certificate store ( for account and system) . Make sure that no FIM related certificate is listed in any folders when looked at certificate manager.
- From Central administration->Application management->Manage service application, Make sure the User profile service application and its proxy are started, If found stopped, its always recommended to recreate them taking necessary backups (mysites)
- Start User profile service on the machine where you would like to start the User profile sync service (use central administration console)
- Once the service is started, try starting the User profile sync service from the central administration site
Comments
Anonymous
March 10, 2011
Thanks for the blog post Manu. I followed all the steps, but still haven't had any success. One of the errors (the first one) I'm getting on ULS that's not listed in your error list is the one below. Any pointers as to what I should do?? Thanks!! User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator. at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties() at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs() at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext)Anonymous
March 11, 2011
Hi Mike, One quick test.. could you please see the action items in the blog ? blogs.technet.com/.../sharepoint-2010-unable-to-start-the-user-profile-synchronization-service.aspx thanks and regards, Manjesh