Troubleshooting MSDTC Permission Issues When a Distributed Transaction Starts
This article is intended to address some common permission issues during the initialization process of a distributed transaction especially when a transaction begins on Microsoft Cluster Server (MSCS) clusters.
Symptoms
Microsoft Distributed Transaction Coordinator (MSDTC) is a transaction manager which enlists multiple resource managers (RMs) in one distributed transaction and coordinates the transaction across all the servers that participate in the transaction. To start a distributed transaction, a client application directly or indirectly makes the request to the MSDTC transaction manager. In the process the application will need to access the Service Control Manager (SCM) and other services such as the Cluster Service (ClusSvc) and MSDTC. If the security context of the client application doesn't have the required access permissions, the initialization process may fail. This is a common issue when an application runs under an account that is not part of the local Administrators group.
For more information on MSDTC, you can read What is MSDTC and why do I need to care about it?
You may see a callstack that resembles the following when an application fails at starting a transaction:
System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)
at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
at System.Data.SqlClient.SqlInternalConnectionTds.Activate(Transaction transaction)
at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
You may see an error message that resembles the following when a transactional SSIS package fails to execute in SQL Server:
Description: The SSIS Runtime has failed to start the distributed transaction due to error 0x80070005 "Access is denied.". The DTC transaction failed to start. This could occur because the MSDTC Service is not running.
Error: 2008-11-27 11:58:24.15
Code: 0x00000004
Source: Is Error Re-tryable?
Description: The Script returned a failure result.
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 11:57:22 AM
Finished: 11:58:24 AM
Elapsed: 61.781 seconds.
The package execution failed. The step failed.
Troubleshooting Steps
You can follow these troubleshooting steps to resolve the permission errors.
1. Grant the client application identity the Full Control permission to the cluster server
The user account associated with the client application must be a member of the local Administrators group, or have the Full Control to the cluster server. Otherwise the RPC call made from the client application to the cluster server will fail and an error message that resembles the following will be logged in the application event log:
Event Type: Error
Event Source: MSDTC Client
Event Category: MSDTC Proxy
Event ID: 4376
Date: 11/12/2008
Time: 1:25:31 PM
User: N/A
Computer: ComputerName
Description: The application could not connect to MSDTC because of insufficient permissions. Please make sure that the identity under which the application is running has permission to access the cluster. Please refer to MSCS documentation on how to grant permissions. Error Specifics: d:\srvrtm\com\complus\dtc\dtc\msdtcprx\src\dtcinit.cpp:652, Pid: 4544
For more information for granting the permission, see You cannot start transactions from a COM+ component on a clustered SQL Server server.
2. Grant the user access rights to Service Control Manager on the cluster server
In addition to the full control cluster permissions, the user account must also have the GENERIC_READ access right to the Service Control Manager (SCM) on the cluster, else it cannot begin the distributed transaction.
If the user doesn’t have the access right to the SCM, the following error is logged in the security event log. Note this error is only logged if “Audit object access” is enabled in the Local Policies. For more info for enabling “Audit object access”, see How to enable and apply security auditing in Windows 2000
Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 11/24/2008
Time: 12:46:33 PM
User: Domain\User
Computer: ComputerName
Description:
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
Handle ID: -
Operation ID: {0,55865185}
Process ID: 1340
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: ComputerName$
Primary Domain: Domain
Primary Logon ID: (0x0,0x3E7)
Client User Name: User
Client Domain: Domain
Client Logon ID: (0x0,0x353882A)
Accesses: READ_CONTROL
Connect to service controller
Enumerate services
Query service database lock state
Privileges: -
Restricted Sid Count: 0
Access Mask: 0x20015
To display the discretionary access control list (DACL) on the Service Control Manager (SCM), run the following SC command at a command prompt:
sc sdshow SCMANAGER
Here is a sample output:
D:(A;;CC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)
In the output above, there is (A;;CC;;;AU). The first ‘A” means allow access. The ‘CC” means the SC_MANAGER_CONNECT right and the ‘AU’ represents the “Authenticated Users” group. This means the “Authenticated Users” have the SC_MANAGER_CONNECT access right to SCM.
To add the GENERIC_READ (GR) access right for “Authenticated Users” to the SCM to the existing DACL, run the following SC command at a command prompt:
sc sdset SCMANAGER D:(A;;CCGR;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)
Run “sc sdshow SCMANAGER” again to display the new DACL on the SCM:
D:(A;;CCLCRPRC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)
You can see in the new output that ‘GR’ is replaced with ‘LCRPRC’. This is because:
GENERIC_READ for SCMANAGER is:
- READ_CONTROL (RC)
- SC_MANAGER_ENUMERATE_SERVICE or Enumerate services (LC)
- SC_MANAGER_QUERY_LOCK_STATUS or Query service database lock state (RP)
This will ensure the user has the GENERIC_READ access right to the SCM.
3. Grant the user access rights to the cluster service on the cluster
The user account must also have the GENERIC_READ access right to the cluster service (ClusSvc) on the cluster. If the user doesn’t have the access right to ClusSvc, an error message that resembles the following will be logged in the security event log. Note this error is only logged if “Audit object access” is enabled in the Local Policies.
Event Type: Failure Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 2/24/2009
Time: 5:28:31 PM
User: Domain\User
Computer: ComputerName
Description:
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: ClusSvc
Handle ID: -
Operation ID: {4,1888529168}
Process ID: 1208
Image File Name: C:\WINDOWS\system32\services.exe
Primary User Name: ComputerName$
Primary Domain: Domain
Primary Logon ID: (0x0,0x3E7)
Client User Name: User
Client Domain: Domain
Client Logon ID: (0x4,0x7085A6C0)
Accesses: READ_CONTROL
Query service configuration information
Query status of service
Enumerate dependencies of service
Query information from service
Privileges: -
Restricted Sid Count: 0
Access Mask: 0x2008D
To display the DACL on Cluster Service (ClusSvc), run the following SC command:
sc sdshow clussvc
Here is a sample output:
D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CR;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
To add the GENERIC_READ (GR) access right for “Authenticated Users” to ClusSvc to the existing DACL, run the following SC command:
sc sdset clussvc D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CRGR;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
Run “sc sdshow clussvc” again to display the new DACL on ClusSvc:
D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
You can see in the new output that ‘GR’ is replaced with ‘CCLCSWLORC’. This is because GENERIC_READ for a service is:
- READ_CONTROL (RC)
- SERVICE_QUERY_CONFIG or Query service configuration information (CC)
- SERVICE_QUERY_STATUS or Query status of service (LC)
- SERVICE_INTERROGATE or Query information from service (LO)
- SERVICE_ENUMERATE_DEPENDENTS or Enumerate dependencies of service (SW)
This will ensure the user has the GENERIC_READ access right to ClusSvc.
4. Grant the user access rights to Service Control Manager (SCM) on the non-clustered server
There is a DWORD registry value named HKLM\Software\Microsoft\Windows NT\CurrentVersion\Cluster Server\ClusterInstallationState on both clustered and non-clustered servers. On clustered Windows Server 2003, the value is 2 or 3. On non-clustered Windows Server 2003, the value is 1, meaning “Files Copied but Cluster Service not configured.”
The GENERIC_READ access right to SCM is not enforced on standalone servers so the event ID 560 for SCM can be ignored on non-clustered servers. The exception is Windows Server 2003 Web Edition where the subkey is absent by default. To be able to ignore this error as on Standard Edition, add the following cluster registry information:
C:\>reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Cluster Server"
C:\>reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Cluster Server" /v ClusterInstallationState /d 1 /t REG_DWORD
5. Grant the user access rights to MSDTC on both the clustered and non-clustered servers
The desired access right to the MSDTC service is SERVICE_QUERY_CONFIG (CC). The event ID 560 for the MSDTC service will be logged in the security event log if the DACL is (A;;CR;;;AU).
To display the DACL on MSDTC, run the following SC command:
sc sdshow msdtc
Here is a sample output
D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CR;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPRC;;;WD)(A;;CCLCSWRPLORC;;;NS)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
If you see the error 560 for MSDTC in the event log, you need add CC to (A;;CR;;;AU) with the following SC command:
sc sdset msdtc D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCCR;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPRC;;;WD)(A;;CCLCSWRPLORC;;;NS)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
The SC command is used throughout this article. Subinacl is another utility that is commonly used to display and grant user rights to any services. Note Subinacl cannot be used for SCM. The article below has examples of using the utility:
Grant Users Rights to Manage Services in Windows Server 2003
For more information about Service Security and Access Rights, ACE Strings and Service DACLs, see the following articles:
Service Security and Access Rights
ACE Strings
Best practices and guidance for writers of service discretionary access control lists
Comments
Anonymous
December 14, 2009
Brilliant! Spot on article regarding my problem, thanks guys!Anonymous
December 14, 2011
The user account associated with the client application must be a member of the local Administrators group ???Anonymous
October 28, 2013
Thanks guys, Adding the user for the client application to the local admin group resolved my issue.Anonymous
October 28, 2013
It is not required to have the client application user account in the local admin group.Anonymous
February 02, 2014
"It is not required to have the client application user account in the local admin group. " What do you mean? How can it run if it's not in the local admin group? (is there another group the identity can be added to to make this work?)Anonymous
February 02, 2014
The user account associated with the client app is implicitly in the authentication users group (AU). You need to ensure the AU group is given the correct permissions to SCM and the services described in the articles.