Support Tip: RTSP and RTSPS authentication in App-V fails after Live Essentials 2011 or later is installed

We’ve seen reports where RTSP and RTSPS authentication in Microsoft Application Virtualization (App-V) fails after Live Essentials 2011 or later is installed.  This occurs because the installation of Live Essentials introduces a new Security Support Provider (SSP) named LiveSSP that provides authentication support for Microsoft accounts. The LiveSSP provider is included by default in Windows 8 and later and is included in the Office 365 Sign-in Assistant. 

Customers who experience this issue report that performing a DC refresh or launching applications fail with an error message that states “valid NTLM credentials are required.” In the case of launching applications, a dialogue box will pop up to prompt for credentials. 

To work around this issue simply remove the LiveSSP entry from the “Security Packages“ value underneath the HKLM\SYSTEM\CurrentControlSet\Control\LSA registry key.

More Information

In our investigation it was determined that the common thread in all of these cases was that there was a mismatch between the SPN that the Application Virtualization Management Service (HWS) server registers and the SPN that the client expects to use. 

By default, the HWS server will create two new SPNs named SoftGrid/HostName and SoftGrid/<FQDN> with the machine account as the owner.  You can also choose to register alternative SPNs that are not derived from the hostname of the HWS server provided the name used is registered in DNS. In that scenario, the client uses the alternate server name provided to form the SPN.

When the App-V client makes an RTSP connection to the server for DCRefresh or to stream a package, the first authentication attempt is an anonymous one in the hope that the server doesn’t have any security configured. When anonymous authentication fails, we use the Negotiate Security Support Provider (SSP) in Windows for authentication.  This SSP provides either Kerberos or NTLM authentication depending on what it determines is the appropriate authentication protocol to use, and the client forms the expected SPN from the address you specified in the OSD file, ASR override, or the DC refresh server list. 

If the client SPN matches the server SPN then authentication will succeed using Kerberos.  However, when there is a mismatch in the expected client and server SPNs, Kerberos authentication is not possible and we expect to fall back to NTLM authentication.  In the case where Live Essentials is not installed (and the LiveSSP provider is not present), NTLM authentication is successful using Negotiate SSP.  If Live Essentials is installed, and the LiveSSP provider is present, the fallback to NTLM fails.  Downstream, we will reattempt NTLM authentication for the Internet Facing Scenario for secure RTSPS connections, but do not do the same for non-secure RTSP connections.  NTLM is known to have issues around security and it not suitable for authentication over the Internet.

This presents an alternate solution which is to make sure you use matching SPNs between client and servers. Ultimately this is the best approach to take because Kerberos provides for far greater security than NTLM and is suitable for use over the Internet. For more information on how to configure SPNs in an App-V environment please see the following:

KB2605366 - How to configure Microsoft App-V with Network Load Balancing (NLB) (https://support.microsoft.com/kb/2605366)