The Microsoft App-V 5.0 Sequencer and Client Troubleshooting Guide

아티클
06/30/2015

~ John Behneman | Senior Support Escalation Engineer

Hello everyone, John Behneman here again. I’d like to share with you a document I just completed that provides strategies, troubleshooting framework, and a consistent framework to assist our customers new to Microsoft Application Virtualization 5.0 (App-V 5.0). It is not a training guide in the traditional sense, but rather it’s meant to provide a quick overview of troubleshooting techniques specific to App-V 5.0. It highlights typical issues we see at Microsoft on a daily basis and provides a starting point with tactics for troubleshooting these issues. Public references from Microsoft employees and MVPs will be provided, however be aware that some of these links are from outside sources and thus may change without warning.

NOTE : Due to the inclusion of 3^rd party references there is no warranty or guarantee of accuracy of the references provided. All information is provided ‘as is’ , and in addition, some of the information may be dated and may not apply to the latest versions of App-V 5.0 due to the evolution of the product since its inception.

General App-V Guidance

As a rule, running the latest versions of the App-V 5.0 client and sequencer are considered a best practice, but a word of caution that thorough testing in a non-production environment should be done for all App-V 5.0 packages before upgrading a production environment. Due to the nature of the product, installing hotfixes can be less of a feature set change than installing service packs. App-V 5.0 was rewritten from scratch and totally re-architected versus it’s version 4.6 predecessor, and with Service Pack 3 it has evolved significantly from the RTM release as noted in this 3^rd party blog post:

Tim on App-V 5.0 SP3: https://www.tmurgent.com/TmBlog/?p=2211

Be thorough in your preparations when making big leaps from say 5.0 RTM to 5.0 SP3. A jump this significant has the potential to run into problems if you’re not aware of how the changes may affect you. The prerequisites have changed during the evolution and failure to read the latest documentation is a recipe for trouble. As of version 5.0 Service Pack 2 Hotfix Package 4, the Microsoft Visual C++ 2013 Redistributable must be installed for proper functionality as indicated in the following KB article.

2956985 - Hotfix Package 4 for Microsoft Application Virtualization 5.0 SP2 (https://support.microsoft.com/en-us/kb/2956985/)

As mentioned in the blog post below regarding error 1053, the MSI installer will not detect missing prerequisites but the EXE installer will detect if it’s necessary to install the prerequisites and do so accordingly. This is an example of how one can get into trouble if they don’t read the documentation and end up using the client setup.msi or update service package .msp installer instead of setup.exe. One can determine a missing prerequisite by running the setup.exe of any App-V component against the end point, whether it is a client or server. The article below documents this known issue with the App-V 5.0 client when upgrading to SP2 or higher:

Support Tip: App-V Client service fails to start and generates error 1053

The article above represents a good reason to be cautious when upgrading. Typically it is recommend to upgrade to the latest hotfix within a given service pack level if the service package level is fairly current. For example, if you are running 5.0 SP2, it is recommend to install hotfix 5. If on 5.0 SP1 then install hotfix 3. Our goal is to fix the immediate problem not create and troubleshoot new ones.

Ref: Current list of App-V 5.0 file versions

Please be aware that the App-V client version should be higher than the package sequencer version. We have seen problems when customers use the current version of the sequencer yet their clients are significantly older. Ideally the sequencer should be within the same service pack level as the client. For example, 5.0 SP2 has two versions of the sequencer, RTM and hotfix 4. An RTM package should work fine on RTM and HF4 or higher client version, whereas running a package sequenced on 5.0 SP2 hotfix 4 on a 5.0 SP2 RTM client may work but it’s not recommended.

As new fixes and features are introduced, a best practice is to use the latest version of both the sequencer and the App-V client. Ideally the sequencer and client versions are the same or the client is a version higher but within the same service pack level. Keep in mind that the sequencer is not updated every time a service pack is released. It is only updated when fixes or features are added to the sequencer. The client, the workhorse and most significant component of App-V, is also the most updated component.

With App-V 5.0, one of the ways to determine the version installed is to audit the Control Panel\Programs and Features utility. As shown here, the version of the App-V client is 5.0.10107.0 which is App-V 5.0 SP3:

Oh, but wait, if we select “View installed updates” from the screenshot above we discover Hotfix 1 is installed:

Because of the possibility for confusion here, the following registry key is the recommended way to verify the installed version of App-V:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client

To determine the version of the sequencer used to create a package, load the package in the most current version of the sequencer by selecting “Modify an Existing Virtual Application Package” and choosing “Edit package” and select the “Change History” tab. There is a lot of noteworthy information for a package on the Sequencer “Change History” page:

Sequencer Guidance

The starting point for troubleshooting packaging issues is knowing the sequencer, how it works and best practices. This requires reading the Sequencing Guide:

By far the most difficult cases we receive are sequencing/packaging cases, which can be due to a number of factors:

Limited Sequencing experience and training.
Limited experience in the industry.
Misconceptions about the product and limited knowledge of best practices.
Trying to sequence old or poorly written applications (the most common issue).
Not knowing how the applications work or application requirements.

The best sequencers are application packagers, professionals who have experience repackaging applications to prevent application incompatibilities when large numbers of applications are delivered to desktops or RDS servers. App-V is sold as solution to mitigate application compatibility issues, however keep in mind that this does not make it an application compatibility solution. In the past the most common incompatibility was the DLL conflict which side-by-side assemblies were designed to correct (we used to call it “DLL hell” back in the Windows 98 days). Sequencing complex applications requires a very special skill set that many do not have, which is why sequencing can be a challenge.

There are 4 basic sequencing problems:

A virtualized application fails to work as expected (due to any number of reasons).
A virtualized application runs slowly (4.6 to 5.0 conversion or application ported from UNIX or another OS?).
The virtualized application works but one or more features do not (can be caused by failing to exercise a feature during sequencing).
Although rare, the sequencer will not sequence an application; the sequencer either hangs or crashes and fails to create a package. In this scenario, open a case with Microsoft or decide whether the application is truly a good candidate for virtualization.

A couple of points to understand about sequencing and testing applications which are easily over looked:

1. If a feature of an application fails to work virtualized, verify it works installed locally. We have seen countless cases where the feature didn’t work locally installed, and obviously if it doesn’t work locally installed it will not work virtualized. This is why it is important to know how an application works locally before virtualizing it. This cannot be emphasized enough.

2. Always use the latest version of the App-V sequencer and client. Hours have been spent troubleshooting a problem just to find out the latest version of the sequencer or client resolves it.

3. If possible, test on a clean OS with no antivirus or other 3^rd party agents installed, preferably a cleanly installed OS as a basic test. This mainly applies to the App-V client but is also applicable to the sequencer. For a sequencing system, it should have the same policies as the endpoints, with the exception of the previous statement.

4. Sequencer OS version and App-V client OS version should match. If they don’t this can cause package failures. It’s possible to create a working package sequenced on one OS version and deployed to another but it’s not guaranteed to work. There is a lot of confusion and misinformation surrounding this issue. Some have been taught to sequence everything on an older OS like Windows 7 so that they can streamline deployment to Windows 7, Windows Server 2008 and Windows Server 2012, and while most of the time this will work, sometimes it will not. Office is a good example of a program that typically doesn’t allow this approach. A few SoftGrid and App-V evangelists have sold App-V as an application compatibility solution but remember that it is not.

PVAD or Not to PVAD

Understand PVAD (SFT_MNT) verses VFS and how it influences the virtualization of an App-V 5.0 package. This is very important. In App-V 4.X, a best practice was to sequence everything using SFT_MNT (MNT) which meant installing applications to the Q drive, the default virtual drive used by the 4.6 sequencer. Unfortunately, not all programs work when installed to the Q drive and have to be installed to the C drive which is consider a VFS install. The following articles provide more information on this topic:

This is also true with App-V 5.0, where some applications will not work when sequenced as PVAD (same as 4.x SFT_MNT or MNT) and others will not work when sequenced as VFS. This is why testing is required to determine which type of installation works for a given application. This means sequencing an application at least twice (once as a PVAD and then VFS) if the original attempt fails to generate a working package.

The best practice for App-V 5.0 is to sequence applications as VFS, which is now the default in App-V 5.0 SP3. The App-V product group removed the PVAD option in the SP3 sequencer although PVAD can be added back via a registry key or command line option (EnablePVADControl). Once again, some applications will not work as a VFS package and must be sequenced as PVAD, therefore you will need to know how to enable PVAD when required. The following article explains this change and provides a method to restore the PVAD prompt:

Merged Roots in App-V 5.0 SP3 – Free from the PVAD

The following article confirms that some applications require that you use PVAD:

Tim on App-V 5.0 SP3: https://www.tmurgent.com/TmBlog/?p=2211

Snippet from above article:

The MVPs have a short list of apps that we have found that need “PVADing”, including:

WinZip
ArcGIS Desktop tool
Avaya CMS Supervisor R17
Wonderware InTouch 10.x

Additional references on this topic:

Problem: The Sequencer fails to capture files required for a package to work properly

A common failure of a new package is missing files. This is often due to default exclusions of the sequencer. Some applications write files to user’s %Appdata% or %LocalAppdata% profile location, and this will be evident in a Process Monitor trace where the application executable fails to find a file in the %AppData% or %LocalAppdata% locations. The Sequencer also reports on excluded files during the final phases of the sequencing process. Most of the time this can be corrected by removing these default exclusions found on Tools\Options\Exclusion Items tab:

The following blog post provides more details, but the bottom line is that in some cases, removing the LocalAppData and LocalAppDataLow exclusion is required to create a working package. One example of this is Apple QuickTime. It installs a dependency to the AppData directory so you will receive a DLL/EXE errors when launching the packaged application if the default exclusions are in place.

App-V Support Tip: Look out for this issue when sequencing your apps

Another way to determine if a nonfunctioning package is missing files is to use the devirtualization technique outlined in the following blog post:

AppV: On Devirtualization

If devirtualization of the package produces a working application on the sequencer then the package problem must be related to one of the following items mentioned in the blog:

Virtualization subsystems (Virtual COM, Virtual Objects, Virtual Services, VFS, etc.)
- Can be tested / addressed by enabling “Allow all named objects to interact with the local system” and “Allow all COM objects to interact with the local system” on the advanced tab of the Sequencer.
Incompatibilities with virtualization as a whole (drivers, COM+, etc.).
Information that could have been captured incorrectly in the FilesystemMetadata.xml file (Short Names, Directory Opacity).
Registry Opacity.
Static Dependencies.

While we are on the Advanced tab it’s worth noting the 3^rd option “Allow virtual applications full write permissions to the virtual file system” introduced in SP2 Hotfix 4. Some old (and poorly written) applications write files to the application installation folder instead of %Appdata%. Prior to the introduction of this feature, applications like these would fail due to an access denied event when the application attempts to create a file in the installation directory. Now, enabling this setting allows this behavior. Please see the following blog posts for more detailed information regarding these 3 settings:

Item 2 in our list above (incompatibilities with virtualization as a whole) is addressed by this blog post, which is a commonly asked question. App-V may not work with some drivers and COM+ objects which means an application may not be suitable for virtualization.

Applications Suitable for Virtualizing in Microsoft Application Virtualization (App-V)

Potential workarounds for these COM issues are detailed in the following:

App-V: On COM+

Item 3 in our list above (information that could have been captured incorrectly) regards incorrect entries in the FilesystemMetadata.xml which is contained in the .AppV file of the package. To access this file, make a copy of the *.AppV file and rename it to .zip, then open the ZIP file and extract the FilesystemMetadata.xml and examine it for incorrect entries. Capture sequencer debug logs on the client (see next section) to get more detail. Please engage Microsoft Support if necessary as incorrect entries in this file are most likely due to an issue with the sequencer.

Item 4 on our list above (Registry opacity)is explained in the following documentation:

Item 5 in our list above addresses the sequenced application’s dependences. If the dependences are not included in the package then they need to be installed on client before the App-V package will work. These could include C++ runtime, Java, .NET, etc.

Sequencing and troubleshooting App-V packages is more of an Art than a science which makes this blog a must read:

On the Art of Troubleshooting App-V Applications

Sequencer logs

The App-V 5.0 logging location has changed from the 4.6 location. Just like the client, it now logs to the Event Viewer, and debug logging is not enabled by default. Sequencer-Debug logging is not visible until View\Show Analytic and debug logs is checked in Event Viewer. Select and right-click one of the 2 debug logs to enable them. The debug logs are useful when the Sequencer misbehaves during sequencing, such as the sequencer crashes or error messages prevent package creation, etc. Unfortunately these logs are not always helpful when trying to determine why a sequenced package fails to work on a client.

After sequencing an application, human readable artifacts can be found in the %LocalAppData%\temp\Scratch folder on the sequencer:

A Report.xml file is created during package generation and this is definitively the first place to start when troubleshooting sequencing issues.

Notes on how to read the Report.xml file (also visible in the Sequencer UI at certain stages):

All UI reporting messages (best practice messages like “Reboot pending” and sequencing issues like “files excluded”) are preserved in the resulting Report.xml file with every package. This is the same format that was introduced in 4.6 SP1.
The report is in XML format and can be viewed easily with an XML reader (e.g. Internet Explorer, Notepad or XML Notepad).
Each entry is an error (“Issue Code”) with a Description (both “Short” and “Full”) which describes the problem.
Pay particular note to the files that were excluded. When a package is not functioning on the client or when devirtualized, these are your first candidates for issues.
Other messages may present potential problems as well, please go through these before looking in the ETL logs or going to Process Monitor.

Example Report.xml file:

One additional topic we should address before moving on is how Environment Variables are processed. See the following for more information:

App-V 5: On Environment Variables

Virtualizing Microsoft Office

Overcoming the challenges of virtualizing Office is a fairly common call at Microsoft. We usually recommend not virtualizing Office due to the limitations with integration, plug-ins and future package upgrade issues, however please review the “Known issues and limitations...” article listed below to before committing to virtualizing Office. Most issues have been documented but sometimes the content online or in our Knowledge Base can be missed because you may not know the proper key words to search. Please reach out to one of the many App-V forums for help. Many people have successfully virtualized Microsoft Office so the forums can be a rich source for answers on this topic.

With Office 2013, the Office Deployment Toolkit (ODT) is used to create App-V packages and not the sequencer. Sequencing Office 2013 using the standard sequencer is not supported. See the following for more information:

Supported scenarios for deploying Microsoft Office as an App-V package

Since Office 2013 virtualization is now achieved through the ODT, this means the Office support team now owns most of the code tree and has the resources necessary to help in many virtualization issues. The following flow chart can be used to determine whether App-V or Office support is required should you need assistance:

While the details of troubleshooting Office packages is beyond the scope of this guide, the following references are available to help with these issues:

Office Add-in references:

References for Internet Explorer and Java:

Troubleshooting the App-V 5.0 client

The App-V client is the most important component of an App-V infrastructure. It hydrates the App-V packages and integrates with the rest of the App-V infrastructure. Knowing how to troubleshoot the App-V client is the most important aspect of supporting App-V. Most of the critical App-V cases we get here in support are due to App-V client failures. The following App-V 5.0 Application Publishing and Client Interaction guide is a must read, especially the Copy on Write (COW) section which covers the component that produces the most problems:

App-V 5 SP2 Application Publishing and Client Interaction

If you don’t have time to read the 43 page “Application Publishing and Client Interaction” document you can get a “Fundamentals” jump start reading the following posts:

I hate to sound redundant but it bears repeating: Always test using the latest version of the App-V 5.0 client, especially if troubleshooting an RDS client. By far, the RDS client is the most difficult to troubleshoot due to profile management, folder redirection and Citrix integration issues. Citrix integration is supported by Citrix, but at times for troubleshooting it is ideal to test without Citrix and profile management in the picture.

A great many App-V client issues are related to 3^rd party components, consequently troubleshooting may require eliminating as many 3^rd party programs as possible. Ideally, test with the latest App-V 5.0 client installed on a clean OS, with no folder redirection, profile management or antivirus. If the issue is not reproducible in this “clean” scenario then you know the App-V 5.0 client works properly. Adding 3^rd party components until the issue returns is the recommended troubleshooting process, unfortunately few like this approach because it is time consuming, however it is the only way to determine root cause. App-V client errors no one has ever seen before can usually be attributed to 3^rd party processes.

Unlike App-V 4.x, App-V 5.0 does not require configuration adjustments to work as a standalone client; it’s standalone by default, making testing very straight forward. An App-V package can be deployed locally using one of 2 methods: PowerShell or install using the package MSI file generated by the sequencer. The following blog post provides the details:

App-V 5.0 Standalone Mode – Adding and Publishing a Package
Another good reference is App-V 5.0 Client PowerShell Deep Dive

If extensive testing is required, I recommend that you check out AppVManage. It’s a free 3^rd party tool that allows you to deploy and remove packages at the click of a button. It is a great training tool as well since using it will introduce all the important App-V 5.0 concepts and features.

AppV_Manage Introduction

As you can see from the screenshot above, AppVManage will publish packages globally or to users, with or without configuration files, unpublish, mount and analyze. The Analyze feature displays package configuration information helpful in determining if two or more packages will work together correctly in a connection group, specifically the COM configuration of the two packages, which must be the same. Please see the following for more information:

AppVManage allows for a standalone test lab using the App-V client; there’s no need for anything else. For App-V 5.0 client and package issues this is the perfect setup. After installing AppVManage, configure it to use a local content store as shown in the screenshots below. To test a package just drop the package folder into the local content share, click refresh and AppVManage will discover it. Also notice that it displays the current version of the App-V client along with the version of AppVManage. AppVManage training is beyond the scope of this document but there is ample help and documentation on the AppVManage website.

Local content share with package folders:

Contents of the Office2013Full package folder:

AppVManage configured to use the local content share, notice the versions at the bottom:

:
The tabs reveal the various App-V 5.0 Client tasks that can be accomplished with this tool:

Publishing Tab – Displays all the packages available in the content share and controls to manage them.
Client Packages – Displays the status and detail about packages and how they are published.
Connection Groups – Shows connection groups and provides the ability to create and manage them.
Debug Packages – Launch various tools within a given package VE in order to troubleshoot it.
Client Config – Shows client configuration settings and tests Management server connectivity.
Event Viewer –Shows events from the default event logs, Client\Admin and Client\Operational.
Debug Events – Displays all available debug etl channels and controls to enable and disable.
Tool Config –Displays the current configuration of AppVManage andcontrols to change.

For more detailed information about AppVManage tool functions please see the following reference:

AppV_Manage Tab Descriptions

Microsoft has a free App-V downloadable troubleshooter available at https://aka.ms/appvdiag . This is a data collection tool used primarily by Microsoft support but it can be leveraged by anyone to collect data and discover known issues present in an App-V environment. After running the diagnostic, a Troubleshooting Report is displayed. This information is also available by opening the ResultReport.xml in Internet Explorer. Double-clicking on it may generate several warning messages about running scripts so select ‘Yes’ and ‘Allow blocked content’ if prompted to do so. The following is an example of the output from the Troubleshooting report, where in this example the machine is missing a TermSrvReadyEvent exclusion which will cause problems if Hotfix 2984972 is installed on a client machine.

To find the data collected, go to Start -> Run and enter in %localAppdata%

This will take you to %userprofile%\AppData\Local (see screenshots below). The Diagnostic data is stored in the ‘ElevatedDiagnostic’ directory:

All the data collected is capture to a *.cab file which you can attach this to a Microsoft Support case if desired. You can also extract the data to see what it collected.

2971404- Microsoft Application Virtualization (App-V) troubleshooter

If a problem is isolated to the App-V client, run it on the client. If related to publishing failures then run it on the Management/Publishing servers and the client simultaneously. If the packages are deployed using Configuration Manager, the ConfigMgr support team may need to be engaged. The App-V diagnostic does capture the ConfigMgr client agent logs.

The diagnostic has a tracing feature which enables all App-V 5.0 debug logs of the App-V component on which its run. When the diagnostic gets to the following screen please duplicate the issue, then once duplicated click Next.

As mentioned previously, when the diagnostic completes it saves all the data to a *.cab file assuming it was downloaded and run from https://aka.ms/appvdiag . If run from a link provided by a Microsoft Support Engineer, it will be uploaded to Microsoft so long as the computer has Internet connectivity.

The following chart contains a list of important logs captured by the diagnostic:

Appv 5.0 Client Information

Log	Content/Purpose
ResultReport.xml	XML that displays all collected Data in IE
Get-AppvClientConfiguration.txt	App-V 5 Client configuration
Get-AppvClientMode.txt	App-V 5 Client mode
Get-AppvClientPackage.txt	App-V 5 Client Packages
Get-AppvPublishingServer.txt	App-V 5 Client Publishing server
Get-AppvVirtualProcess.txt	App-V 5 running virtual processes
<Computer_Name>_evt_AppVClient-Admin.txt	AppV 5.0 Admin event logs enabled by default
<Computer_Name>_evt_AppVClient-Operational.txt	AppV 5.0 Operational event logs enabled by default
<Computer_Name>_sftlog.txt	App-V 4.6 Client log
<Computer_Name>_CCMLogs.zip	Collection of SCCM agent logs
<Computer_Name>_APPVdebug-4895_000001.etl.txt	App-V 5 Debug event output consolidated into one text file.
<Computer_Name>_PendingRebootStatus (Status is TRUE).txt	Reboot status, if “Status is True” the machine needs to be rebooted.
<Computer_Name>_evt_System.txt	OS System event log
<Computer_Name>_evt_Application.txt	OS Application event log
<Computer_Name>_Hotfixes.TXT	Recent hotfix updates installed
<Computer_Name>_Installed-Software.txt	List of installed Software on the OS
<Computer_Name>_AppVRegistryKeys.zip	Collection of App-V registry hives
<Computer_Name>_GPResult.txt	List of Group Policy objects applied to the OS
<Computer_Name>_msinfo32.nfo	System information collected from the machine

The logs highlighted in yellow are the ones to look at first. The <Computer_Name>_APPVdebug-4895_000001.etl.txt is the log created when duplicating an issue on the client and it contains all debug tracing.

Details regarding the required fundamental knowledge to troubleshoot issues and the tools to do so, how to translate the errors in these logs to something meaningful and how learn the various troubleshooting techniques can be found in the following:

Before we leave the diagnostic, there are a couple of things that I should mention. When you run the troubleshooter you will have two options: Normal and Advanced. Always choose NORMAL unless told otherwise by Microsoft.

If a computer does not have Internet connectivity or the diagnostic fails to run for some reason, the following screen may be displayed:

Either try the suggested solutions or run the diagnostic on another machine and choose to run it on a different computer. It will download a payload to transfer to the other machine; please follow the prompts, the wizard will walk you through it.

Screenshot with instructions on running the diagnostic from another computer:

One last note about the diagnostic: Please run with an account that has local administrative rights, otherwise it will not be able capture all of the necessary data.

Common App-V client issues

Moving to App-V 5 as App-V 4.6 Mainstream Support Ends

Adobe Acrobat

Requires App-V 5.0 SP3 HF1 client. Adobe Acrobat must be sequenced with 5.0 SP3 sequencer. It is believed that Adobe will post an official 5.0 recipe for Adobe Acrobat on their website in the future.

Citrix Servers

Recommended updates for RDS/Citrix servers

Configuration Information

Information on Connection Groups

Various Other Errors

How To’s

Operations: How Things Work

Performance Related Issues

PowerShell Information

Configuration Manager Integration

Publishing Errors

Roaming Profiles/UE-V/ User Profile Management

Scripting and Deployment Configuration Files

VDI Scenarios

Integrating App-V with Microsoft VDI
App-V 5.0 and Citrix
App-V 5 and Citrix Integration Resources
Optimize OS using VDI Optimization Scripts
- Windows 7
- Windows 8
At least Drain NGEN Queue (Right click on link to download script)
Wondering why mscorsvw.exe has high CPU usage? You can speed it up (Provides an explanation and public download location)
Windows 7 VDI image hot fixes
App-V 5: Revisiting Registry Staging
App-V: A Configuration Template for Deploying to Stateless RDS Clients on Citrix Published Desktops with Citrix UPM for Profile Management
Unable to Launch Applications Integrated with App-V Using Receiver
Profile management and App-V
Deploying Microsoft App-V 5.0 and Citrix XenDesktop 7

Virtual Environments

Other Troubleshooting Resources

In conclusion, I want to thank Tanner Slayton for reviewing this doc and doing edits, as well as the App-V worldwide community for the content and tools referenced in this document.

John Behneman | Senior Support Escalation Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter :