On Debugging Virtual Applications: Part 1: Overview (or Let’s Start at the Beginning)
For many application packagers, virtual application sequencers, and general IT pros, the concept of actual “debugging” can take on many meanings. Often the words “troubleshooting” and “debugging” are interspersed – especially when reading articles and blogs dealing with the topic of trying to dissect what may be occurring when a virtual application is not functioning as expected. When we speak of the word “debugging” in the context of its meaning with regards to programming and compiled software code, it is simply the dissection and reverse engineering of binaries to determine the root cause of an issue or basically “find the bug” in the code.
The level of depth may vary depending on the tools being leveraged and the amount of access to code or symbols. For example, open-sourced code projects on the web are very easy to debug because – well – the code is distributed alongside of the binaries. In addition, special files called “symbols” are also often available if need be. This is especially helpful in the world of Windows debugging. For Closed-Source binaries – like Windows, access is limited to what API’s are exposed and documented within MSDN, the SDK’s, and publically available symbol files. Still, this tremendously aids our ISV partners when they are troubleshooting issues with their own code running on top of the Windows platform.
Enter the Virtual Application
What makes this especially complex in the world of virtual applications is that the surface area expands to not only include the original application but also the virtualization engine that is maintaining its sandbox – specifically its isolation and/or state separation mechanisms. With this, you have essentially increased your variables for issues. Whereas a native application involves one vendor; running on top of another vendor’s operations system, a virtual application now deals with potentially three different vendors (not even counting the potentially amount of 3rd-party vendors that could also be hooked into the kernel via filter and device drivers.) In the case of Microsoft and App-V, if the application being virtualized is a Microsoft application, there are unlimited resources internally to work on that application. In most cases, that represents less than one-hundredth of one percent of the applications out there in the ecosystem – at best. Most cases, the application is external. When that is the case, the debugger must determine the following:
Is the application open sourced or closed source?
If the application is open-sourced, the application can be easily investigated alongside the virtualization subsystems and likely debugged pending that the individual doing the debugging understand the source code and has the proper tools to debug the application from within the share operating system (in the case of App-V – that would be Windows.)
If the application is closed source, what resources are available from the vendor?
This is where it can be challenging. When you are debugging a closed source application running virtually, it requires significant insight into the application – especially if the application is running in native code. While Microsoft makes public symbols available for ISV’s to help with debugging, often the opposite is not true. As a result, the debugging is “best-effort” at best and is usually limited to basic reverse engineering tools like Process Monitor, API Monitor, or DbgView. One exception to this - that I have encountered - have been situations where the application encounters specific issues when virtualized – and those issues cannot be reproduced on a natively installed instance of the application. In those cases, the focus can shift to the virtualization engine however, even in these situations, working in triangulation with the application vendor yields more success – much quicker.
Is the application using a 3rd-party application virtualization engine by a vendor different from the vendor of the underlying operating system?
In this scenario, the application is written by one vendor, running on top of an operating system by a different vendor, and then sandboxed using an application virtualization by yet another vendor. In the case of Windows, the application is using a non-Microsoft virtualization solution. There have been many times where I was working support for App-V and a customer would call in with an issue they were having virtualizing a version of Office or Visual Studio on a non-Microsoft platform. I would always re-direct the customer to the vendor of the app virt stack – even though we were the vendor of the application being virtualized as well as the underlying operating system. I would then direct the customer to reach out to the Office or Visual Studio team as well to work in triangulation.
Relationships of Application to Support Vendors
When debating the best source for debugging virtual applications, please feel free to leverage the following matrix I constructed to assist you in reaching out to the most likely resources that will be able to help resolve the issue.
| Application Vendor | Operating System Vendor | AppVirt Stack Vendor | Best Vendor(s) for Virt Debugging | |
| Best Case Scenario | Vendor A | Vendor A | Vendor A | Vendor A |
| Rare in the Windows World | Vendor A | Vendor B | Vendor A | Vendor A |
| Typical 3rd-party AppVirt Scenario | Vendor A | Vendor B | Vendor C | Vendor C first & Vendor A optional |
| Most Common at Microsoft | Vendor A | Vendor B | Vendor B | Vendor B first & Vendor A optional |
The reason I make the above recommendations is because at some point the application, the application virtualization engine, or even perhaps the operating system may require some debugging – especially if there is a potential bug. If the resources troubleshooting the issue do not have access to the resources and tools needed to debug the issue – then you are essential throwing darts against the wall – and it will lead you potentially down a rabbit hole.
Why Discuss Debugging?
I have decided to start discussing the topic of virtual application debugging to serve the following purposes:
- To demystify the concept for application packagers and IT Pros in the Application Virtualization space. There are tools and concepts that can help these professionals to further arm their skills and enhance their arsenals and toolboxes. Many reverse engineering tools such as ProcMon can only go so far.
- To aid software vendors in how to debug applications running under App-V and how their applications may be affected.
- To aid customers in how to gather and collect the appropriate debugging information to help Microsoft and other software vendors diagnose issues, isolate root cause, and resolve problems and bugs quicker.
Next Up Part 2: Types, Modes, and Situations
Comments
- Anonymous
February 20, 2016
Thanks for taking this on for everyone, Steve. Looking forward to the next part(s).
The situation depicted in the chart is completely analogous to what we had with Citrix 10-15 years ago (where you replace Citrix/RDS with the AppVirt column). Vendors didn’t really know what Citrix was and often did not support the customer under Citrix – asking them to reproduce the problem without Citrix involved.
Over time this changed for the Citrix column version of the chart. As the Independent Software Vendors (ISVs) found more customers using Citrix they became more comfortable troubleshooting, and eventually they recognized the value of revenue associated with those deployments. So they took it upon themselves to include Citrix in their test matrix and support their customers properly. Many software vendors now support these deployments.
We have always wanted the vendors to support App-V similarly, but more often than not it is difficult to get past the first level support, especially for an IT Pro at a smaller company, thus we often don’t even try. However, in the last nine months I have detected a positive change in attitudes of many of the more proactive software vendors out there to start supporting App-V. They are starting to listen. I would encourage enterprises using App-V to just ask the vendor for support. - Anonymous
February 20, 2016
The comment has been removed - Anonymous
February 22, 2016
Awesome as always. Thanks Steve! - Anonymous
February 24, 2016
Please rate so i can keep doing them. :)