Anatomy of an Outlier Bug: The Issue of the Failed App-V Report Upload
When software malfunctions, product support teams attempt to narrow the issue down to user error, configuration, or code. If the software attempts to perform a function where a specific result is expected and the actual results are different, we classify that as a bug. Of course, there is great debate on what is perceived as a bug, or is simply “by design” meaning the software was doing exactly what it was programmed to do – it just may be completely different from what the end user expected.
Let’s assume we are not having the “by design” vs. “bug” debate. When a bug confirmed, the next common item that is measured is the impact of that bug - impact within the specific customer’s environment, the frequency and likelihood of the bug occurrence. Bugs that have only occurred one or two times where the impact is minimal is usually referred to as an “outlier bug.”
Outlier bugs can be tricky – especially if there is not a known workaround. Software vendors have to weigh the risk and cost of implementing bug fixes via current version patches (as opposed to correcting the code for the next release.)
A while back I encountered one of those outlier bugs in which this bug only affected one customer (at the time) and it only revolved around one specific App-V package. It involved the App-V Client Reporting Agent failing to upload reporting XML data. The following issue was occurring.
Client machines were failing to upload reporting data. Upon a manual test using the Send-AppVClientReport cmdlet, it would yield the following error:
PS C:\Windows\system32> Send-AppvClientReport
Send-AppvClientReport : No reporting data has been sent to the specified URL. Verify the URL and try again.
Operation attempted: Send reporting data to reporting server.
AppV Error Code: 1300000013.
Please consult AppV Client Event Log for more details.
At line:1 char:1
+ Send-AppvClientReport
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (:) [Send-AppvClientReport], ClientException
+ FullyQualifiedErrorId : SendReportError,Microsoft.AppV.AppvClientPowerShell.SendAppvClientReport
==============
This issue was NOT occurring on all of the clients, so the first major step was to find the common denominator. Eventually we found the common denominator was a specific package. In addition, we actually got quite lucky. In many cases, the machines getting this issue only had one specific package deployed to them.
So we grabbed the XML file (C:\ProgramData\Microsoft\AppV\Client\Reporting\0dd6c24e-be93-48f7-b531-4eaa007128ec.xml) to view the Reporting cache and eventually narrowed it down to only occurring when the specific application was published.
What makes this issue an outlier is that the element that was causing the issue involved a package where the primary application had a version field that was greater than 16 characters. On a lark, we started modifying the XML data manually and found that if the reporting XML file was manually edited, and enough digits were removed from the version field, the upload worked. For some reason, only 16 characters were allowed in the version field. All of the fields were supposed to allow for 32.
Once root cause was narrowed down, the time came to assess the impact of the bug. It was definitely viewed as an outlier bug due to the fact there was only one known occurrence (again – at the time.) The assessment of how likely this would reoccur soon became a moot point as the good news here as the issue was within the database itself. The App-V 5 reporting service is stateless and writes directly to the temporary tables in the AppVReporting database. So we were able to fix the issue for the customer without cracking open binaries for patching.
If you are interested in the database script, it was published out on the TechNet Gallery here:
https://gallery.technet.microsoft.com/App-V-5-Fix-for-App-V-f8c1ac29
This fix was included in the App-V 5.1 release.