Running the Windows 8 App Compatibility Tests

One other problem I mentioned with getting this most recent update to OneNote into the store was ensuring we were running the Windows Application Certification Kit tests in our lab. I wanted to walk through some of the work I did here - mostly this is a "lessons learned" type of task.

The tests can be downloaded as part of the Windows SDK for Win8 and, from one point of view, is just an executable that runs and tests your appx file for compliance. There is a long list of tests that it runs over on the WACK page - tests like performance testing, API deprecation Direct 3D tests and so on.

We have a BVT test automated that runs for every build of Office we create. It installs Windows 8, installs the 64 bit build of OneNote then runs the tests. It was passing, which meant that all the Windows tests would pass. It had not failed so I thought we were in good shape.

But when we handed off (what we thought to be) the final version of OneNote, the windows store sent us back an email that said we had failed the API deprecation test. Details are over on the page I linked above, but the general point is that Windows is removing some older APIs that no longer are supported and if your application tries to use them, they won't work - obviously. I did not expect this type of result - we had completed all this work and the tests were passing.

First thing was to run the tests locally. We copied over the WACK tests and ran the test against the 64 bit build again and they passed. I was still stymied after this since this test is completely deterministic until someone asked about x86 (32 bit) builds. Sure enough, when we tested the x86 build, someone had managed to check in some code that used an API that was on the deprecated list. We fixed that, triple checked all the builds (x64, x86 and ARM), resubmitted and passed the tests.

In the meantime, I wanted to find out why our test did not check the 32 bit builds. It was a simple automation setting - we had the test set to run every day on 64 bit builds, but not daily on x86. To get that setting enabled, we have a practice that the test has to run 1000 times with no errors (to prove it is a stable test) so I kicked off an automation run that ran that test 1000 times. It passed and I enabled this for daily runs.

We are now in a better position and we call this our "culture of constant improvement." We just want to make each build better than the previous and this engineering effort will help us out with that goal.

Questions, comments, concerns and criticisms always welcome,

John