Test Team vs. The Matrix: Fight!

For the past few weeks, we've been running our Full Automation Run (FAR) against several dozen different Team Foundation Server configurations. We've tested everything from Workgroups to VPCs, Chinese to Turkish, IPv6 to 64bit machines. We reverently call the spreadsheet of test configurations "The Matrix", and The Matrix definitely has us...

It's very interesting how testcode that worked fine on English product/English OS in a domain environment needed to be tweaked to pass as expected on other configurations. Two of the most interesting issues we've dealt with are as follows.

In our Workgroup environment, usernames are in the form [AppTierMachineName]\joeblogs. Since we'd originally used System.Windows.Forms.SystemInformation.UserDomainName on the client to get the domain name, our "UserFullName" would be [ClientMachineName]\joeblogs and any output that expected full name would fail. The workaround turned out to be more difficult than just dropping in the AT Machine name, since many of our test runs have two AppTiers. We need to first determine which server we're executing against, and then find its name. At the same time, we don't want to always make this change since we still have to run automation in Domain environments. The fix turned out to be a bit of a headache to piece together, but should now cover all the bases.

The second issue occurred in non-English product FARs. We load the localized resource strings as part of each testcase so we can properly match expected to actual output. However, not all resources are localized. Some, such as "{0}: {1}" (also known as "AColonB") are only found in the default Neutral Language Resources. Therefore, we had to load both the neutral resources dll and the localized resource dll from two separate files. To do this, we'd been using Assembly.LoadFile for quite some time. We also used a testcode method on the GUI resource strings to convert them from resources to actual expected values. However, the combination of these two only gave us English (i.e. Neutral) resources and so tests were failing left and right. It kind of made us feel like Neo during the fight scene in Matrix: Reloaded where Neo battled dozens of Agent Smiths. Turns out that LoadFile was the culprit- in 2.0, LoadFile doesn't really do what you wanted it to. After making the three character change, our pass rates skyrocketed.

While it's definitely confidence inspiring product-wise to discover that a test failure is due to test code rather than the product code, it's good to see more recent test runs experiencing fewer and fewer of these issues as we prepare to sign-off on the RC soon.