Share via

Tao of the Windows Installer, Part 5

  • Article

Testing is a vital part of the package development process and you can avoid many problems in production if your testing strategy is sound. So this time we will look at a couple of pointers on testing.

I must admit I'm less happy with the arrangement of this section than with some of the others. I think Rule 53 is making a bid for its own section and breaks away from the notion of keeping this to some fairly compact bullet points; in any case, it hopefully gives you a starting point on what to think about in your testing. Let me know what you think.

Series links:

Testing and Support Rule 53: Test thoroughly

It is crucial that you test your packages thoroughly before deployment. A classic mistake is for developers and package authors to test only on their own systems where they have full administrator rights and then discover that normal users cannot use their applications. You may want to divide your testing into these phases: Installation TestingTest your package under different installation scenarios, including:

  • Full user input and silent installation
  • Different combinations of selected features
  • Different user roles (e.g. normal user, administrator, etc.)
  • Different deployment methods (e.g. manual, SMS, GPO, etc.)
  • Administrative and advertised installations

Enable installer logging for each test and examine the system event logs for errors after each run. Make sure that all installation initialisation and run-time errors are resolved. An important, but often overlooked, part of installation testing is to ensure that your setup UI is accessible and can handle different screen resolutions and font sizes. Patching and RepairAs mentioned in the Patching section, it is very important that you think about patching before deployment. In fact you should not only think about it but test it thoroughly. You should include these scenarios:

  • Small update
  • Minor upgrade
  • Major upgrade
  • Product Repair
  • Multiple simultaneous patch installs and uninstalls
  • Installing and uninstalling patches "out of order"

Uninstall TestingMany operating system related errors caused by a repackaged application happen during an uninstall session. Therefore it is extremely important to verify that a generated package does not uninstall “too much” and only uninstalls data that belongs to the package itself and not to the operating system or other applications. Test uninstall as thoroughly as install, including:

  • Uninstall under different user contexts (e.g. normal user and administrator)
  • Uninstall via different methods (e.g. Control Panel, command line, etc.)
  • Make sure that Operating System does not report any integrity problems
  • Reboot a test machine after uninstall (even if the uninstall did not require a reboot)
  • Verify integrity of common system tools and other standard applications

As with install, use Installer logs and event logs to look for errors, and make sure you resolve all those you find. Product Functionality TestA successful installation and uninstall sessions can be insufficient for an application to work correctly. Therefore you need to ensure that the application works as expected. Some things to check:

  • Verify each shortcut in the package starts the corresponding application correctly
  • Walk through main menu items in the application
  • Follow the application functionality test provided by the vendor, if any
  • Verify that file associations by launching a file for each registered extension.
  • Make sure that application opens, save and prints the files correctly

It is recommended that corporations provide application functionality tests allowing a repackaging engineer to quickly verify major application functionality right away, and make sure that a developed MSI package correctly installs an application, and an application is ready to work properly. User Acceptance Test (UAT) Just as important, but often overlooked, is user input into the testing strategy. User Acceptance Testing (UAT) is especially important when re-packaging legacy applications to MSI format. Some points to note for UAT:

  • Application should be deployed for UAT in exactly same manner as it will be deployed in the production environment
  • Pilot the deployment with a selection of “typical” users for the application
  • Make it easy for users to report problems and provide feedback
  • For internal applications, the owning Business Unit must ensure each application passes the UAT to be defined by that Business Unit. Where the package is processed externally an amount of time must be agreed upon, after which it is reasonable to assume the package has been accepted into production
  • It is recommended that product functionality tests be provided for all internal applications. Such tests will allow a packaging engineer to verify major application functionality before submitting a migrated package for UAT
Rule 54: Validate Your Packages
Validation is a means by which you can run certain standard checks (called ICEs) against your package to check for inconsistencies and non-compliance with Installer rules. Many ICEs ship with the Installer SDK in .CUB files and you can write your own. They can be run against your packages using the ORCA or MsiVal2.exe tools from the SDK. You should always fully validate your packages and, much like when working with application development, correct any reported errors and treat all warnings seriously. Validation is also a useful troubleshooting technique, particularly if you are working with 3-rd party packages.
Rule 55: Use Installer Logging
As mentioned previously, the Installer supports extensive logging. This is by far the best tool for troubleshooting Installer issues and the time spent understanding the log files will be repaid many times over when problems arise later. At the least logging should be used throughout testing. You are certain to be asked to provide verbose logs if you contact Microsoft support, so either have these ready or be prepared to reproduce your problem to gather some.
Rule 56: Use Virtual Environments for Testing and Support

Using virtual machines (VMs) for testing rather than real systems has a couple of advantages:

  • You can build a VM for each of the different systems in your environment and store these on a single server or workstation to be brought on-line when needed
  • With “undo”-style features of VMs you can easily revert to a clean, known state after each test
  • The same VM images can be used by testing and helpdesk staff

However, do note that if you think you have discovered a bug in the Installer you may be required to reproduce the problem on real hardware when you contact Microsoft support: Support policy for Microsoft software running in non-Microsoft hardware virtualization software

[Author: Richard Macdonald]

This posting is provided "AS IS" with no warranties, and confers no rights.

Comments

  • Anonymous
    July 28, 2006
    I think the testing rule is probably outside the topic (which I think is how to design an MSI setup that works, is robust, can be serviced, and won't come back later in some way to bite you).
    Given that there are 50+ rules, I'm wondering if they need categorising by priority rather than by area. Some are more important to follw than others, I think that's true to say.

  • Anonymous
    July 29, 2006
    The comment has been removed

  • Anonymous
    July 31, 2006
    The comment has been removed

  • Anonymous
    July 31, 2006
    I think you're mixing together rules for developers, rules for administrators wanting to make use of Windows Installer technology to deploy applications, and rules for administrators wanting to repackage applications built with older technologies. The first and second groups don't really overlap, while the third overlaps somewhat with both. I think it might be better to have a set of rules for developers, a set for administrators, and point the third group at both sets, with a few extra rules for them.

  • Anonymous
    July 31, 2006
    I like the idea of the rules for everyone being in the same place. At least then there should be less ambiguity in the business about where to get your help and advice from.

    On the subject of "categorising by priority rather than by area", I agree that a by area approach seems sensible and by priority could lead to a ruckus, but how about marking each rule as 'strict' or 'preferred'?

  • Anonymous
    September 18, 2006
    Is there any special ICE warnings that MAY be ignored? I am working on an msi package and I have a number of ICE 33 warnings that seem to result from some merge modules that I integrate in my msi. I have no errors, but I have a series of warnings such as:

    **********************
    Reg key Global_Controls_MSADODataOcx_r17.7EBEDD1B_AA66_11D2_B980_006097C4DE24 is used in an unsupported way. This key need not be written as it already exists in the system. ice33.html Registry Registry Global_Controls_MSADODataOcx_r17.7EBEDD1B_AA66_11D2_B980_006097C4DE24
    Evaluation: ICE33
    *****************************

    I am still fairly new to windows installer and I am definitely taking rule# 1 seriously. However, in case someone has a clue how to sort this out, let me know.
    Cheers,
    Moses

  • Anonymous
    September 18, 2006
    Well, here we are with the last part in this series, where we'll be looking at security in relation to...

  • Anonymous
    October 04, 2006
    PingBack from http://929.hu/blog/?p=113

  • Anonymous
    October 16, 2006
    Windows Installeriga on küllaltki keerukas algajal otsi lahti teha. Mul endalgi on üks installer käsil ja tänu keerukamale funktsionaalsusele, mida see peab pakkuma, tuli ka rohkem süveneda Windows Installeri hingeellu. Leidsin internetis ringi tu...

  • Anonymous
    January 25, 2008
    PingBack from http://softwareinformation.247blogging.info/windows-installer-team-blog-tao-of-the-windows-installer-part-1/

  • Anonymous
    January 25, 2008
    PingBack from http://softwareinformation.247blogging.info/windows-installer-team-blog-tao-of-the-windows-installer-part-5/

  • Anonymous
    April 01, 2008
    PingBack from http://copyrightrenewalsblog.info/windows-installer-team-blog-tao-of-the-windows-installer-part-5/