Leverage a Risk-Based Approach to Application Compatibility Testing

Once upon a time, application compatibility projects usually followed these basic tenets:

1.)     Find all the applications being used.

2.)     Rationalize as much as possible.

3.)     Thoroughly test as many as possible.

4.)     Repackage all the applications.

Depending on the size of the portfolio and the number end-users, these projects lasted from 9 to 12 months on average – longer for large organizations. Each project had a definitive start date and finish date (which often slipped.) The amount of time it took along with potential slippage of dates was never a major issue when the average operating system migration took place every 5 to 7 years.

Times have changed. Shadow IT has creeped into the enterprise and innovation within this world has created a desire for faster feature delivery. Combine that with the ongoing agility of cloud products beginning with minimum viable products that continuous expand their feature set, and the traditional delivery of Windows does not make the most sense going forward for most enterprises. Like Office, Azure, and Intune, the Windows service delivery has evolved with WaaS (Windows as a Service.) Instead of drastic operating system updates every few years, more gradual feature updates occur on a semi-annual basis.

This warrants a paradigm shift in the world of application management – especially managing compatibility between these gradual operating system updates. The classic project management method of application compatibility just will not work because it 1.) Takes too long and 2.) is often littered with unnecessary work and redundancies.

A Risk-Based Approach

First introduced in 2016 by Chris Jackson, the recommended approach to application compatibility going forward is a risk-based, ongoing, iterative one. The iteration part is easy to grasp because it basically looks on paper as if there will always be some sort of application testing going on, be it for the current release for some applications, or the next release for the more critical applications. The primary purpose of adopting a risk-based approach is to ultimately determine how and when you will test an application for compatibility with the next feature release. In the case of WaaS, while the critical applications are still thoroughly regression tested, the others might be relegated to simple smoke (ILT – Install-Launch-Tested) or automated testing or even reactive testing (test in Pilot or do not test at all.)

Solutions such as Windows Analytics (or partner solutions) can automate and reconcile supportability for a vast amount of commercial vendor (COTS) applications. This can cut down on the time it takes to inventory and rationalize applications. In addition, applications that have been verified as adopted or supported can be relegated out of the critical space in many situations. But what about the rest of the portfolio? How do we get an accurate assessment of how important it will be to test that application?

One recommended approach is to adopt a very common risk management practice of leveraging a simple x/y graph where the variables are impact of regression to the business and the probability of regression. The higher of each, the more critical the application and the more necessary it will be to regression test that application. The probability of regression is rooted in history as well as application composition. How an application impacts a business or organization will vary especially across different industries so there is not really a cookie-cutter one-size-fits-all approach to calculating an application’s impact to the business.

It will be a potentially difficult process at first yet it will be one that, if implemented properly, will be a permanent risk profile for each application’s lifecycle and adjustments to risk profiles will only need to be managed as new applications are introduced and old ones are retired.

Thoughts on Repackaging

Bear in mind that for the majority of WaaS Feature Update scenarios, IPU (In-place Upgrades) will be the approach organizations will adopt if they are on a cadence of upgrading every 6-12 months. Operating system deployment images and provisioning packages will still be updated and maintained for new devices, but for the existing devices, the applications will already be installed, and the operating system will be only experiencing a minor in-place upgrade. In most cases, there will be no reason to reinstall and there will be no reason as well to repackage an application unless the purpose of repackaging is to modernize the application for newer delivery mechanisms such as leveraging the Desktop Bridge to bring an application to the Windows Store.

The Tenets are Evolving

As the deployment and servicing of Windows has evolved so have approaches to managing application compatibility. What we are finding that works well for customers who are able to stay current with Windows and Waas, is an application compatibility management approach that adopts these tenets:

1.)     Leverage tools and telemetry to discover and inventory, and even rationalize.

2.)     Rationalize into risk profiles.

3.)     Test according to risk profiles.

4.)     Don’t repackage applications for the sake of repackaging.