GUI Automation and ROI
It seems that many test automation efforts around the industry tend to focus on GUI automation, or automating functional tests primarily by manipulating GUI objects. In general, GUI automation tends to be a very expensive approach to test automation, and the automation efforts often end in failure or achieve less than satisfactory results.
The majority of automated tests at Microsoft are below the GUI; however, automated tests that manipulate GUI objects are quite useful within specific contexts. Unfortunately, many testers attempt to develop automated GUI tests way too early in the project cycle while the user interface design is still unstable. I guess the assumption is that constantly maintaining automated tests is somehow better then executing manual tests. But, in general, when the UI is in flux it is usually counter-productive and a loss of return to automate GUI level tests too soon in the development lifecycle.
Dan Mosley and Bruce Posey (Just Enough Software Test Automation) suggest that on average an automated test must run approximately 17 times in order to break even. But, this doesn’t imply that we break even if we simply run the same test on a daily build for the next 17 days. The presumption underlying the ROI after 17 runs of a test is that something changed in the build that is covered by that particular test, and so by executing that (regression) test we are providing important information (changes in the build did not destabilize that area) to the team. Steve Rowe also has an excellent blog post on Too Much Test Automation that you should consider reading, and Dustin Andrews also has an excellent blog post on getting great results from test automation with an 8-minute video.
The cost of test automation is never easy to figure out and is certainly not a straight forward comparison of automated time versus manual time. Comparing automation time versus manual execution time is an overly simplistic measure that rarely takes into consideration the design and the development time of the initial tests (or the overall costs of building and enhancing test frameworks or drivers), or the time required to identify false negatives, troubleshoot the cause, fix the problem, verify the fix (manually), and then check the source back in for the next run. (And, there are other tangible costs, and intangible costs such as loss of confidence that must also be considered in any cost model.)
So, in general any automation that requires constant maintenance is usually not cost effective, and the more our test automation throws false negatives the less our management team views automation as a viable resource to provide us with valuable and reliable information for improved risk analysis. Ultimately the decision comes down to how much perceived and measurable value a test has in providing important information for improved risk assessment and quality measurement.
Just because we can automate something, doesn’t always mean we should!
Comments
Anonymous
March 28, 2008
Hi, I agree with your statements and it would be much easier if you could skip the GUI in automation. Unfortunate it is for many applications "necessary evil" since errors in the GUI level directly affects the end user (even if the error itself is minor, most users often is not that happy about it). In addition, some applications has a lot of exclusive functionality/business logic in the GUI layer (not recommended from my point of view, but for example AJAX seems popular these days). So the trick is to find a cost effective way to do it. Good skills, tools and a robust framework will get you decent ROI I guess even for GUI automation. You could always considering what automation effort ratio that is appropriate for the different UIs in an application but at least I want to exercise all the UIs including GUIs in a new build to some degree automaticly even though my tests mainly aims at the business logic layer via Web Services or other "non-graphical" UIs. Regards, /StefanAnonymous
March 28, 2008
The comment has been removedAnonymous
March 28, 2008
Hi Stefan, Spot on! I didn't mean to suggest that GUI automation should be avoided, but that we need to be smarter about how and when to do it. I absolutely concur that GUI automation is a great approach to simulate end user interaction; especially if we model the UI, define use cases as actions, and use a model based testing approach to simulate an even greater number of scenarios. Good class design should separate the core logic and functionality from the GUI layer. But, of course it is easy to write poorly designed code, which is why there will always be jobs in testing. :-) As you say there are some automation frameworks that provide an abstraction layer to help 'ease the pain' or improve ROI a bit, but timing and balance are key.Anonymous
March 28, 2008
Hi Erich, The other day I read an article about how the industry spent so much time studying usability and getting consistency into the GUI of client applications, and then came the Internet and standardization and usability of the UI went right out the window. I guess there is a parallel with OOP design and some scripting languages such as AJAX. :-) This is not to suggest AJAX is a bad thing...it just complciates things a bit...and sometimes complication keeps things interesting, or at least gives us something to talk about. :-)Anonymous
March 28, 2008
The comment has been removedAnonymous
March 28, 2008
Hi Ian, Great points regarding the inexperience and poor practices, and that was the intent of my message. I suspect that many people simply attempt to automate from the GUI too soon in the project. However, I don't necessarily agree with your comment that [the GUI] "seems to most consistently produce useful infomration about bugs." I suspect most testing approaches over the past few years have tended to focus on testing from the GUI; however, there is a lot of evidence that demonstrates the single most effective means of identifying funcitonal and security issues is code reviews or inspections, and well designed unit testing can also consistently produce information about functional problems. I do agree that GUI testing is useful in identifying usability type issues, but a lot of functional issues could be identified much earlier in the process...of course it all depends on how much an organization is willing to invest in testing.Anonymous
March 28, 2008
I stand corrected. I should have qualified this with "at our company, where we don't have a system of code reviews." I'm still on the fence with about unit testing. It's got maintenance problems too. Could you point me to the evidence - some objective measures of each type of testing? I haven't seem much in the way of objective measures of unit testing ROI on the web. Personally, I'd love to go with more unit testing, however, I'm afraid that I want to because I like to code, not because it identifies a maximum number of important, fixable problems. I frequently suspect this is why developers like it, in general.Anonymous
April 02, 2008
As a quick data point, OneNote moved recently to a non_UI based automation system. Short story: everyone involved in the process (test, dev, and PM) concurred, spurious failures fell from (some random number between 10-50% per run) to <1%. Best quote from a tester: “I no longer get a knot in my stomach when I have to do automation investigations.” I feel justified with this decision and "losing" the UI based tests. After all, if I click the button to "make it green" and it is made gold instead, our dogfood use will hit that. We cna automate the functional test of making it green and gold separately. John Guin OneNote TestAnonymous
April 04, 2008
The comment has been removedAnonymous
April 04, 2008
http://blogs.msdn.com/imtesty/archive/2008/03/28/gui-automation-and-roi.aspx I.M. Testy says conciselyAnonymous
April 04, 2008
Hi Shrini, Agreed. In my opinion many people make too many incorrect assumptions about the value a good automation effort provides to an organization. I would also say that automated GUI testing can be useful in identifying certain types of usability issues. For example, missing key mnemonics, tab order, layout errors such as control overlap, truncation, etc. can be highly automated. (So, I guess I am that someone. :-) But, I also agree there are other aspects of usability testing such as the 'ease of use' perspective may never be automated, nor should it be.Anonymous
April 04, 2008
The comment has been removedAnonymous
April 04, 2008
The comment has been removedAnonymous
April 06, 2008
>>There is a common myth that test automation will reduce the product development lifecycle. However, this is a foolish assumption for 2 reasons. Here are my reasons...
- You would do lots of things "extra" in a automation cycle (automation setup, automation script failures/investigation/fix/rerun, analysis of results) that are not presenting in a pure manual testing cycle. Depending upon how good is your automation environment, Application under test and automation code - some times these extra items eat up time and net-net your overall cycle time with automation "might" be higher than pure manual testing time. Not withstanding the fact that any comparison between manual testing cycle and automation cycle are really vague and stupid.
- What would you do if automation test cycle throws up some interesting bugs and development struggles to fix them ... will you take your product to market faster than before? We need to understand every cycle of testing reveals a set of bugs that need to be addressed so there is some development/testing work to be done after every testing cycle. Will automation reduce or eliminate that time? Not likely - those tasks are out of purview of automation The fact that there is automation, you can gain some time and "accomplish" more testing but not faster time to market as there are other things to take care of anyway. I always say "with good automation, you will do more testing (accomplish more) than before". This has taken many IT managers by surprise. Some told me that they bought automation tool so that they can do less testing and believed the promise given by sales person of the tool vendor/offshore service provider !!! Funny but true Shrini
- Anonymous
April 06, 2008
The comment has been removed - Anonymous
June 07, 2008
做軟體測試的人, 大多會希望把所有的測試都自動化, 就算沒辦法全部 (事實上, 全部自動化在目前是不可能的...), 至少也要愈多自動化測試愈好, 畢竟, 自動化測試的好處很多, 使用得當, 絶對能大大幫助軟體測試人員的效率,