GUI schmooey
I answered a few questions this week about automating GUI tests. One question was about recommendations for GUI automation tools for non-coders, and the other was about how much time to spend on the GUI in an MVC (model-view-controller) application.
The answers were easy. In the first case, I said that they weren’t going to get ROI from the effort, and they should just test the GUI manually. In the second case, I suggested that they do all of the automation ignoring the view/GUI, and test the GUI manually.
I could expand an entire post on why I gave those answers, but it doesn’t matter. I’m going to go out on a limb and make the following statement.
For 95% of all software applications, automating the GUI is a waste of time.
For the record, I typed 99% above first, then chickened out. I may change my mind again. The point is that I think testers, in general, spend too much time trying to automate GUIs. I’m not against automation – just write automation starting at a level below(*) the GUI and move down from there. I’m not saying that you shouldn’t test the GUI at all – I just don’t see why you wouldn’t want to test it manually, and get people knowledgeable in user experience to help. I just think that in most cases we are wasting our time when we try to automate GUIs, and wonder if anyone has the guts to stop.
* What I beam by a “level below the GUI” is automation that works with IAccessible or an object model rather than interacting with UI elements directly.
Comments
- Anonymous
September 18, 2008
PingBack from http://www.easycoded.com/gui-schmooey/ - Anonymous
September 18, 2008
The comment has been removed - Anonymous
September 19, 2008
I'm writing a post for testers - whether they're in IT or not. Regardless of what you're testing, chances are that if you're writing automated GUI tests that you're not being productive.I have seen good UI automation that tests with button clicks. It can be convenient to run tests like this sometimes, but in most cases, it just isn't worth the effort for an all out investment. - Anonymous
September 19, 2008
Alan,totally agree.. 99% might be too bold, but I'm with ya at 95% :) - Anonymous
September 20, 2008
I wanted to expand on the main point of my last post – specifically on what a “layer below the GUI” means. - Anonymous
September 21, 2008
From my experience, GUI automation will work if you write the maintainable scripts. Otherwise you need to redo the whole thing for every release which is waste of time and effort. And automation won't find any new bugs but will ensure that new code didn't break the old features.Rajasankarerrajasankarchem at gmail dot com - Anonymous
September 21, 2008
My experience suggests that while we should be automating at object interface level, as Shrini suggests, the option simply isn't there for many applications, and where it is, it is not accessible to the tester. While it may be inefficient to automate through the GUI, in many instances it is still the most pragmatic approach and can work very well with a good framework. Saying that it is a waste of time in 95% of cases is IMO very far from the truth, not to mention pretty brash as it assumes that you are aquainted with those cases. Now if you'd said that for most applications that are still at design stage, there is a much better way of planning and executing automated testing than via the GUI, I'd be with you all the way.Ideally, I'd like to have a recorder that recorded at interface object level rather than GUI level, and have pestered a couple of the COTs vendors to allow hooks into their apps to do this. For many graphically intensive domains, such as CAD, GIS, etc... the user interface is still the best place to generate the test cases. Not much response so far... - Anonymous
September 21, 2008
The following discussion over on SQAforums may also be of interest;http://www.sqaforums.com/showflat.php?Cat=&Board=UBB34&Number=412420&Searchpage=1&Main=412291&Words=testable++smacl&topic=&Search=true#Post412420 - Anonymous
September 22, 2008
The comment has been removed - Anonymous
September 22, 2008
I stand by my statement. IN fact, I shuddered a bit when Shane used the word "pragmatic" to use fragile tools.I have written automated gui tests for entire applications using some of these frameworks, and they all break down in future revs of the application, or when the underlying application changes.If your application isn't designed well (well designed applications never put the logic in the GUI - if you're stuck with one of these applications, you're going to have bad quality regardless of whether you write automated tests or not, so why bother? - Anonymous
September 22, 2008
So if you're presented with an application that has been written without an object level automation interface, you deem it badly designed, and hence not a candidate for automation? Certainly not in my experience.Like yourself, I have written automation frameworks to drive testing through a GUI, which years later are still running just fine. One of the principal purposes of the framework in this instance is to abstract the GUI to logical object style interfaces, such that the tests are not fragile. It might not be as robust in most cases as using a COM interface, but that doesn't render it either unworkable or a waste of time. You simply have to be aware of, and cater for, the pitfalls. It is also worth noting that object based interfaces can change from release to release, breaking object based testing. It is bad programming, sure, but what is the point of testing if not to pick up on bad programming. Similarly, the GUI shouldn't contain any logic, or work differently from other interfaces, but it isn't always the case. Good designs can be badly implemented.Don't get me wrong, testing through a programmatic or object based interface is way superior if its available. If it isn't, IMO, GUI is a viable alternative in many cases. - Anonymous
September 22, 2008
To me, part of good testing is having sufficient early influence in the product to ensure I never had a poorly designed application plopped on my lap. However, I realize that for many testers, this isn't a possible solution (see my post on the Test Test for more information).The point I didn't make strong enough is that whiole, of course, it's possible to drive an application purely through GUI automation, this is one are of automation where I worry that the time investment isn't worth the effort vs. just performing manual testing on the GUI in the first place. - Anonymous
September 22, 2008
I don't think that Shane is referring to record and playback tools. I don't think that any of these tools are all that useful, for the reason you state above. In general, the application will always break.What Shane does mention is havign a solid framework. I can completely agree with this. I actually designed the majority of my most recent project at the GUI layer. However, I used my experience in automation to avoid the pitfalls that often plague automation projects. In the end, we've managed to create a project that has actually just successfully bridged two separate versions of the software.This automation suite was built entirely through the GUI, using manual test cases, with the exception that we understood that things happening at this layer were likely to change. We used a record and playback tool, but one with a robust scripting engine also. In the end we probably used the record and playback 1% of the time. Maybe for identifying objects, maybe to see how the scripting engine wants us to express something.In the end, we had a substantial suite which included 3 separate layers, the global framework, the project framework, and the test cases. Built in a heirachal system, the global framework was designed to be portable to other projects. It can be used across both Windows and Java platforms (which was the requirement of our organization) and also acts as a communication medium with the automation tool. The project framework utilizes the global framework, but also implements simple elements related specifically to the project itself. At the test cases layer, I expect to see function calls over tons of scripting. When I do see scripting at the test cases layer, I question it.In the end, the project has survived the transition from version 1 to version 2 of the software, even though each test case is designed right off the GUI elements.So, maybe I misunderstand what you mean by testing at the GUI level, but in my experience, if you don't plan ahead then you will ultimately fail anyway. Sometimes, even the best laid plans will fail. However, I think that a poor plan will fail regardless of where you decide to put your chips. - Anonymous
September 22, 2008
The comment has been removed - Anonymous
September 22, 2008
I totally agree with you UIGuy. There are certainly limits to what I think should be tested. We haven't, and won't, automate 100% of our application, for the exact reason that you mention.What I'm trying to illustrate, possibly poorly, is that if we had modules X, Y, and Z, they might all function properly as standalone modules. We might test every possible combination on each of these modules, individually. However, even though we know what they are meant to do, without a GUI we're never really sure how these modules will interact. That is, of course, unless we are allowed source-level access to the modules. Then it becomes an organizational question of what level of access to automation staffers have?At the same time, it could be a matter of semantics. If we're talking about automating a GUI to verify the GUI layout, button location, etc., that's one thing. If we're talking about doing all of our automation within the layers underneath the GUI, then I think that is also lacking. Then there are X-Factors (I don't know why I always capitalize that) like how much of the app are you planning on automating? Are you being asked to automate test cases or automate a module? Do you have access to teh application source code? Are you treated as a developer or a QA person who is acting like a developer ( :P )?So is Alan wrong? Meh, probably not. I think that the number might be misleading to a lot of people only because I think that there are fewer companies who would be ok with you trudging through their source code than those who don't mind. Not to mention, I would expect that most companies would not allow juniors or even many seniors to work at this low a level. That's why they buy the tools to work on top of those layers. For many, the GUI layer will often be their only access to any low-level items for automation. Now, this might be my own assumption. Maybe I'm wrong. In all my time, though, I haven't granted someone full source-level access, even if I had it myself. - Anonymous
September 22, 2008
UIGuy - you'e posted in a comment exactly what I menat, but failed to communicate - thanks.