Tricky Trojans
Wayne writes seeking suggestions for dealing with simulacra of complex and complicated interfaces. While I usually want to do some end-to-end testing in order to see how the system as a whole functions, I generally prefer to do most of my testing at lower levels, banging on individual components, modules, or even classes and functions in isolation from each other. I find that the more I reduce the scope of my focus, the simpler verification tends to become, the larger the number and more varied models I can consider, and the more I feel confident that I am considering and covering most of the problems which are most likely to affect most of my customers. High Goodness.
Also, alas, the more likely I am to have to either chivvy another part of the system into a particular state or build my own simulation of another part of the system. High Pain.
In the best of situations building a fake component over which I have complete control is a trivial affair. Such was the case when I replaced a COM object by building my own second DLL which reimplemented the source interfaces and then registering my "Trojan" component over top of the source component. Bingo! - my own completely scriptable object which I proceeded to use to throw all manner of unusual conditions and errors at its host application. Similarly, on my current team I am building a fake version of my application which acts enough like the real thing to test our servicing stack in depth while bypassing the complicatedness of dealing with the actual application.
In the worst of situations building a fake component is a quaqmire. Such is the case in which Wayne finds himself. He would like to test an addin to Microsoft Visual Studio without actually running it inside VS. If you have ever attempted this yourself you know that the interfaces you must simulate are numerous, large, involved, and at times perplexing. Definitely not a trivial task.
And then comes the real humdinger: regardless of how (non)trivial duplicating the source interfaces is, there is always the risk that my simulated implementation will somehow behave differently than the real deal, in which case any tests I build on top of my counterfeit likely will mislead me into thinking that all is well when such is not necessarily the case.
This, like much else in testing, is a Hard Problem for which I do not have any clear answers or even guidelines. Because, you know, It Depends.
When faced with such a quandry, I compare my guesstimate of the work necessary to realistically-enough duplicate the interface against my guesstimate of the work necessary to test against the actual implementation. And then I ignore that information and choose whichever my gut tells me to do. <g/>
*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.
Comments
Anonymous
October 10, 2007
PingBack from http://www.artofbam.com/wordpress/?p=6881Anonymous
October 28, 2007
Oh boy! I dunno if I would even attempt the VS plugin testing outside VS by simulating VS thingy. Even simple apps become highly tricky when trying to simulate stuff - and this one sounds like a killer. I've found that it helps a lot if you are able to predict upfront that you need a good testability point to replace dummy implementations with the real thing for your testing. Having that will make the dev watch out before making a big complicated mess of the simpler pieces of app as well. But of course, that's a luxury you have only if you are not saddled with testing legacy code.