Tests aren't methods!
In the feedback section of one of my blogs, Aaron Junod wrote:
“In writing this, I just realized something. I think someone should draw a line between writing tests, and actual program code, and then offer different options based on that. They are totally diff coding paradigms, and really merit diff behaviors not only in the compiler, but the IDE.. Hmm.. think I gotta think about this more.”
I told him I wasn't going to respond and I'd let Jay get a chance to voice his opinion on this, and he has! (Hurrah!) The issue arose when dealing with our frustration that NUnit forces both your TestFixture class (which is unnecessary) to be public (which is unnecessary) and requires that all tests in that class (which is unnecessary) to be public (which is unnecessary). Wow. That's a whole lot of unnecessary stuff. What irked us was that when we were trying out the new VS Team System tools, they also had these restrictions. Jay inquired into the reasoning behind these decisions and raised several good points for why this restrictions was a “Bad Thing”. They were:
- Nothing about the concept of a unit test has anything to do with a method. A unit test is just code that runs and tests something for you. We package it in a method because a method is a convenient way of naming something and packaging a set of statements together. However, a method carries unnecessary overhead that just isn't necessary in a test. The concept of it having accessibility, input or output parameters is just meaningless and unnecessary. Imagine if a unit test required you to take an integer as an argument. It would be pointless and would just burden the programmer. Similarly, by forcing a test to be contained in a method which is public you are unnecessarily burdening the programmer. This makes the test visible to people interacting with the system. Which is completely unnecessary. The test should only exist so that the engine can run them and report the results back. Of course, if a user wants to make his tests publicly accessible, he can. But he should not be forced to.
- Forcing the tests to be public goes against the XP philosophy of “do the simplest thing that could possibly work”. In order to have this restriction someone had to go out of their way to prevent it from happening. Why would you ever do that? Why do extra work in order to limit what the customer wants to be able to do? If you listen to the new Office System commercials, they espouse the philosophy of “Do more with less”. But here we are doing the opposite. We are doing extra work to make sure that you can't do something.
- The Team System software isn't supporting a completely reasonable coding style. With the team system method you will end up with a loosely couple system where you have “class MyClass” and a “public class MyClassTests”. Jay wants to code in the manner of “class MyClass{ class Tests {} }” so that he has tight coupling between the class and the code that tests that class. That way if he moves the class around, the tests stay with it. If he renames the class he doesn't have to worry about renaming the test class, and he doesn't have the redundancy of “MyClass” specified twice.
Note: Jay is not saying that everyone should test in his style. He is saying that both styles can be supported with less work on the Team System team.
I'd like to know your thoughts on this so I can take back some feedback on what the community thinks. I would appreciate a discussion on why you think it's a good or bad idea to force tests to be public. Addressing all of Jay's issues would be very useful, and if you can introduce points as to why limiting what the user can do here is a good thing, then we'd like to here them. Remember, jay is not saying “force the test to be private”, he's saying “support this behavior which doesn't prevent anything that a user can currently do with the system”
Thanks!
Comments
- Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
AT: I never said that you shouldn't have public tests. I said that you shouldn't require public tests. - Anonymous
July 01, 2004
Martin: I agree. There is no reason to make create your tests as methods, and you do not have to do so. However, currently, a method is simply a convenient container for the test. However, that container need not be public.
As you stated a test is a test irregardless of visibility.
You can currently write tests in any language, but the current ones supported directly in team system are any .net language.
I all agree with testing blocks of code. I was having adebate with neil about that the otherday. If you really think of everything as an object, then a block is just another object that you'd like to be able to test. - Anonymous
July 01, 2004
AT: "Do you believe if you test your business objects for cash balance 100USD and 200USD you must go to source code, change value and recompile to test for 200.000.000.000USD ?? Or you must be able to do this using config file or even more - flexible UI ? "
No. I would do this:
[Test]
void TestBusinessObject() {
TestCashBalance(100);
TestCashBalance(200);
}
void TestCashBalance(Dollars dollars) { ... }
But again, there is no reason for the test to take any parameters, return any value, or be public :-) - Anonymous
July 01, 2004
AT: "2. What's wrong with public ? "
Nothing. As I stated. Public is fine. The question is "what's wrong with private" :)
You say: "IMHO, correct Unit test code must looks like : [TestMethod] public"
Why? - Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
I'm not at all keen on shipping tests with the product. I do believe that the tests should be run against the released binaries. Which means that my tests always go in a separate dll. It also means that any code that I want to test must also go in a dll because the IDE does not allow exe's as references.
I see no reason that tests be public, the test methods are loaded by reflection with full trust so it makes no odds on their protection level.
To me the idea of nested classes, or worse still, code in comments justs adds clutter to the code.
I do think that test attributes could be extended to allow for special setup methods, parameters, etc.
[TestFixture]
class addTest : Assertion
{
// default setup
[Setup(true)] void fullInit(){}
[Setup] void smallInit(){}
[TearDown()] fullTear(){}
// default tear down
[TearDown(true)] anotherTear(){}
[Test("fullInit")] void testAdd1(){}
[Test("smalllInit", "fullTear")] void testAdd2(){}
[Test] void testAdd3(){}
// load the resource into the
// parameter before calling the method
[Test, Param("ds", "resource.DsTestAdd4")]
void testAdd4(DataSet ds){}
} - Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
The comment has been removed - Anonymous
July 01, 2004
I find this interesting.
First, for quite some time, I have been embedding my tests within my class (not nested, just #ifdefed) and I have been testing private methods of my classes. Recently I have been reading Jim Newkirk and others recommendation to only test public interfaces of classes, so I started moving my tests to a separate fixture class but still within the same file as my class under test.
I agree with Jay. I like the locality and I guess the coupling, though I had not thought of it that way. In fact I like testing my private methods. In some of the places where I have to plug into/extend a frame work, I find it difficult to do only test public interfaces. I am trying to code faster and more reliably, so I don't mind if a few of my tests on private functions break as I change my understanding, refactor, or expand my needs.
If you drop the test fixture attribute, then you would have to examine every class in detail looking for tests, not a bad thing, but certainly it would slow you down a bit loading the test engine. One could certainly make a strong argument that most if not all classes should have tests, so I look at this as a mixed blessing.
I have trouble imagining a system where your tests are not encapsulated in methods. Are you looking for another set of attributes that would define your tests? Kind of like eXtensible C# http://www.resolvecorp.com/Products.aspx? (By the way I really like their work. Declarative asserts are great by me.) - Anonymous
July 01, 2004
When someone requsted exception filters in C# you replied "show me an example why this is useful". What's your example of a useful private test?
Sometimes it is preferable to let developers do more work to restrict the users. Another related example comes from a blog post (of ericgu, I think): non-public fields in a class should be protected only if you can think of a useful scenario in which a derived class needs access to them, otherwise they should be private. I guess it is all about being "defensive". - Anonymous
July 01, 2004
I'll preface by stating I'm an NUnit noob.
Like Adam, I don't like including test code with a shipped product. I see it as code which should never run out in the wild. The only justification I can see for doing so is in the possibility these test will need to run on the client machine (which I don't think should happen in the first place). Why would the client need to run the unit tests? Because they are having a problem? If the client really must run the unit tests, this can be mitigated by shipping them a test exe to run the same tests. But even then, you're only proving your unit tests run successfully on this machine. The problem probably occurred because the unit test were deficient in the first place. In which case, the problem still exists on the client's machine.
Yes, I'm generalizing the situation some what. I guess I'm in a similar boat as Radu. I haven't seen a clear need to include these tests in the shipping assembly.
I do agree with the points about public vs private test methods, though. It doesn't make any sense why these methods must be public. - Anonymous
July 01, 2004
Cyrus:
a) Can you provide example of test that must be marked as private ?
b)
"[Test]
void TestBusinessObject() {
TestCashBalance(100);
TestCashBalance(200);
}"
Bad ... Really bad... Test run must give you as much as possible information about all failures. Testing both 100 and 200 in same method is wrong. Once 100 failed - you will get no information about test results for 200.
Instead of
void TestBalance100() { TestCashBalance(100); }
void TestBalance200() { TestCashBalance(200); }
You must be able to specify input params in config files or UI, so can play with test data without source edit/recompile.
For example I would like to be able test email address validator for different emails or long account creation/money transfer/deletion process. I must be able to pass email address or account number using nice UI. Once I will find inputs that fail I will add them to set of automated test runs.
In addition to fully-manual and fully-automated testing you must be able to do semi-automated testing with manual input of params.
c) Replace #ifdef DEBUG -> #ifdef TESTING ;o)
adam:
d) What the point of fullTear/littleTear specifications as attribute if you can create more flexible code like this one:
--
using(TinyInit(configParams)) {
// Do magic
}
--
Using attributes for Init/Cleanup do not allow you to pass params. For example if you use Init with 10-items common array (used by all tests) it will use it forever, you will be unable to rerun test/tests with array initialized by 1 or 100 items.
I hope you understand my concern - Anonymous
July 02, 2004
The comment has been removed - Anonymous
July 02, 2004
Jim:
"If you drop the test fixture attribute, then you would have to examine every class in detail looking for tests, not a bad thing, but certainly it would slow you down a bit loading the test engine. One could certainly make a strong argument that most if not all classes should have tests, so I look at this as a mixed blessing. "
I'll do some perf tests. I'm pretty confident that that could happen iwth negligible overhead.
"I have trouble imagining a system where your tests are not encapsulated in methods. Are you looking for another set of attributes that would define your tests? Kind of like eXtensible C"
That's simply because a method is such a useful container for executing tests. However, i can imagine weird things with attributes used. Or declarative tests through some sort of XAML system. However, I'm not saying "i don't think methods should be used for tests" just that "you shouldn't think of a method as a test" - Anonymous
July 02, 2004
Radu: A useful private test:
The exact same use of the public test without exposing it to all callers :-)
"I guess it is all about being 'defensive'. "
You stole the words right out of my mouth :-)
private test allows me to test and be defensive, while tightly coupling tests to the code they test. - Anonymous
July 02, 2004
Ron: Thanks for the input. Just to clarify. We are not saying "tests must be shipped with release software" Merely, developers have that choice. And if we've chosen to ship that way, then it should be supported.
An argument for it is TrustWorthComputing. By showing our tests to our users we achieve yet another level of transparency to help convince people that we do take safety and security seriously. - Anonymous
July 02, 2004
The comment has been removed - Anonymous
July 03, 2004
Adam and Cyrus.
Note that in VS2005 it's possible to add a reference to an EXE. - Anonymous
July 03, 2004
Kevin: Whoa! No way!! He's right! I just tried in C# express. I'm so happy. Oh so happy! - Anonymous
September 01, 2005
Your blog is very interesint - Anonymous
May 29, 2009
PingBack from http://paidsurveyshub.info/story.php?title=cyrus-blather-tests-aren-t-methods