Living on the edge - testing without mocking (Part 2)

The (micro-) epic continues...

In the first part of this N-part series (where N may or may not equal 2), I thoroughly convinced you that while testing against mock systems is wise and all, it's always cool to also live dangerously and write/run unit tests against live systems at least every once in a while. In this post I'd like to continue this with another pattern that has proved useful to me in this thrilling world.

The secret wild side pattern

So you're a hip start-up programmer writing a super-cool machine learning system to figure out the best coffee blends you can make for any time of day (and you're of course reading this in 2018 when Microsoft has become cool with start-ups again - isn't it great?). C-Blenda relies on a web service that always gives the available coffee beans and their prices/characteretics in any city in the world:

 IEnumerable<BeanInfo> GetAllCoffee(Location location)

Now as a conscientious programmer, you abstract this out into an abstract base class (not an interface - a pet peeve of mine that I may expand on some day), with a concrete implementation that actually calls into the web service, and various mock implementations that you use in your tests. You then code up your brilliant implementation with awesome tests to go with it, for example this test that checks that you always give at most three recommendations for the morning (can't tax that caffeine- deprived brain too much in the morning):

 [TestMethod]
public void GoodNumberOfMorningRecommendations()
{
  var coffeeInfoProvider = new MockInfo();
  var recommender = new Recommender(coffeeInfoProvider);
  var ideas = recommender.TastiestBlends(Seattle(), Morning());
  Assert.IsLessThan(4, ideas.Count());
}

Your tests pass, you do the obligatory dance-around-the-office celebration, but now you crave more: your tests should also pass against real data. You want your tests to be prim & proper and run against mock implementations in the everyday world, but have the occasional wild night against live implementations.

Introducing two-face

Here is how I usually give a wild side to my tests: first I make the test class an abstract base class, with an abstract method to get the concrete implementation of the dependent system:

 protected abstract CoffeeInfoProvider NewCoffeeInfoProvider(); 

[TestMethod]
public void GoodNumberOfMorningRecommendations()
{
  var coffeeInfoProvider = NewCoffeeInfoProvider() ;
  var recommender = new Recommender(coffeeInfoProvider);
  var ideas = recommender.TastiestBlends(Seattle(), Morning());
  Assert.IsLessThan(4, ideas.Count());
}

I keep all of the actual tests in the base class, and I then create two thin derived classes that just implement this abstract method differently: the mock one that just returns a mock implementation:

 public class MockCBlendaTests : BaseCBlendaTests
{
  protected override CoffeeInfoProvider NewCoffeeInfoProvider()
  {
    return new MockCoffeeInfoProvider();
  }
}

And a live one that either creates a real implementation based on configured information, or skips the tests if not configured for that:

 public class LiveCBlendaTests : BaseCBlendaTests
{
  protected override CoffeeInfoProvider NewCoffeeInfoProvider()
  {
    var credentials = TestConfiguration.GetCoffeeInfoProviderCredentials();
    if (credentials == null)
      Assert.Inconclusive("Skipping live tests because we're not configured for it.");
    return new CoffeeInfoProvider(credentials);
  }
}

(See part 1 for my discussion of this TestConfiguration). And voila: I just need to enter my credentials to the web service in a config file whenver I want to test against a live system, the mock tests always run, and I can keep all the tests that don't care about which implementation they go against in one clear place.

Luckily all the test systems I've tried this against (NUnit, JUnit and MSTest) all have the necessary ingredients to make this work: they don't choke on abstract base test classes, and they have a way of skipping the test in the running of it.

This pattern has for the most part served me well. There are a couple of drawbacks though:

1. It doesn't enforce running the live tests regularly. Ideally it'd be coupled with a continuous build system that does run the tests with live credentials at regular intervals, otherwise we're at the mercy of my and my team's discpline to actually run them when needed (which of course is always an excellent idea to depend on my discipline...).
2. Visual Studio at least doesn't differentiate in its UI that much between the two versions of the tests: you sort of see two results for a test called BaseCBlendaTests.GoodNumberOfMorningRecommendations, which can be confusing on which one is the mock and which is the live. It's a bit annoying but it never hurt me too much.