How would you test a... C# code generator with Visual Studio Team Test
Introduction
I spent some time this week thinking about how to test a code generator that generates C# code and is itself written in C# (so it's once again all about managed code). At first I thought about how to verify the generator output directly. Since the output is C# code - which ultimately is just a bunch of text - one approach is to create a collection of base line outputs and tests that run the generator with different inputs and compare the outputs to the base lines which have been verified when then the tests were written. There are two major flaws in this idea: Verifying the base lines in this scenario will be a manual task and even small changes to the generator output will cause tests to fail even if the new code behaves exactly the same way then before. Another approach would be to actually analyze the generator output but at the very least this would require parsing C# and I'll leave that to people who enjoy (and know how to) build compilers.
And? How would you test it?
So, coming to the conclusion that verifying the output was not an option I started thinking about the next best thing: Writing tests against the generated code and thus implicitly testing the generator. Makes sense; however, the question remains how to do that. One could run the generator manually for all interesting inputs, add the outputs to a test project and then write tests for them. This is actually worse than the approach with base lines since now there are a bunch of source files to update every time the generator output changes but the tests won’t automatically break when that happens. What I really wanted was running the code generator with every test pass. That said - and without further ado - let's look at some code. Consider a C# class library project containing the following "generator":
public static class CodeGenerator
{
public static string Generate(string name)
{
return @"
public static class PersonalizedHelloWorld
{
public static string Name { get { return " + (name != null ? "@\"" + name.Replace("\"", "\"\"") + "\"" : "null") + @"; } }
public static string GetHelloWorldMessage()
{
if (Name == null)
return ""Hello World! Hello anonymous user!"";
else
return ""Hello World! Hello "" + Name + ""!"";
}
}";
}
}
I say "generator" because obviously it doesn't do anything useful and I just wrote it for demonstrating my approach to testing generators which is also why I'm not using the CodeDOM. Now, let's think about the output we get for the string John "\/\/" Doe as the input. What I would normally do to test that code is create a new test project and write a test class like this one:
[TestClass]
public class NonemptyNameTests
{
[TestMethod]
public void NameProperty()
{
Assert.AreEqual<string>(@"John ""\/\/"" Doe",
PersonalizedHelloWorld.Name);
}
[TestMethod]
public void GetHelloWorldMessageMethod()
{
Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",
PersonalizedHelloWorld.GetHelloWorldMessage());
}
}
Apparently I can't do that because the class this code calls into does not exist when the test assembly is built. So instead I add a new source file to the test project with the following code and set its build action to Embedded Resource.
public static class NonemptyNameTests
{
public static void NameProperty()
{
Assert.AreEqual<string>(@"John ""\/\/"" Doe",
PersonalizedHelloWorld.Name);
}
public static void GetHelloWorldMessageMethod()
{
Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",
PersonalizedHelloWorld.GetHelloWorldMessage());
}
}
The idea is to compile this code together with the compiler output for the input John "\/\/" Doe during test execution. In order to do this I introduce a little helper method - we need to run and test the generator multiple times with different inputs after all.
public static class TestCompiler
{
private static string unitTestFrameworkAssemblyPath = typeof(Assert).Assembly.Location;
public static Assembly Compile(string generatedCode,
string generatedCodeFileName,
string dynamicTestCodeResourceName,
string dynamicTestCodeFileName,
string assemblyFileName)
{
string dynamicTestCode;
using (StreamReader reader = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(dynamicTestCodeResourceName)))
dynamicTestCode = reader.ReadToEnd();
File.WriteAllText(generatedCodeFileName, generatedCode);
File.WriteAllText(dynamicTestCodeFileName, dynamicTestCode);
CompilerParameters parameters = new CompilerParameters();
parameters.GenerateExecutable = false;
parameters.GenerateInMemory = false;
parameters.OutputAssembly = assemblyFileName;
parameters.ReferencedAssemblies.Add(unitTestFrameworkAssemblyPath);
#if DEBUG
parameters.IncludeDebugInformation = true;
#else
parameters.IncludeDebugInformation = false;
#endif
CSharpCodeProvider codeProvider = new CSharpCodeProvider();
CompilerResults results = codeProvider.CompileAssemblyFromFile(parameters,
generatedCodeFileName,
dynamicTestCodeFileName);
#if DEBUG
foreach (string message in results.Output)
Debug.WriteLine(message);
#endif
return results.CompiledAssembly;
}
}
All Compile() does is take a couple of names and generated code and return a new assembly. A key characteristic of this implementation is that the source code as well as the assembly are written to the file system which allows further analysis in case the code does not compile or test execution fails (the code provider does write code passed in as strings to the file system automatically - but it writes it to a temp directory and it is very quick to delete those temporary files as soon as the compiler finishes execution). Now we can add what VSTT/MSTest.exe will consider to be the test class:
[TestClass]
public class NonemptyNameTests
{
public const string AssemblyFileName = "NonemptyNameTests.dll";
public const string GeneratedCodeFileName = "NonemptyNameTests.target.cs";
public const string DynamicTestCodeFileName = "NonemptyNameTests.dynamic.cs";
public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NonemptyNameTests.dynamic.cs";
private static Type dynamicTestType;
[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
string generatedCode = CodeGenerator.Generate(@"John ""\/\/"" Doe");
Assembly assembly = TestCompiler.Compile(generatedCode,
GeneratedCodeFileName,
DynamicTestCodeResourceName,
DynamicTestCodeFileName,
AssemblyFileName);
dynamicTestType = assembly.GetType("NonemptyNameTests");
}
[TestMethod]
public void NameProperty()
{
dynamicTestType.GetMethod("NameProperty").Invoke(null, null);
}
[TestMethod]
public void GetHelloWorldMessageMethod()
{
dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);
}
}
Note that the only real work this class does is run the code generator, build another test assembly with the generator output and the test code we embedded as a resource and store a Type instance for the new test class. Even more important, it does so in the class initialization method meaning that if an exception is thrown during the generation or the compilation step it will cause all tests in this class to fail. Furthermore, for every test to be added you need to add two methods: One in the class with the actual test logic which gets compiled at runtime and one in the class above that just calls the first one using reflection. And that's actually all we need to make this work. What's left for this post is a non-exhaustive list with the advantages and disadvantages of this approach.
Advantages
- No manual steps
Running the code generator is an integral part of the test pass (which I considered to be a requirement) and so is compiling the code. - Unit test framework friendliness
Just to be clear - it doesn't matter if you consider any of this to be unit testing or not. The point is that you can use any unit testing framework you're already using anyway (like VSTT/MSTest.exe, NUnit, [insert favorite unit test framework here] ) as long as it allows defining an initialization step for a test class. - Agnostic to changes not affecting behavior
Tests do not have to be updated in case of changes to the generated code unless its behavior changes (meaning changes in comments, white space, formatting and refactoring without API changes are free). - Test assembly is self-contained
In terms of test execution and from VSTT's/MSTest.exe's point of view there is absolutely no difference between the test project I described in this post and a "normal" one. All you need to run the tests is the assembly with the code generator (and of course its dependencies) and the test assembly (and its dependencies). This is also the reason why I chose to embed the test code to be compiled during text execution as a resource instead of keeping it as an external file.
Disadvantages
- Can be difficult to write actual test code
Since the test classes actually doing the work are embedded as resources and refer to types not available when building the test project it can be more difficult to write them since IntelliSense and some other IDE features are not available unless you keep switching the build action from Embedded Resource to Compile and back and - if that is not enough help - temporarily copy the generator output into one of the test project's source files. - Can be difficult to debug
This applies primarily to the compilation failing since you have to rely on the raw compiler output to figure out why it failed. Failing tests are less of a problem because the debugger will open the source files compiled at runtime when it is configured to break when a first chance exception occurs. However, this can also be confusion at times because one must refrain from editing the file the debugger opened and fix issues in the original source file or generator code respectively. Also, since the test methods marked with the TestMethod attribute invoke the actual test methods through reflection, failing asserts will cause the test to fail because of a TargetInvocationException instead of the failed assert. Luckily, the original assert information is available through the InnerException property. - Cannot verify comments, formatting, etc.
One of the strengths of this approach is also a weakness. As the tests never look at the generated source code there is also no way to easily hook in steps to check things like comments or formatting.
Full annotated sample source code
/**********************************
* Project: CodeGenerator *
* File Name: CodeGenerator.cs *
* Build Action: Compile *
**********************************/
public static class CodeGenerator
{
// Useless code generator method for
// demonstration purposes only.
//
// DO NOT USE IN PRODUCTION ENVIRONMENTS!
public static string Generate(string name)
{
return @"
public static class PersonalizedHelloWorld
{
public static string Name { get { return " + (name != null ? "@\"" + name.Replace("\"", "\"\"") + "\"" : "null") + @"; } }
public static string GetHelloWorldMessage()
{
if (Name == null)
return ""Hello World! Hello anonymous user!"";
else
return ""Hello World! Hello "" + Name + ""!"";
}
}";
}
}
/************************************
* Project: CodeGeneratorTests *
* File Name: TestCompiler.cs *
* Build Action: Compile *
************************************/
using Microsoft.CSharp;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;
public static class TestCompiler
{
private static string unitTestFrameworkAssemblyPath = typeof(Assert).Assembly.Location;
// Use by test class initializers to compile the generated
// code together with actual test code into an assembly
// during the test pass.
public static Assembly Compile(string generatedCode,
string generatedCodeFileName,
string dynamicTestCodeResourceName,
string dynamicTestCodeFileName,
string assemblyFileName)
{
string dynamicTestCode;
// Reads the actual test code from a resource
// stream in the static test assembly.
using (StreamReader reader = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(dynamicTestCodeResourceName)))
dynamicTestCode = reader.ReadToEnd();
// Writes the generated code as well as the actual
// test code from the resource stream to the file
// system in order to allow debugging the actual test code.
File.WriteAllText(generatedCodeFileName, generatedCode);
File.WriteAllText(dynamicTestCodeFileName, dynamicTestCode);
// Parameters for the code provider telling it to generate
// a DLL and write it to the file system.
CompilerParameters parameters = new CompilerParameters();
parameters.GenerateExecutable = false;
parameters.GenerateInMemory = false;
parameters.OutputAssembly = assemblyFileName;
// The paths to/names of all assemblies containing types
// referenced in the code to compile need to be added to
// the CompilerParameters object ReferencedAssemblies collection.
parameters.ReferencedAssemblies.Add(unitTestFrameworkAssemblyPath);
#if DEBUG
// Need to make debugging easy in case a class initializer or
// test fails by having the compiler include debug information
// and create a PDB file.
parameters.IncludeDebugInformation = true;
#else
parameters.IncludeDebugInformation = false;
#endif
// Creating code provider instance and invoking C# compiler.
CSharpCodeProvider codeProvider = new CSharpCodeProvider();
CompilerResults results = codeProvider.CompileAssemblyFromFile(parameters,
generatedCodeFileName,
dynamicTestCodeFileName);
// Writing full compiler output to trace listeners in debug
// builds to preserve all errors in case the generated/dynamic
// test code does not compile.
#if DEBUG
foreach (string message in results.Output)
Debug.WriteLine(message);
#endif
// Returning new assembly to caller.
return results.CompiledAssembly;
}
}
/**************************************
* Project: CodeGeneratorTests *
* File Name: NonemptyNameTests.cs *
* Build Action: Compile *
**************************************/
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Reflection;
// Test class verifying the code generator
// by testing its compiled output.
[TestClass]
public class NonemptyNameTests
{
// File and resource name constants for this test class.
public const string AssemblyFileName = "NonemptyNameTests.dll";
public const string GeneratedCodeFileName = "NonemptyNameTests.target.cs";
public const string DynamicTestCodeFileName = "NonemptyNameTests.dynamic.cs";
public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NonemptyNameTests.dynamic.cs";
private static Type dynamicTestType;
[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
// Running the code generator we want to test
// with one set of the inputs we need to cover.
string generatedCode = CodeGenerator.Generate(@"John ""\/\/"" Doe");
// Compiling the generated code together with the test code
// for the chosen generator input into a new assembly. Note
// that if the generator produced code with syntax errors
// all tests in this test class will fail because of an
// exception in the class initializer.
Assembly assembly = TestCompiler.Compile(generatedCode,
GeneratedCodeFileName,
DynamicTestCodeResourceName,
DynamicTestCodeFileName,
AssemblyFileName);
// Storing a Type instance for the test class type in the new
// assembly to be used by the test methods in this class.
dynamicTestType = assembly.GetType("NonemptyNameTests");
}
// The following two test methods do not contain actual
// verification code. Instead they call methods in the new
// assembly using reflection.
[TestMethod]
public void NameProperty()
{
dynamicTestType.GetMethod("NameProperty").Invoke(null, null);
}
[TestMethod]
public void GetHelloWorldMessageMethod()
{
dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);
}
}
/**********************************************
* Project: CodeGeneratorTests *
* File Name: NonemptyNameTests.dynamic.cs *
* Build Action: Embedded Resource *
**********************************************/
using Microsoft.VisualStudio.TestTools.UnitTesting;
// Class containing the actual verification logic for a specific set
// of code generator input. This code is not compiled into
// CodeGeneratorTests.dll but embedded as a resource. It is compiled
// together with the generator output and called by methods marked
// with the TestMethod attribute using reflection during the test pass.
public static class NonemptyNameTests
{
public static void NameProperty()
{
Assert.AreEqual<string>(@"John ""\/\/"" Doe",
PersonalizedHelloWorld.Name);
}
public static void GetHelloWorldMessageMethod()
{
Assert.AreEqual<string>(@"Hello World! Hello John ""\/\/"" Doe!",
PersonalizedHelloWorld.GetHelloWorldMessage());
}
}
/************************************
* Project: CodeGeneratorTests *
* File Name: NameNullTests.cs *
* Build Action: Compile *
************************************/
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Reflection;
[TestClass]
public class NameNullTests
{
public const string AssemblyFileName = "NameNullTests.dll";
public const string GeneratedCodeFileName = "NameNullTests.target.cs";
public const string DynamicTestCodeFileName = "NameNullTests.dynamic.cs";
public const string DynamicTestCodeResourceName = "CodeGeneratorTests.NameNullTests.dynamic.cs";
private static Type dynamicTestType;
[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
string generatedCode = CodeGenerator.Generate(null);
Assembly assembly = TestCompiler.Compile(generatedCode,
GeneratedCodeFileName,
DynamicTestCodeResourceName,
DynamicTestCodeFileName,
AssemblyFileName);
dynamicTestType = assembly.GetType("NameNullTests");
}
[TestMethod]
public void NameProperty()
{
dynamicTestType.GetMethod("NameProperty").Invoke(null, null);
}
[TestMethod]
public void GetHelloWorldMessageMethod()
{
dynamicTestType.GetMethod("GetHelloWorldMessageMethod").Invoke(null, null);
}
}
/******************************************
* Project: CodeGeneratorTests *
* File Name: NameNullTests.dynamic.cs *
* Build Action: Embedded Resource *
******************************************/
using Microsoft.VisualStudio.TestTools.UnitTesting;
public static class NameNullTests
{
public static void NameProperty()
{
Assert.IsNull(PersonalizedHelloWorld.Name);
}
public static void GetHelloWorldMessageMethod()
{
Assert.AreEqual<string>("Hello World! Hello anonymous user!",
PersonalizedHelloWorld.GetHelloWorldMessage());
}
}
This posting is provided "AS IS" with no warranties, and confers no rights.