Using a little Ruby to automate mstest code coverage runs
I hate repetitive tasks - especially ones that are easily automated. Generating code coverage runs using mstest is an example of that.
The goal was to create a script that could do that following:
1) Validate that the input binaries were available
2) Validate that the necessary runtime components were available
3) Instrument the input binaries
4) Build up the test sandbox
5) Start the coverage service
6) Perform the tests using the instrumented binaries
7) Stop the coverage service
Being able to start the process at any point in there was bonus points.
I choose to use ruby, and specifically Rake, to automate this process. Rake is perfect for this type of task. I can easily break the tasks down into their core parts and build up the automation one bit at a time. Creating dependencies between the tasks is trivial. It reads well (even with my poor ruby skills) and has a very nice command line experience.
What I ended up with was a process where I could do this:
C:\ruby\coverage>rake --tasks
(in C:/ruby/coverage)
coverage coverage:all # Perform a complete code coverage pass...
coverage coverage:clean # Clean the coverage temp directory and sandbox
coverage coverage:cover # Instrument the assemblies for code coverage
coverage coverage:merge # Merge the test results
coverage coverage:populate # Populate the sandbox
coverage coverage:restore # Restores the backed up files if they exist
coverage coverage:run # Run the test suite
coverage coverage:start # Start the coverage service
coverage coverage:stop # Stop the coverage service
I can do all the things I set out to do (using “coverage:all”) or onesy-twosy as I need them (e.g. if rake aborts midway through the “run” step the coverage service will need to be manually stopped before the next run – I could modify a task to handle this more gracefully but I haven’t yet since the payoff isn’t there yet).
But you came for code ...
Rakefile
require 'rake'
require "win32/dir"
require 'fileutils'
require 'yaml'
def assert_config ( config )
[ 'vsperfcmd' ,
'prodbin' ,
'root' ,
'sandbox' ,
'covtemp' ,
'covorig' ,
'vsinstr' ,
'suitebin' ]. each { | setting |
assert_config_setting ( config , setting )
}
end
def assert_config_setting ( config , setting )
fail "Setting \" #{ setting } \" not found in coverage.yml" if config [ setting ]. nil?
end
Rake . application . init ( 'coverage' )
config = YAML :: load ( File . open ( 'coverage.yml' ))
assert_config ( config )
covfiles = FileList . new (
'Microsoft.TeamFoundation.Admin.dll' ,
'Microsoft.TeamFoundation.Management.Core.dll' ,
'Microsoft.TeamFoundation.Management.SnapIn.dll' ,
'Microsoft.TeamFoundation.Management.Controls.dll' ,
'tfsconfig.exe' ,
'tfsmgmt.exe'
)
namespace :coverage do
desc "Instrument the assemblies for code coverage"
task :cover => [ :clean_coverage , :clean_orig ] do
Dir . chdir ( config [ 'prodbin' ]) do
mkdir config [ 'covtemp' ]
covfiles . each do | f |
fail "Input file not found: #{ f } " if not File . exists? ( f )
puts "Instrumenting #{ f } "
cmd = "\" #{ config [ 'vsinstr' ]} \" \" #{ f } \" /COVERAGE"
sh cmd
assert_file_exists ( File . join ( config [ 'covtemp' ], f ), cmd )
copy_coverage ( f , config [ 'covtemp' ])
end
end
end
desc "Restores the backed up files if they exist"
task :restore do
Dir . chdir ( config [ 'prodbin' ]) do
covfiles . each do | f |
origfile = " #{ f } .orig"
if File . exists? ( origfile )
rm f if File . exists? ( f )
mv origfile , f
instrpdb = f . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )
rm instrpdb if File . exists? ( instrpdb )
end
end
end
end
task :clean_backup do
Dir . chdir ( config [ 'prodbin' ]) do
covfiles . each do | f |
origfile = " #{ f } .orig"
rm origfile if File . exists? ( origfile )
instrpdb = f . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )
rm instrpdb if File . exists? ( instrpdb )
end
end
end
task :clean_coverage do
FileUtils . rm_rf ( config [ 'covtemp' ])
end
task :clean_orig do
FileUtils . rm_rf ( config [ 'covorig' ])
end
task :clean_sandbox do
FileUtils . rm_rf ( config [ 'sandbox' ])
end
desc "Start the coverage service"
task :start do
mkdir config [ 'sandbox' ] unless File . exists? ( config [ 'sandbox' ])
sh "\" #{ config [ 'vsperfcmd' ]} \" /start:coverage \"/output: #{ File . join ( config [ 'sandbox' ], 'adminops.coverage' )} \" /user:redmond\\vseqa1 /user:redmond\\tfssvc /user: #{ ENV [ 'USERDOMAIN' ]} \\ #{ ENV [ 'USERNAME' ]} /waitstart"
end
desc "Stop the coverage service"
task :stop do
sh " #{ config [ 'vsperfcmd' ]} /shutdown"
end
desc "Run the test suite"
task :run do
Dir . chdir ( config [ 'sandbox' ]) do
puts "Running Unit Tests"
sh "aotest.exe /assembly:AdminOps.UnitTests.dll /test * /out:logs"
end
end
desc "Merge the test results"
task :merge do
end
desc "Populate the sandbox"
task :populate => [ :clean_sandbox ] do
mkdir config [ 'sandbox' ]
puts "Copying Product binaries..."
FileUtils . cp_r ( Dir . glob ( File . join ( config [ 'prodbin' ], 'Microsoft.TeamFoundation.*' )), config [ 'sandbox' ])
cp File . join ( config [ 'prodbin' ], 'NetFwTypeLib.dll' ), config [ 'sandbox' ]
puts "Copying Test binaries..."
FileUtils . cp_r ( File . join ( config [ 'suitebin' ], '.' ), config [ 'sandbox' ])
puts "Copying Instrumented binaries..."
FileUtils . cp_r ( Dir . glob ( File . join ( config [ 'covtemp' ], '.' )), config [ 'sandbox' ])
end
desc "Perform a complete code coverage pass from scratch"
task :all => [ :clean , :cover , :populate , :start , :run , :stop , :merge ]
desc "Clean the coverage temp directory and sandbox"
task :clean => [ :clean_coverage , :clean_sandbox , :restore , :clean_backup ]
end
def assert_file_exists ( file , cmd )
if not File . exists? ( file )
puts "COVERAGE ERROR: file not found: #{ file } "
puts "Executed: #{ cmd } "
end
end
def copy_coverage ( file , dir )
if File . exists? ( file )
cp file , dir
else
die "Assembly not found for assembly #{ file } "
end
pdbfile = file . gsub ( /(.*)\.(dll|exe)$/ , '\1.pdb' )
if File . exists? ( pdbfile )
cp pdbfile , dir
else
puts "WARNING: PDB file not found for assembly #{ pdbfile } "
end
instrpdbfile = file . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )
if File . exists? ( instrpdbfile )
cp instrpdbfile , dir
else
puts "WARNING: INSTR PDB file not found for assembly #{ instrpdbfile } "
end
end
coverage.yml
vsperfcmd: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsperfcmd.exe
prodbin: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386
root: C:/VMDEV/TfsArch1/binaries/x86chk
sandbox: C:/SANDBOX
covtemp: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage
covorig: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage/orig
vsinstr: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsinstr.exe
suitebin: C:/VMDEV/TfsArch1/binaries/x86chk/SuiteBin/i386/tfs/AdminOps/bin
So there you have it…
I go to my ruby command line (which is just a normal command line with the ruby bin directory in the path - e.g. “set PATH=%PATH%;c:\ruby\bin”) and perform:
C:\ruby\coverage>rake coverage:all
When the tests are done the coverage file I created is in c:\sandbox ready to be imported into VS or merged with other files or dumped to excel or whatever.
And one note about the rakefile – you may notice that I call “aotest.exe” not “mstest.exe” – aotest is just a little wrapper around mstest that one of our test devs wrote. I could have just as easily used mstest.