Using a little Ruby to automate mstest code coverage runs

I hate repetitive tasks - especially ones that are easily automated. Generating code coverage runs using mstest is an example of that.

The goal was to create a script that could do that following:

1) Validate that the input binaries were available

2) Validate that the necessary runtime components were available

3) Instrument the input binaries

4) Build up the test sandbox

5) Start the coverage service

6) Perform the tests using the instrumented binaries

7) Stop the coverage service

Being able to start the process at any point in there was bonus points.

I choose to use ruby, and specifically Rake, to automate this process. Rake is perfect for this type of task. I can easily break the tasks down into their core parts and build up the automation one bit at a time. Creating dependencies between the tasks is trivial. It reads well (even with my poor ruby skills) and has a very nice command line experience.

What I ended up with was a process where I could do this:

C:\ruby\coverage>rake --tasks

(in C:/ruby/coverage)

coverage coverage:all # Perform a complete code coverage pass...

coverage coverage:clean # Clean the coverage temp directory and sandbox

coverage coverage:cover # Instrument the assemblies for code coverage

coverage coverage:merge # Merge the test results

coverage coverage:populate # Populate the sandbox

coverage coverage:restore # Restores the backed up files if they exist

coverage coverage:run # Run the test suite

coverage coverage:start # Start the coverage service

coverage coverage:stop # Stop the coverage service

I can do all the things I set out to do (using “coverage:all”) or onesy-twosy as I need them (e.g. if rake aborts midway through the “run” step the coverage service will need to be manually stopped before the next run – I could modify a task to handle this more gracefully but I haven’t yet since the payoff isn’t there yet).

But you came for code ...

Rakefile

 

require 'rake'

require "win32/dir"

require 'fileutils'

require 'yaml'

def assert_config ( config )

  [ 'vsperfcmd' ,

   'prodbin' ,

   'root' ,

   'sandbox' ,

   'covtemp' ,

   'covorig' ,

   'vsinstr' ,

   'suitebin' ]. each { | setting |

      assert_config_setting ( config , setting )

  }  

end

def assert_config_setting ( config , setting )

  fail "Setting \" #{ setting } \" not found in coverage.yml" if config [ setting ]. nil?

end

Rake . application . init ( 'coverage' )

config = YAML :: load ( File . open ( 'coverage.yml' ))

assert_config ( config )

covfiles = FileList . new (

'Microsoft.TeamFoundation.Admin.dll' ,

'Microsoft.TeamFoundation.Management.Core.dll' ,

'Microsoft.TeamFoundation.Management.SnapIn.dll' ,

'Microsoft.TeamFoundation.Management.Controls.dll' ,

'tfsconfig.exe' ,

'tfsmgmt.exe'

)

namespace :coverage do

  desc "Instrument the assemblies for code coverage"

  task :cover => [ :clean_coverage , :clean_orig ] do

    Dir . chdir ( config [ 'prodbin' ]) do

      mkdir config [ 'covtemp' ]

      covfiles . each do | f |

        fail "Input file not found: #{ f } " if not File . exists? ( f )

        puts "Instrumenting #{ f } "

        cmd = "\" #{ config [ 'vsinstr' ]} \" \" #{ f } \" /COVERAGE"

        sh cmd

        assert_file_exists ( File . join ( config [ 'covtemp' ], f ), cmd )

        copy_coverage ( f , config [ 'covtemp' ])

      end

    end

  end

 

  desc "Restores the backed up files if they exist"

  task :restore do

    Dir . chdir ( config [ 'prodbin' ]) do

      covfiles . each do | f |

        origfile = " #{ f } .orig"

        if File . exists? ( origfile )

          rm f if File . exists? ( f )

          mv origfile , f

         

          instrpdb = f . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )

          rm instrpdb if File . exists? ( instrpdb )

        end

      end

    end

  end

 

  task :clean_backup do

    Dir . chdir ( config [ 'prodbin' ]) do

      covfiles . each do | f |

        origfile = " #{ f } .orig"

        rm origfile if File . exists? ( origfile )

        instrpdb = f . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )

        rm instrpdb if File . exists? ( instrpdb )

      end

    end

  end

 

  task :clean_coverage do

    FileUtils . rm_rf ( config [ 'covtemp' ])

  end

  task :clean_orig do

    FileUtils . rm_rf ( config [ 'covorig' ])

  end

  task :clean_sandbox do

    FileUtils . rm_rf ( config [ 'sandbox' ])

  end

 

  desc "Start the coverage service"

  task :start do

    mkdir config [ 'sandbox' ] unless File . exists? ( config [ 'sandbox' ])

    sh "\" #{ config [ 'vsperfcmd' ]} \" /start:coverage \"/output: #{ File . join ( config [ 'sandbox' ], 'adminops.coverage' )} \" /user:redmond\\vseqa1 /user:redmond\\tfssvc /user: #{ ENV [ 'USERDOMAIN' ]} \\ #{ ENV [ 'USERNAME' ]} /waitstart"

  end

  desc "Stop the coverage service"

  task :stop do

    sh " #{ config [ 'vsperfcmd' ]} /shutdown"

  end

 

  desc "Run the test suite"

  task :run do

        Dir . chdir ( config [ 'sandbox' ]) do

          puts "Running Unit Tests"

          sh "aotest.exe /assembly:AdminOps.UnitTests.dll /test * /out:logs"

    end

  end

  desc "Merge the test results"

  task :merge do

  end

 

  desc "Populate the sandbox"

  task :populate => [ :clean_sandbox ] do

    mkdir config [ 'sandbox' ]

   

    puts "Copying Product binaries..."

    FileUtils . cp_r ( Dir . glob ( File . join ( config [ 'prodbin' ], 'Microsoft.TeamFoundation.*' )), config [ 'sandbox' ])

    cp File . join ( config [ 'prodbin' ], 'NetFwTypeLib.dll' ), config [ 'sandbox' ]

    puts "Copying Test binaries..."

    FileUtils . cp_r ( File . join ( config [ 'suitebin' ], '.' ), config [ 'sandbox' ])

    puts "Copying Instrumented binaries..."

    FileUtils . cp_r ( Dir . glob ( File . join ( config [ 'covtemp' ], '.' )), config [ 'sandbox' ])

  end

  desc "Perform a complete code coverage pass from scratch"

  task :all => [ :clean , :cover , :populate , :start , :run , :stop , :merge ]

  desc "Clean the coverage temp directory and sandbox"

  task :clean => [ :clean_coverage , :clean_sandbox , :restore , :clean_backup ]

end

def assert_file_exists ( file , cmd )

  if not File . exists? ( file )

    puts "COVERAGE ERROR: file not found: #{ file } "

    puts "Executed: #{ cmd } "

  end

end

def copy_coverage ( file , dir )  

  if File . exists? ( file )

    cp file , dir

  else

    die "Assembly not found for assembly #{ file } "

  end

  pdbfile = file . gsub ( /(.*)\.(dll|exe)$/ , '\1.pdb' )

  if File . exists? ( pdbfile )

    cp pdbfile , dir

  else

    puts "WARNING: PDB file not found for assembly #{ pdbfile } "

  end

  instrpdbfile = file . gsub ( /(.*)\.(dll|exe)$/ , '\1.instr.pdb' )

  if File . exists? ( instrpdbfile )

    cp instrpdbfile , dir

  else

    puts "WARNING: INSTR PDB file not found for assembly #{ instrpdbfile } "

  end

end

coverage.yml

vsperfcmd: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsperfcmd.exe

prodbin: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386

root: C:/VMDEV/TfsArch1/binaries/x86chk

sandbox: C:/SANDBOX

covtemp: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage

covorig: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage/orig

vsinstr: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsinstr.exe

suitebin: C:/VMDEV/TfsArch1/binaries/x86chk/SuiteBin/i386/tfs/AdminOps/bin

 

So there you have it…

 

I go to my ruby command line (which is just a normal command line with the ruby bin directory in the path - e.g. “set PATH=%PATH%;c:\ruby\bin”) and perform:

C:\ruby\coverage>rake coverage:all

When the tests are done the coverage file I created is in c:\sandbox ready to be imported into VS or merged with other files or dumped to excel or whatever.

And one note about the rakefile – you may notice that I call “aotest.exe” not “mstest.exe” – aotest is just a little wrapper around mstest that one of our test devs wrote. I could have just as easily used mstest.