Automating Jetstress…

Artykuł
06/11/2009

One of the things I like about Jetstress is that it tends to identify failing or failed components before the servers go live. Its not all about how many IOPS a storage array can provide, sometimes its just as important to prove the stability of the infrastructure and not just its peak performance.

So, we all know that Jetstress is a great tool. It helps us prove our storage design and is also really good for finding problems in a storage deployment before it goes live. The problem is that it takes a while to initialize and run – and quite often on a large project different people will configure the test slightly differently, even with clear documentation.

On my current project I was faced with the requirement of deploying and therefore jetstress testing, 20 CCR clusters, which meant 40 jetstress tests!. Along with all of the other deployment “stuff” this wasn't going to leave me with much time…. not to mention that setting up jetstress and watching it run for a few hours is like watching paint dry…

This is where I decided I would look at automating the jetstress process from end to end. I already had a pretty solid automated build process (I will blog about this in the next few weeks), so I just needed to tag in the jetstress bits prior to installing Exchange and I would be all set…

Jetstress process outline

Before I could begin automating things I needed to determine exactly what I needed to automate…

Create Jetstress directories
Install Jetstress
Copy over ESE DLL files
Start Jetstress to register the DLL files
Close Jetstress
Start Jetstress Again
Configure test
Initialize databases
Run Test
Copy output files centrally
Remove test databases
Uninstall jetstress

Pre-Requisites

In addition to the actual functions, I also had some pre-requisites…

Disks need to be ready
Need to have a copy of the intended ESE Dll’s
Need to have exported the Jetstress XML config file from the GUI
Must have powershell installed

Phase 1 (Steps 1 – 3)

The first phase is where we create the directory structure, install jetstress and copy over the ESE dll files we need to use.

Step 1 – Create Jetstress directories

To do this I decided to use powershell. To make things really easy I wrote some code to query the jetstress xml file and look at the directories specified within it. This would allow me to use the process with any jetstress configuration file without needing to update any code. Copy the following lines of code and save it as create_jetstress_dirs.ps1

create_jetstress_dirs.ps1

###### BEGIN COPY UNDERNEATH THIS LINE ############

################################################################### #
# Read a JetStress Config File and Create the required dirs
#
# Written: Neil Johnson (neiljohn@microsoft.com)
#
###################################################################

if ($args[0])
{
$configfile = $args[0]
}
else
{
$configfile = ".\jetstress.xml"
}

if (!(Test-Path -path "$configfile"))
{
throw "No Config File Found - create_jetstress_dirs.ps1 <path_to_config_file>"
}

$jetstress = Get-Content $configfile

foreach ($line in $jetstress)
{
    if ($line -match "<path>" -or $line -match "<logpath>")
    {
        $path_array = $line.split('<>');
        $path = $path_array[2]

        if ((Test-Path -path $path))
        {
            write-host -f green "$path already exists"

        }
        else
        {
            write-host -f yellow "Creating $path..."
            $nul = New-Item "$path" -type directory -force

}

###### BEGIN COPY ABOVE THIS LINE ############

Step 2 – Install Jetstress

Jetstress supports silent installation, so installing it is as simple as running the following command from a batch file or powershell script…

start /wait msiexec /i c:\jetstress\jetstress.msi /quiet

Step 3 – Copy over ESE DLL files

Jetstress simply uses the Exchange ESE dll files to create and control the databases for testing. As such it requires a copy of the ESE dll files before it can begin. The following files are required for jetstress to function…

ese.dll
eseperf.dll
eseperf.hxx
eseperf.ini

I recommend copying these files from a production server that will be running the same version of Exchange into a directory you can keep for later use.

Now we have our ESE DLL files and a copy of jetstress installed, we just need to copy them into the jetstress installation directory…

xcopy C:\JetStress\ESE\SP1-RU5-x64\*.* "C:\Program Files\Exchange Jetstress" /y

So, with phase 1 complete, we have a batch file that looks like this…

Phase 2 - (Steps 4 - 9)

So this bit took me quite a while to get working exactly as I wanted. Essentially on first run JetStress will register the ESE files and performance counters with the operating system. If you are doing this via the GUI, it simply asks you to close and re-open JetStress after it has registered the DLL files. In the command line things are more complicated…

By default when you run JetStressCmd.exe it will register the DLL files and then spawn off a second copy of the process in a new window. This upsets the natural flow of my batch file because essentially the original JetStress process has now terminated so the batch file continues and in my case reboots! – I needed to run JetStress then carry on and do other things. The way I got around this behaviour was to initially begin JetStress with the /? switch – this forces the registration of the DLL files, but allows my batch file to continue as expected…

cd "C:\Program Files\Exchange Jetstress"
JetStressCmd.exe /?
JetstressCmd.exe /c c:\jetstress\jetstress.xml /timeout 2H0M0S /new

So, our batch file now looks like this…

Phase 3 - (Steps 10 - 12)

So, after roughly 4 hours we have completed our JetStress test – the next step is to copy the results and logs into a directory we can harvest later on. For my deployment I chose c:\jetstress\jetstress_results – however given sufficient permissions you could choose to directly copy the results to a shared location on a server somewhere. With hindsight this is the route I should have chosen for this deployment and I will aim to do this for the next one.

We are ideally looking to retain copies of the HTML, BLG, LOG and XML files that exist within the JetStress installation directory. Assuming you have installed into the default directory the following lines of batch file will gather the results…

mkdir c:\jetstress\jetstress_results
xcopy "c:\program files\exchange JetStress\*.html" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.blg" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.log" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.xml" c:\jetstress\jetstress_results\ /y

The next step is to uninstall JetStress from the server ready for its Exchange installation. The following lines of batch file will uninstall JetStress and remove any files that are left in that directory…

start /wait msiexec /uninstall c:\jetstress\jetstress.msi /quiet
cd c:\jetstress
rmdir "C:\Program Files\Exchange Jetstress" /Q /S

The next thing left to do is remove the JetStress test databases and directories. For that we will use a slightly modified version of our earlier powershell script.

remove_jetstress_dirs.ps1

###### BEGIN COPY UNDERNEATH THIS LINE ############

################################################################### #
# Read a JetStress Config File and Delete the required dirs
#
# Written: Neil Johnson (neiljohn@microsoft.com)
#
###################################################################

if ($args[0])
{
$configfile = $args[0]
}
else
{
$configfile = ".\jetstress.xml"
}

if (!(Test-Path -path "$configfile"))
{
throw "No Config File Found - remove_jetstress_dirs.ps1 <path_to_config_file>"
}

$jetstress = Get-Content $configfile

foreach ($line in $jetstress)
{
    if ($line -match "<path>" -or $line -match "<logpath>")
    {
        $path_array = $line.split('<>');
        $path = $path_array[2]

        if ((Test-Path -path $path))
        {
            write-host -f yellow "Removing $path"
            $nul = remove-Item "$path" -force -recurse
        }
        else
        {
            write-host -f green "$path already removed"

}

###### BEGIN COPY ABOVE THIS LINE ############

After this point we have completed our jetstress test and we are ready to move on. I simply saved this batch file as c:\jetstress\runjetstress.cmd and called it from powershell at the point in my automated build where I wanted to run it.

So our final automated jetstress batch file looks like this…

@echo off
set configfile=c:\jetstress\jetstress.xml
cd c:\jetstress
powershell -file c:\jetstress\create_jetstress_dirs.ps1 %configfile%
start /wait msiexec /i c:\jetstress\jetstress.msi /quiet
xcopy C:\JetStress\ESE\SP1-RU5-x64\*.* "C:\Program Files\Exchange Jetstress" /y
cd "C:\Program Files\Exchange Jetstress"
JetStressCmd.exe /?
JetstressCmd.exe /c %configfile% /timeout 2H0M0S /new
mkdir c:\jetstress\jetstress_results
xcopy "c:\program files\exchange JetStress\*.html" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.blg" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.log" c:\jetstress\jetstress_results\ /y
xcopy "c:\program files\exchange JetStress\*.xml" c:\jetstress\jetstress_results\ /y
start /wait msiexec /uninstall c:\jetstress\jetstress.msi /quiet
cd c:\jetstress
rmdir "C:\Program Files\Exchange Jetstress" /Q /S
powershell -file c:\jetstress\remove_jetstress_dirs.ps1 %configfile%

So, there you have it – a totally automated and predictable method of running JetStress on your production servers. So far I have been using this process for the last six months and it has proven to be incredibly robust.

One last thing, I also chose to parse the results HTML file to attempt to determine if the test had passed or failed. The following powershell shows how I did this…

    if ((test-path("c:\jetstress\jetstress_results")))
    {
        write-host -f green "$phase - JetStress Test Completed: " -nonewline
        $result_file = dir c:\jetstress\jetstress_results\stress*.html
        $results = get-content $result_file

        if ($results -match "<td class=`"success`">Pass</td>")
        {
            write-host -f green "PASSED"
        }
        else
        {
            write-host -f red "FAILED"
        }

}

Posted by Neil Johnson , MCS UK, MCM Exchange 2007

Comments

Anonymous
January 01, 2003
is there any way to automatically acknowledge the 'press enter to initiate jetstress database instance shutdown' prompt at the end of each test run?
Anonymous
May 23, 2012
Thanks, it is really good article

Udostępnij za pośrednictwem