Jaa

Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

  • Artikkeli

Series Index

+++

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

  • AUT (application under test) – this is the tested application, in which faults are being injected;
  • Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
    • a Method Signature, determining the method where the fault will be injected;
    • a Fault Condition, determining when the specific fault should be triggered (e.g. every Nth call)
    • a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
  • Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

 //
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe 0) 2 + 3 = 5 1) 2 + 3 = 5 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

 

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

 // 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

FaultInjectionSample.zip

Comments

  • Anonymous
    February 24, 2010
    I am trying to use the FaultInjection libraries to simulate data access failures. Is this proper method signature to get the ExecuteNonQuery() method on DbCommand?        Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()" Thanks.

  • Anonymous
    February 24, 2010
    Yep, this should work. Let me know if you have trouble and we will help.

  • Anonymous
    March 01, 2010
    I had success when using the fault injection libraries on some sample code I had written, but now that I'm trying to use it against our intended target code it doesn't seem to be working. I want to use the ReturnValueFault on an internal static method which returns an int. The method is within an internal sealed class with the SecuritySafeCritical attribute set. I'm creating the fault rule like this: FaultRule rule = new FaultRule("static FullNamespace.ClassName.GetDeviceId(uint, IntPtr, IntPtr, IntPtr)", BuiltInConditions.TriggerOnEveryCall, BuiltInFaults.ReturnValueFault(1000)); I also tried prepending out to IntPtr as these are specified as out parameters in the method signature. I didn't have success either way. Am I doing something wrong here? My other thought is that this class and method are in a library/dll, and not within the exe code itself which is the process I'm starting a fault injection session for. Any help is greatly appreciated. So far I've been very happy with the low cost of entry for using this API and the good documentation.

  • Anonymous
    March 01, 2010
    Hi David, can you try to replace "IntPtr" with "System.IntPtr" in the parameters list of the method? the data type of the parameter need to be fully qualified name too. let me know if it still doesn't work.

  • Anonymous
    March 03, 2010
    I tried System.IntPtr and had no luck here either. Is it an issue if the binary I am trying to inject into is signed?

  • Anonymous
    March 04, 2010
    no, the issue is not related to the sign. i noticed that you mention about 'out parameter'. please make sure that 'out' is also part of the method signature. i.e your method signature should be: static FullNamespace.ClassName.GetDeviceId(uint, out System.IntPtr,out System.IntPtr,out System.IntPtr) i wrote a dummy app with the same method signature above, and the fault injection api works for me without problem. if this still doesn't work for you. most likely reason here is that the method signature still not correct somehow, i.e it does not match what clr expecting in the runtime). a few thing you can duble check:

  1. make sure the method got invoked.
  2. make sure the method signature is correct or no typo (i often see people has typo in the method signature by mistakes).
  3. in the %temp% folder, there should have some log, check if there are some error.

  • Anonymous
    March 05, 2010
    I have tried with the out parameter and also it did not work. I coulnd't find any logs in %temp% folder. I realized today that the AUT is a native program that runs the code from the managed assemblies by hosting a CLR runtime. Is this my problem, is there a way to still use TestApi fault injection or will I have to use some native fault injection library instead?

  • Anonymous
    March 05, 2010
    The comment has been removed

  • Anonymous
    March 05, 2010
    Darn, this would have been the perfect solution. Thanks for your help!

  • Anonymous
    March 09, 2010
    I am just getting around to responding to Ivan's response to my original comment. Ivan, I am unable to get ExecuteNonQuery() to throw an exception for me. Here is all my code from within a (slightly verbose) unit test:        Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()"        'Dim condition = BuiltInConditions.TriggerOnNthCall(2)        Dim condition = BuiltInConditions.TriggerOnEveryCall        Const ExceptionMessage As String = "This is a fault-injected exception."        Dim fault = BuiltInFaults.ThrowExceptionFault(New DataException(ExceptionMessage))        Dim faultRule = New FaultRule(method, condition, fault)        Dim s = New FaultSession(faultRule)        Dim rp = TestDataFactory.CreateYrtReinsurancePolicy(PolicyNumberFactory.CreateUnique())        Dim isExceptionThrown As Boolean        Try            rp.Save()        Catch ex As DataException            If ex.Message.Equals(ExceptionMessage) Then                isExceptionThrown = True            End If        End Try        Assert.IsTrue(isExceptionThrown) Note that I gave up on the NthCall and went with EveryCall just to see if I could get it to work. Any ideas?

  • Anonymous
    March 16, 2010
    The comment has been removed

  • Anonymous
    March 18, 2010
    The problem with that workflow is that I would need to write a process wrapper for each and every unit test that I write. I think my solution file will get out of control quickly. Given that, I think that I will wait for your private version. Thanks for the clarification. I just didn't infer that launching an additional process was necessary.

  • Anonymous
    May 02, 2010
    Hi, When i try to add the FaultInjectionEngine.dll as a reference to my code in VisualC#, it is giving an error which says, "Please make sure that the file is accessible and that it is a valid assembly or COM object". Please help me on this.

  • Anonymous
    May 03, 2010
    In your VS project, you should add reference to TestApiCore.dll, not faultinjectionengine.dll.

  • Anonymous
    May 04, 2010
    Yah did the same, Still the same error crops in.

  • Anonymous
    May 05, 2010
    hmm, that's strange. if you haven't, can you try to create a brand new project and add testapicore.dll? what's the version of your VS, OS, 32bit or 64 bit? also check my blog which has step by step instructions. you can see if you missed any important one. http://blogs.msdn.com/billliu/default.aspx

  • Anonymous
    October 21, 2010
    Hi I am planning to use FI API's to introduce faults in to my application hosted on IIS 7.0. Can you please guide me on this.. ? Thanks

  • Anonymous
    October 24, 2010
    Re: fault injection in IIS 7.0: Check out Bill's recent post demonstrating that: blogs.msdn.com/.../using-fault-injection-api-for-web-application-or-windows-service.aspx

  • Anonymous
    April 24, 2011
    Hi, I am new to this, I copied the sample solution given here and tried on visual studio 2010 . It simply isn't working .. I mean the fault injection test is not injecting the fault at all. Please help.

  • Anonymous
    April 25, 2011
    Hi , when i try to run the same sample project on VS 2010 (after conversion) on winXP(SP3) the fault injection simply doesnt work. Please help. I have tried the sample code on VS 2008 on a windows 7. It does work there though I follow the same steps.

  • Anonymous
    June 10, 2012
    how to create custom fault and custom condition? please help

  • Anonymous
    June 11, 2012
    for vs2010 or .net 4.0 application, you need to enable one env variable. see my blog for steps: blogs.msdn.com/.../if-it-still-not-working.aspx

  • Anonymous
    June 11, 2012
    the following code snipet to show how to custom the condition. the condition is to trigger fault if the current machine is management server (regkey settings):    [Serializable]    class MyCondition : ICondition    {        public bool Trigger(IRuntimeContext context)        {            RegistryKey masterKey = Registry.LocalMachine.CreateSubKey("SOFTWARE\Microsoft\FaultInjection");            string reg = "";            if (masterKey == null)            {                Console.WriteLine("Null Masterkey!");            }            else            {                reg = masterKey.GetValue("ServerType").ToString();                Console.WriteLine("MyKey = {0}", masterKey.GetValue("ServerType"));            }            masterKey.Close();            return reg.Equals("Management", StringComparison.OrdinalIgnoreCase);        }    }

  • Anonymous
    January 10, 2013
    Wait a minute. I thought you said "It uses the .NET Profiler API to dynamically instrument the binaries as they are running so that the binaries are only altered in memory and not on the hard disk. " But from the examples it seems you always have launch the application under test itself using the test apis/exe but can't actually target an already launched application and inject code into it, even though that's what you claimed.

  • Anonymous
    January 16, 2013
    The comment has been removed

  • Anonymous
    February 08, 2013
    Thank you for sharing this very nice post, please keep continue the sharing of this types of information. Here we are waiting for more

  • Anonymous
    February 10, 2013
    @Performance Injector: Thanks for the nice words! :) We are looking into extending the TestApi facilities further.

  • Anonymous
    February 12, 2013
    i hate this it didnt give me what i need

  • Anonymous
    February 12, 2013
    @none What didn't work? Can I help?

  • Anonymous
    September 22, 2013
    what if Class A implement Interface B, with method C. should I injection to specify to inject "A.C" or "B.C"?