How to unit test bots
APPLIES TO: SDK v4
In this topic we'll show you how to:
- Create unit tests for bots.
- Use assert to check for activities returned by a dialog turn against expected values.
- Use assert to check the results returned by a dialog.
- Create different types of data driven tests.
- Create mock objects for the different dependencies of a dialog, such as language recognizers, and so on.
Prerequisites
The CoreBot Tests sample used in this topic references the Microsoft.Bot.Builder.Testing package, XUnit, and Moq to create unit tests.
The core bot sample uses Language Understanding (LUIS) to identify user intents; however, identifying user intent isn't the focus of this article. For information about identifying user intents, see Natural language understanding and Add natural language understanding to your bot.
Note
Language Understanding (LUIS) will be retired on 1 October 2025. Beginning 1 April 2023, you won't be able to create new LUIS resources. A newer version of language understanding is now available as part of Azure AI Language.
Conversational language understanding (CLU), a feature of Azure AI Language, is the updated version of LUIS. For more information about language understanding support in the Bot Framework SDK, see Natural language understanding.
Testing Dialogs
In the CoreBot sample, dialogs are unit tested through the DialogTestClient
class, which provides a mechanism for testing them in isolation outside of a bot and without having to deploy your code to a web service.
Using this class, you can write unit tests that validate dialogs responses on a turn-by-turn basis. Unit tests using DialogTestClient
class should work with other dialogs built using the botbuilder dialogs library.
The following example demonstrates tests derived from DialogTestClient
:
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Msteams, sut);
var reply = await testClient.SendActivityAsync<IMessageActivity>("hi");
Assert.Equal("Where would you like to travel to?", reply.Text);
reply = await testClient.SendActivityAsync<IMessageActivity>("Seattle");
Assert.Equal("Where are you traveling from?", reply.Text);
reply = await testClient.SendActivityAsync<IMessageActivity>("New York");
Assert.Equal("When would you like to travel?", reply.Text);
reply = await testClient.SendActivityAsync<IMessageActivity>("tomorrow");
Assert.Equal("OK, I will book a flight from Seattle to New York for tomorrow, Is this Correct?", reply.Text);
reply = await testClient.SendActivityAsync<IMessageActivity>("yes");
Assert.Equal("Sure thing, wait while I finalize your reservation...", reply.Text);
reply = testClient.GetNextReply<IMessageActivity>();
Assert.Equal("All set, I have booked your flight to Seattle for tomorrow", reply.Text);
The DialogTestClient
class is defined in the Microsoft.Bot.Builder.Testing
namespace and included in the Microsoft.Bot.Builder.Testing NuGet package.
DialogTestClient
The first parameter of DialogTestClient
is the target channel. This allows you to test different rendering logic based on the target channel for your bot (Teams, Slack, and so on). If you're uncertain about your target channel, you can use the Emulator
or Test
channel IDs but keep in mind that some components may behave differently depending on the current channel, for example, ConfirmPrompt
renders the Yes/No options differently for the Test
and Emulator
channels. You can also use this parameter to test conditional rendering logic in your dialog based on the channel ID.
The second parameter is an instance of the dialog being tested. In the sample code in this article, sut
represents the system under test.
The DialogTestClient
constructor provides additional parameters that allow you to further customize the client behavior or pass parameters to the dialog being tested if needed. You can pass initialization data for the dialog, add custom middleware or use your own TestAdapter and ConversationState
instance.
Sending and receiving messages
The SendActivityAsync<IActivity>
method allows you to send a text utterance or an IActivity
to your dialog and returns the first message it receives. The <T>
parameter is used to return a strong typed instance of the reply so you can assert it without having to cast it.
var reply = await testClient.SendActivityAsync<IMessageActivity>("hi");
Assert.Equal("Where would you like to travel to?", reply.Text);
In some scenarios your bot may send several messages in response to a single activity, in these cases DialogTestClient
will queue the replies and you can use the GetNextReply<IActivity>
method to pop the next message from the response queue.
reply = testClient.GetNextReply<IMessageActivity>();
Assert.Equal("All set, I have booked your flight to Seattle for tomorrow", reply.Text);
GetNextReply<IActivity>
will return null if there are no further messages in the response queue.
Asserting activities
The code in the CoreBot sample only asserts the Text
property of the returned activities. In more complex bots you may want to assert other properties like Speak
, InputHint
, ChannelData
, and so on.
Assert.Equal("Sure thing, wait while I finalize your reservation...", reply.Text);
Assert.Equal("One moment please...", reply.Speak);
Assert.Equal(InputHints.IgnoringInput, reply.InputHint);
You can do this by checking each property individually as shown above, you can write your own helper utilities for asserting activities or you can use other frameworks like FluentAssertions to write custom assertions and simplify your test code.
Passing parameters to your dialogs
The DialogTestClient
constructor has an initialDialogOptions
that can be used to pass parameters to your dialog. For example, the MainDialog
in this sample, initializes a BookingDetails
object from the language recognition results, with the entities it resolves from the user's utterance, and passes this object in the call to invoke BookingDialog
.
You can implement this in a test as follows:
var inputDialogParams = new BookingDetails()
{
Destination = "Seattle",
TravelDate = $"{DateTime.UtcNow.AddDays(1):yyyy-MM-dd}"
};
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Msteams, sut, inputDialogParams);
BookingDialog
receives this parameter and accesses it in the test the same way as it would have been when invoked from MainDialog
.
private async Task<DialogTurnResult> DestinationStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var bookingDetails = (BookingDetails)stepContext.Options;
...
}
Asserting dialog turn results
Some dialogs like BookingDialog
or DateResolverDialog
return a value to the calling dialog. The DialogTestClient
object exposes a DialogTurnResult
property that can be used to analyze and assert the results returned by the dialog.
For example:
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Msteams, sut);
var reply = await testClient.SendActivityAsync<IMessageActivity>("hi");
Assert.Equal("Where would you like to travel to?", reply.Text);
...
var bookingResults = (BookingDetails)testClient.DialogTurnResult.Result;
Assert.Equal("New York", bookingResults?.Origin);
Assert.Equal("Seattle", bookingResults?.Destination);
Assert.Equal("2019-06-21", bookingResults?.TravelDate);
The DialogTurnResult
property can also be used to inspect and assert intermediate results returned by the steps in a waterfall.
Analyzing test output
Sometimes it's necessary to read a unit test transcript to analyze the test execution without having to debug the test.
The Microsoft.Bot.Builder.Testing package includes a XUnitDialogTestLogger
that logs the messages sent and received by the dialog to the console.
To use this middleware, your test needs to expose a constructor that receives an ITestOutputHelper
object that is provided by the XUnit test runner and create a XUnitDialogTestLogger
that will be passed to DialogTestClient
through the middlewares
parameter.
public class BookingDialogTests
{
private readonly IMiddleware[] _middlewares;
public BookingDialogTests(ITestOutputHelper output)
: base(output)
{
_middlewares = new[] { new XUnitDialogTestLogger(output) };
}
[Fact]
public async Task SomeBookingDialogTest()
{
// Arrange
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Msteams, sut, middlewares: _middlewares);
...
}
}
Here's an example of what the XUnitDialogTestLogger
logs to the output window when it's configured:
For additional information on sending test output to the console when using XUnit see Capturing Output in the XUnit documentation.
This output will be also logged on the build server during the continuous integration builds and helps you analyze build failures.
Data Driven Tests
In most cases the dialog logic doesn't change and the different execution paths in a conversation are based on the user utterances. Rather than writing a single unit test for each variant in the conversation it's easier to use data driven tests (also known as parameterized test).
For example, the sample test in the overview section of this document shows how to test one execution flow, but not others, such as:
- What happens if the user says no to the confirmation?
- What if they use a different date?
Data driven tests allow us to test all these permutations without having to rewrite the tests.
In the CoreBot sample, we use Theory
tests from XUnit to parameterize tests.
Theory tests using InlineData
The following test checks that a dialog gets canceled when the user says "cancel".
[Fact]
public async Task ShouldBeAbleToCancel()
{
var sut = new TestCancelAndHelpDialog();
var testClient = new DialogTestClient(Channels.Test, sut);
var reply = await testClient.SendActivityAsync<IMessageActivity>("Hi");
Assert.Equal("Hi there", reply.Text);
Assert.Equal(DialogTurnStatus.Waiting, testClient.DialogTurnResult.Status);
reply = await testClient.SendActivityAsync<IMessageActivity>("cancel");
Assert.Equal("Cancelling...", reply.Text);
}
To cancel a dialog, users can type "quit", "never mind", and "stop it". Rather than writing a new test case for every possible word, write a single Theory
test method that accepts parameters via a list of InlineData
values to define the parameters for each test case:
[Theory]
[InlineData("cancel")]
[InlineData("quit")]
[InlineData("never mind")]
[InlineData("stop it")]
public async Task ShouldBeAbleToCancel(string cancelUtterance)
{
var sut = new TestCancelAndHelpDialog();
var testClient = new DialogTestClient(Channels.Test, sut, middlewares: _middlewares);
var reply = await testClient.SendActivityAsync<IMessageActivity>("Hi");
Assert.Equal("Hi there", reply.Text);
Assert.Equal(DialogTurnStatus.Waiting, testClient.DialogTurnResult.Status);
reply = await testClient.SendActivityAsync<IMessageActivity>(cancelUtterance);
Assert.Equal("Cancelling...", reply.Text);
}
The new test will be executed four times with the different parameters and each case will show as a child item under the ShouldBeAbleToCancel
test in Visual Studio Test Explorer. If any of them fail as shown below, you can right click and debug the scenario that failed rather than rerunning the entire set of tests.
Theory tests using MemberData and complex types
InlineData
is useful for small data driven tests that receive simple value type parameters (string, int, and so on).
The BookingDialog
receives a BookingDetails
object and returns a new BookingDetails
object. A non-parameterized version of a test for this dialog would look as follows:
[Fact]
public async Task DialogFlow()
{
// Initial parameters
var initialBookingDetails = new BookingDetails
{
Origin = "Seattle",
Destination = null,
TravelDate = null,
};
// Expected booking details
var expectedBookingDetails = new BookingDetails
{
Origin = "Seattle",
Destination = "New York",
TravelDate = "2019-06-25",
};
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Test, sut, initialBookingDetails);
// Act/Assert
var reply = await testClient.SendActivityAsync<IMessageActivity>("hi");
...
var bookingResults = (BookingDetails)testClient.DialogTurnResult.Result;
Assert.Equal(expectedBookingDetails.Origin, bookingResults?.Origin);
Assert.Equal(expectedBookingDetails.Destination, bookingResults?.Destination);
Assert.Equal(expectedBookingDetails.TravelDate, bookingResults?.TravelDate);
}
To parameterize this test, we created a BookingDialogTestCase
class that contains our test case data. It contains the initial BookingDetails
object, the expected BookingDetails
and an array of strings containing the utterances sent from the user and the expected replies from the dialog for each turn.
public class BookingDialogTestCase
{
public BookingDetails InitialBookingDetails { get; set; }
public string[,] UtterancesAndReplies { get; set; }
public BookingDetails ExpectedBookingDetails { get; set; }
}
We also created a helper BookingDialogTestsDataGenerator
class that exposes a IEnumerable<object[]> BookingFlows()
method that returns a collection of the test cases to be used by the test.
In order to display each test case as a separate item in Visual Studio Test Explorer, the XUnit test runner requires that complex types like BookingDialogTestCase
implement IXunitSerializable
, to simplify this, the Bot.Builder.Testing framework provides a TestDataObject
class that Implements this interface and can be used to wrap the test case data without having to implement IXunitSerializable
.
Here's a fragment of IEnumerable<object[]> BookingFlows()
that shows how the two classes are used:
public static class BookingDialogTestsDataGenerator
{
public static IEnumerable<object[]> BookingFlows()
{
// Create the first test case object
var testCaseData = new BookingDialogTestCase
{
InitialBookingDetails = new BookingDetails(),
UtterancesAndReplies = new[,]
{
{ "hi", "Where would you like to travel to?" },
{ "Seattle", "Where are you traveling from?" },
{ "New York", "When would you like to travel?" },
{ "tomorrow", $"Please confirm, I have you traveling to: Seattle from: New York on: {DateTime.Now.AddDays(1):yyyy-MM-dd}. Is this correct? (1) Yes or (2) No" },
{ "yes", null },
},
ExpectedBookingDetails = new BookingDetails
{
Destination = "Seattle",
Origin = "New York",
TravelDate = $"{DateTime.Now.AddDays(1):yyyy-MM-dd}",
},
};
// wrap the test case object into TestDataObject and return it.
yield return new object[] { new TestDataObject(testCaseData) };
// Create the second test case object
testCaseData = new BookingDialogTestCase
{
InitialBookingDetails = new BookingDetails
{
Destination = "Seattle",
Origin = "New York",
TravelDate = null,
},
UtterancesAndReplies = new[,]
{
{ "hi", "When would you like to travel?" },
{ "tomorrow", $"Please confirm, I have you traveling to: Seattle from: New York on: {DateTime.Now.AddDays(1):yyyy-MM-dd}. Is this correct? (1) Yes or (2) No" },
{ "yes", null },
},
ExpectedBookingDetails = new BookingDetails
{
Destination = "Seattle",
Origin = "New York",
TravelDate = $"{DateTime.Now.AddDays(1):yyyy-MM-dd}",
},
};
// wrap the test case object into TestDataObject and return it.
yield return new object[] { new TestDataObject(testCaseData) };
}
}
Once we create an object to store the test data and a class that exposes a collection of test cases, we use the XUnit MemberData
attribute instead of InlineData
to feed the data into the test, the first parameter for MemberData
is the name of the static function that returns the collection of test cases and the second parameter is the type of the class that exposes this method.
[Theory]
[MemberData(nameof(BookingDialogTestsDataGenerator.BookingFlows), MemberType = typeof(BookingDialogTestsDataGenerator))]
public async Task DialogFlowUseCases(TestDataObject testData)
{
// Get the test data instance from TestDataObject
var bookingTestData = testData.GetObject<BookingDialogTestCase>();
var sut = new BookingDialog();
var testClient = new DialogTestClient(Channels.Test, sut, bookingTestData.InitialBookingDetails);
// Iterate over the utterances and replies array.
for (var i = 0; i < bookingTestData.UtterancesAndReplies.GetLength(0); i++)
{
var reply = await testClient.SendActivityAsync<IMessageActivity>(bookingTestData.UtterancesAndReplies[i, 0]);
Assert.Equal(bookingTestData.UtterancesAndReplies[i, 1], reply?.Text);
}
// Assert the resulting BookingDetails object
var bookingResults = (BookingDetails)testClient.DialogTurnResult.Result;
Assert.Equal(bookingTestData.ExpectedBookingDetails?.Origin, bookingResults?.Origin);
Assert.Equal(bookingTestData.ExpectedBookingDetails?.Destination, bookingResults?.Destination);
Assert.Equal(bookingTestData.ExpectedBookingDetails?.TravelDate, bookingResults?.TravelDate);
}
Here's an example of the results for the DialogFlowUseCases
tests in Visual Studio Test Explorer when the test is executed:
Using Mocks
You can use mock elements for the things that aren't currently tested. For reference, this level can generally be thought of as unit and integration testing.
Mocking as many elements as you can allows for better isolation of the piece you're testing. Candidates for mock elements include storage, the adapter, middleware, activity pipeline, channels, and anything else that isn't directly part of your bot. This could also involve removing certain aspects temporarily, such as middleware not involved in the part of your bot that you're testing, to isolate each piece. However, if you're testing your middleware, you may want to mock your bot instead.
Mocking elements can take a handful of forms, from replacing an element with a different known object to implementing minimal hello world functionality. This could also take the form of removing the element, if it's not necessary, or forcing it to do nothing.
Mocks allow us to configure the dependencies of a dialog and ensure they're in a known state during the execution of the test without having to rely on external resources like databases, language models or other objects.
In order to make your dialog easier to test and reduce its dependencies on external objects, you may need to inject the external dependencies in the dialog constructor.
For example, instead of instantiating BookingDialog
in MainDialog
:
public MainDialog()
: base(nameof(MainDialog))
{
...
AddDialog(new BookingDialog());
...
}
We pass an instance of BookingDialog
as a constructor parameter:
public MainDialog(BookingDialog bookingDialog)
: base(nameof(MainDialog))
{
...
AddDialog(bookingDialog);
...
}
This allow us to replace the BookingDialog
instance with a mock object and write unit tests for MainDialog
without having to call the actual BookingDialog
class.
// Create the mock object
var mockDialog = new Mock<BookingDialog>();
// Use the mock object to instantiate MainDialog
var sut = new MainDialog(mockDialog.Object);
var testClient = new DialogTestClient(Channels.Test, sut);
Mocking Dialogs
As described above, MainDialog
invokes BookingDialog
to obtain the BookingDetails
object. We implement and configure a mock instance of BookingDialog
as follows:
// Create the mock object for BookingDialog.
var mockDialog = new Mock<BookingDialog>();
mockDialog
.Setup(x => x.BeginDialogAsync(It.IsAny<DialogContext>(), It.IsAny<object>(), It.IsAny<CancellationToken>()))
.Returns(async (DialogContext dialogContext, object options, CancellationToken cancellationToken) =>
{
// Send a generic activity so we can assert that the dialog was invoked.
await dialogContext.Context.SendActivityAsync($"{mockDialogNameTypeName} mock invoked", cancellationToken: cancellationToken);
// Create the BookingDetails instance we want the mock object to return.
var expectedBookingDialogResult = new BookingDetails()
{
Destination = "Seattle",
Origin = "New York",
TravelDate = $"{DateTime.UtcNow.AddDays(1):yyyy-MM-dd}"
};
// Return the BookingDetails we need without executing the dialog logic.
return await dialogContext.EndDialogAsync(expectedBookingDialogResult, cancellationToken);
});
// Create the sut (System Under Test) using the mock booking dialog.
var sut = new MainDialog(mockDialog.Object);
In this example, we used Moq to create the mock dialog and the Setup
and Returns
methods to configure its behavior.
Mocking LUIS results
Note
Language Understanding (LUIS) will be retired on 1 October 2025. Beginning 1 April 2023, you won't be able to create new LUIS resources. A newer version of language understanding is now available as part of Azure AI Language.
Conversational language understanding (CLU), a feature of Azure AI Language, is the updated version of LUIS. For more information about language understanding support in the Bot Framework SDK, see Natural language understanding.
In simple scenarios, you can implement mock LUIS results through code as follows:
var mockRecognizer = new Mock<IRecognizer>();
mockRecognizer
.Setup(x => x.RecognizeAsync<FlightBooking>(It.IsAny<ITurnContext>(), It.IsAny<CancellationToken>()))
.Returns(() =>
{
var luisResult = new FlightBooking
{
Intents = new Dictionary<FlightBooking.Intent, IntentScore>
{
{ FlightBooking.Intent.BookFlight, new IntentScore() { Score = 1 } },
},
Entities = new FlightBooking._Entities(),
};
return Task.FromResult(luisResult);
});
LUIS results can be complex. When they are, it's simpler to capture the desired result in a JSON file, add it as a resource to your project, and deserialize it into a LUIS result. Here's an example:
var mockRecognizer = new Mock<IRecognizer>();
mockRecognizer
.Setup(x => x.RecognizeAsync<FlightBooking>(It.IsAny<ITurnContext>(), It.IsAny<CancellationToken>()))
.Returns(() =>
{
// Deserialize the LUIS result from embedded json file in the TestData folder.
var bookingResult = GetEmbeddedTestData($"{GetType().Namespace}.TestData.FlightToMadrid.json");
// Return the deserialized LUIS result.
return Task.FromResult(bookingResult);
});