Note
Please see Azure Cognitive Services for Speech documentation for the latest supported speech solutions.
Generating Simulations and Performing Analysis
The following procedure describes how to use Simulator together with Simulator Results Analyzer to generate simulated recognition from utterances (such as “.wav” files containing speech), and to analyze the results of recognition.
Typical steps for using Simulator with Simulator Results Analyzer
Identify a set of utterances (“.wav” files) for which you want to obtain recognition results.
Identify the set of grammars to use for recognizing the utterances.
Create an input EMMA document that contains paths to the grammar files and the “.wav” files.
Create a configuration file that specifies the speech recognition engine that you want to use and configures its settings.
Run Simulator.exe and reference the EMMA document and the configuration file. Here’s an example command-line entry for running the tool:
Simulator.exe /In EmmaBatchFile.xml /Out MyRecognitionResults.xml /RecoConfig MyRecognizerConfiguration.xml
- Simulator.exe generates an EMMA output file that contains the raw recognition results for the batch of utterances. You can use the output EMMA file as input to your own analysis tools. Optionally, you can use the EMMA output file as input to SimulatorResultsAnalyzer.exe, as follows:
Run SimulatorResultsAnalyzer.exe and specify the output file from Simulator at the /In option. Here is an example command-line entry:
SimulatorResultsAnalyzer /In MyRecognitionResults.xml /Out MyRecognitionAnalysis.xml
- The output file generated by SimulatorResultsAnalyzer.exe includes calculation of CA/FA rates for the complete range of confidencelevel settings on the speech recognition engine, identifies out-of-grammar phrases, and calculates error rates that illustrate the overall accuracy of your grammars. You can use the output from SimulatorResultsAnalyzer.exe to generate tables and graphs using XML transforms of your design or as the basis for further analysis by your own tools.
Important
The configuration file (step 4) should set the confidence threshold to zero (confidencelevel="0") to ensure that all recognitions are returned, regardless of confidence score. This will ensure that you properly assess the confidence threshold setting for your application via the output of Simulator Results Analyzer. See Speech Recognition Engine Configuration File Settings.
For more information on valid command-line options and their use, see Simulator Command-Line Syntax and Simulator Results Analyzer Command-Line Syntax.