Running the DirectShow Performance Tests (Windows Embedded CE 6.0)
1/6/2010
This document deals with tasks necessary to run the DirectShow Performance Tests, including setting up the server, setting up the test environment, modifying the playlist, and collecting output logs.
Setting Up the Server
The DirectShow Performance Tests require a server in addition to the Windows Embedded CE-based test device. Set up the server as follows.
To set up the server for the DirectShow Performance Tests
Create a server with IIS Server or another HTTP server (for HTTP tests) and with WM Server (for MMST, MMSU, and intelligent HTTP streaming tests).
Copy over all content that you want to use in the test onto the server.
Create an HTTP publishing point that points to the content, and name the publishing point dshowmedia. (If you wish to use another name, you must update your HTTP URL in playlist.xml to reflect the name you have chosen.)
Note
Please refer to server documentation for instructions on installing and setting up publishing points.
- Create a media streaming publishing point on the WM server, and name that publishing point dshowmedia. (As above, if you wish to use another name, you must update your MMST and MMSU URLs in playlist.xml to reflect the name you have chosen.) Make sure that both MMS TCP and MMS UDP are enabled on this share point.
The playlist.xml file specifies from where the tests get the tested media. This file must be modified with information specific to your system, as detailed in the Modifying the Playlist section below. By default, the test types get the media from the locations as follows:
- HTTP tests look for media under http://<servername>:18080/dshowperf (HTTP over the WM server)
- MMS UDP tests look for media under mmsu://<servername>/dshowperf
- MMS TCP tests look for media under mmst://<servername>/dshowperf
Setting Up the Test Environment
To run the DirectShow Performance Tests, set up the test environment as follows.
To set up the test environment for the DirectShow Performance Tests
Set up a private network. Put your media server on this same network, and be sure your Windows Embedded CE device has access to the server.
Copy your playlist.xml from the source into your release folder, or into the same folder as your binary.
Modify the IP address or server name in playlist.xml to point to your media server on the private network. On occasion, your test device may not be able to resolve names and may need to have the server's IP address specified. If this is the case, be sure the playlist.xml file is updates with this information.
If you are testing with local storage, be sure the media folder is also present in the test device's storage. Copy the media content to the storage device. The default address for content in playlist.xml is \<disk storage>\dshowperf, and must be updated if you copy these files to another location.
Ensure your device has enough memory to run, especially for the high bitrate content. If using the standard shell, you can go to Control Panel | System | Memory and move the memory slider as far as possible to the left, leaving about 5% of the memory for object store only.
If the Windows Embedded CE-based device has a firewall, ensure the firewall is disabled for MMSU tests.
For HTTP streaming, you may also have to disable the proxy in the options for Microsoft Internet Explorer 6.0 for Windows Embedded CE.
Modifying the Playlist
A default media XML file, called playlist.xml, is provided with the DirectShow Performance Test. You must modify this file to your system needs or create your own. The XML schema is as follows:
<?xml version="1.0" encoding="utf-8" ?>
<Playlist>
<MediaContent id="">
<Description></Description>
<ClipName></ClipName>
<Codec></Codec>
<Locations>
<Local></Local>
<HTTP></HTTP>
<MMSU></MMSU>
<MMST></MMST>
</Locations>
<VideoStats height="" width="">
<BitRate></BitRate>
<ContainsAudio></ContainsAudio>
</VideoStats>
<AudioStats>
<BitRate></BitRate>
<Hz></Hz>
</AudioStats>
<QualityControl>
<BitRate></BitRate>
<Duration MarginOfError=""></Duration>
<AvgFrameRate MarginOfError=""></AvgFrameRate>
<AvgSyncOffset MarginOfError=""></AvgSyncOffset>
<DevSyncOffset MarginOfError=""></DevSyncOffset>
<FramesDrawn MarginOfError=""></FramesDrawn>
<FramesDroppedInRenderer MarginOfError=""></FramesDroppedInRenderer>
<FramesDroppedInDecoder MarginOfError=""></FramesDroppedInDecoder>
<Jitter MarginOfError=""></Jitter>
</QualityControl>
</MediaContent>
</Playlist>
The following table shows the tags used in the above XML schema and provides a description of each tag.
| XML Tag | Description |
|---|---|
<MediaContent id=""> |
Required tag specifying the beginning and end of the data for a test clip. The id number must be unique in the XML. The ID must be [short name of the clip] + [bitrate] + [anything else]. This is the same as the ClipID passed in as a command-line parameter for the test. |
<Description> |
Optional tag used to describe the clip and its purpose |
<ClipName> |
Tag used to indicate a nickname for the clip. This is the filename of the media clip. |
<Codec> |
Optional tag used to indicate the codec used to encode the content, such as WMV7, WMV8, WMV9, and MPEG4 |
<Locations> |
Required tag that specifies where to find the clip to test. At least one location must also be specified. |
<Local> |
The local location to find the content, meant to point toward a hard drive or storage drive |
<HTTP> |
Used to play clips located on an HTTP server |
<MMSU> |
Used to play clips from a WM server via MMS/UDP |
<MMST> |
Used to play clips from a WM server via MMS/TCPIP |
<VideoStats height="" width=""> |
Optional tag used to feed information about the media. If used, height and width must be specified with positive integers |
<BitRate> |
The encoding bit rate of the video portion of the clip, in bits per second; must be a positive integer. For example, a 1 MBit video encoding would be specified as <BitRate>1000000</BitRate> |
<ContainsAudio> |
Used to specify if there is audio in the test clip. Must use the word true or false. For example: <ContainsAudio>true</ContainsAudio> |
<AudioStats> |
Optional tag used to describe the audio data contain in the clip |
<BitRate> |
The bit rate at which the audio was encoded, in bits per second; must be a positive integer. For example, a 256 kbit audio encoding would be specified as <BitRate>256000</BitRate> |
<Hz> |
The rate at which the media was sampled when encoded; must be a positive integer. For example, 441.Khz would be <Hz>44100</Hz> |
<QualityControl> |
Required field used to pass or fail the DirectShow performance tests. Some or all of the fields below associated with Quality Control must be filled out. |
<BitRate> |
Total bit rate of the test clip (audio and video) in bits per second; must be a positive integer. For example, a 500 kbit clip encoding would be specified as <BitRate>500000</BitRate> |
<Duration MarginOfError=""> |
Length of the clip in seconds, reported as a floating point number. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<AvgFrameRate MarginOfError=""> |
The average frame rate multiplied by 100 and recorded as an integer. For example, 29.95 frames per second would be represented as 2995. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<AvgSyncOffset MarginOfError=""> |
The average sync offset, expressed as an integer. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<DevSyncOffset MarginOfError=""> |
The deviation sync offset, expressed as an integer. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<FramesDrawn MarginOfError=""> |
The number of frames drawn by the renderer, expressed as a positive integer. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<FramesDroppedInRenderer MarginOfError=""> |
The number of frames that should be dropped in the renderer, expressed as a positive integer. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<FramesDroppedInDecoder MarginOfError=""> |
The number of frames that should be dropped in the decoder, expressed as a positive integer. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
<Jitter MarginOfError=""> |
The amount of jitter that should be visible in the clip. The margin of error parameter must also be specified; use 0 to indicate no error or a negative value to disable this parameter from failing the test. |
The following example shows an XML file that specifies playing a video clip found on the server named "acedxmedia"; expects no more than 5 frames out of 3180 to be dropped; and expects the DRM clip to be in the same location as the non-DRM clip but with a different name. This file specifies the mediacontent ID as HMC(short name of clip) + 64 (encoding). This is used by the log parser, so having a different naming convention will cause the log parser to parse the information incorrectly.
<MediaContent id=" HMC64">
<Description> 160x120 64 kbit</Description>
<ClipName>highmotion_64K_160x120_main.wmv</ClipName>
<DRMClipName>DRM_highmotion_64K_160x120_main.wmv </DRMClipName>
<Codec>WMV 3</Codec>
<Locations>
<Local> \Hard Disk\dshowperf\HighMotion</Local>
<MMSU> mmsu://acedxmedia/dshowperf/HighMotion</MMSU>
<MMST> mmst://acedxmedia/dshowperf/HighMotion</MMST>
<HTTP> http://acedxmedia/dshowperf/HighMotion</HTTP>
</Locations>
<VideoStats height="160" width="120">
<BitRate>64000</BitRate>
<ContainsAudio>false</ContainsAudio>
</VideoStats>
<QualityControl>
<AvgFrameRate>2997</AvgFrameRate>
<FramesDroppedInRenderer MarginOfError="0">0</FramesDroppedInRenderer>
<FramesDroppedInDecoder MarginOfError="0">0</FramesDroppedInDecoder>
<FramesDrawn MarginOfError="5">3180</FramesDrawn>
</QualityControl>
</MediaContent>
Collecting Output Logs
Performance logs will only be generated if you specify the /perflog command line option. In this case, the default log generated is dshow_pb_stats.log. For each test run, this log contains a summary consisting of clip id, pass/fail status, total frames dropped, frames rendered, and average CPU usage. Future test runs append log data onto the existing log file.
Once the testing is completed, use PerfReporter.exe to parse the log file. First, open dshow_pb_stats.log and save it as an ASCII file. Run this on the desktop side by executing the following command line:
PerfReporter.exe dshow_pb_stats.log outfile.xls
Note
Failure to open dshow_pb_stats.log and resave it as an ASCII file will result in the parser throwing an exception.
After this completes, you can open your results file (called outfile.xls in the command line above) in Microsoft Excel. PerfReporter generates statistics by bitrate, clip name, and protocol. It also reports pass/fail based on the criteria provided in the XML file for each clip. A pass is indicated by '100' and a fail by '0'. The overall summaries fail if more than 1% of the runs in a category fail.
If the /status parameter was used, a log will be generated for each repetition. These logs are comma-separated lists with filenames beginning with dshow_pb_cpu_*. Each contains a running time log and an instantaneous CPU or memory load measurement. If dropped frames data is available, each time point will also contain statistics on frames rendered, frames dropped by the decoder, and frames dropped by the renderer. If the IAMNetworkStats interface is present, each time point will contain statistics on network packets lost, network packets recovered, and network packets received.