Protocol Test Suites (Part Two)
This is the second part of the two-part series on Open Specifications Protocol Test Suites. In the previous post, we gave an overview of Test Suites and their different components. Now let's go over the prerequisites of a Test Suite environment, the typical contents of a Test Suite package, cover the end-to-end process of configuring and running Test Suites, and analyze the results.
Setting up a Test Suite Environment
The following table lists the tools and software necessary to set up your environment for Exchange and SharePoint Test Suites.
Note: For information on setting up a test environment for a specific Test Suite, refer to the Test Suite Reference Guide for that Test Suite. For more information on the Reference Guide and other contents of a Test Suite folder, see the subsequent section.
| Machine | Test suite environment (Exchange) | Test suite environment (SharePoint) |
| *SUT |
|
|
| Client |
|
|
* SUT may be replaced in a non-Microsoft implementation
Using a Test Suite
When you download a Test Suite, the package is downloaded as a compressed folder in .zip format. When you unzip the folder, you will see several different folders and subfolders. It is helpful to understand what each one of these folders contains.
Folder structure and contents
A Test Suite folder mainly consists of configuration files (both client and server-side), test cases, and associated supporting files. These can include scripts, methods, properties, source code, technical specifications for protocols associated with the Test Suite, and end-user help documentation such as a Reference Guide, a Test Suite Deployment Guide, and a Test Suite Specification. The folder also contains a Requirements Specification spreadsheet (a list of normative requirements extracted from the technical document) for each protocol associated with the Test Suite.
The following diagram shows how the contents of a Test Suite folder are organized:
Figure 1: Test Suite Folder Structure
Note: For more information on each component in a specific Test Suite folder, refer to the Test Suite Deployment Guide for that Test Suite.
Running Test Suites
The following illustration shows the basic steps of running a Test Suite.
Note: Information for running a specific Test Suite is available in the Test Suite Deployment Guide for that Test Suite, found in the Test Suite folder.
Figure 2: Running Test Suites
Note: The Test Suite Deployment Guide contains detailed guidance on how to configure the client and the SUT, run test cases, and view debug reports.
Analyzing the Test Results
After a Test Suite has run successfully, log files and reports are generated. These files enable you to identify errors and determine what exactly failed in the validation. The following types of files are available for analysis:
- TRX log: A log file produced by Visual Studio. This file contains information about how many test cases were run, and pass/fail results for each test case.
- HTML report: Produced alongside the TRX log, this report contains the verification status (pass/fail) of the protocol requirements.
- XML log: A log created by the Protocol Test Framework (PTF).
Note: The Protocol Test Framework (PTF) is one of the client components of the test environment described in the first section of this post. The PTF is designed to provide support for the Microsoft Protocol Test Suites for Office and Windows Protocol Interoperability testing. PTF enables you to develop custom Test Suites for your implementation, and provides an interface called ITestSite to represent the environment for Test Case execution.
Figure 3: A sample TRX log file
Where to Get the Test Suites
You can download the Exchange, Web Application Open Platform Interface (WOPI), and SharePoint Test Suites from the Test Suites page on Microsoft Connect. This site is also where you can download the Protocol Test Framework (PTF) and Message Analyzer.
Currently, the following Test Suites are available for download.
Note: More Test Suites are always being added, so this list may become out of date. Visit the Microsoft Connect Test Suites page for the most current selection.
| Test Suite | Applicable protocols | Protocol Name |
| Exchange ActiveSync (EAS) | MS-ASAIRS | Exchange ActiveSync: AirSyncBase Namespace Protocol |
| MS-ASCAL | Exchange ActiveSync: Calendar Class Protocol | |
| MS-ASCMD | Exchange ActiveSync: Command Reference Protocol | |
| MS-ASCNTC | Exchange ActiveSync: Contact Class Protocol | |
| MS-ASCON | Exchange ActiveSync: Conversations Protocol | |
| MS-ASDOC | Exchange ActiveSync: Document Class Protocol | |
| MS-ASDTYPE | Exchange ActiveSync: Data Types | |
| MS-ASEMAIL | Exchange ActiveSync: Email Class Protocol | |
| MS-ASHTTP | Exchange ActiveSync: HTTP Protocol | |
| MS-ASNOTE | Exchange ActiveSync: Notes Class Protocol | |
| MS-ASPROV | Exchange ActiveSync: Provisioning Protocol | |
| MS-ASRM | Exchange ActiveSync: Right Management Protocol | |
| MS-ASTASK | Exchange ActiveSync: Tasks Class Protocol | |
| MS-ASWBXML | Exchange ActiveSync: WAP Binary XML (WBXML) Algorithm | |
| Exchange Web Services (EWS) | MS-OXWSATT | Attachment Handling Web Service Protocol |
| MS-OXWSBTRF | Bulk Transfer Web Service Protocol | |
| MS-OXWSCONT | Contacts Web Service Protocol | |
| MS-OWXWSCORE | Core Items Web Service Protocol | |
| MS-OXWSFOLD | Folders and Folder Permissions Web Service Protocol | |
| MS-OXWSMG | Email Message Types Web Service Protocol | |
| MS-OXWSMTGS | Calendaring Web Service Protocol | |
| MS-OXWSSYNC | Mailbox Contents Synchronization Web Service Protocol | |
| MS-OXWSTASK | Tasks Web Service Protocol | |
| Exchange Remote Procedure Call (RPC) | MS-OXCFOLD | Folder Object Protocol Specification |
| MS-OXCFXICS | Bulk Data Transfer Protocol Specification | |
| MS-OXCMSG | Message and Attachment Object Protocol Specification | |
| MS-OXCPERM | Exchange Access and Operation Permissions Protocol Specification | |
| MS-OXCPRPT | Property and Stream Object Protocol Specification | |
| MS-OXCROPS | Remote Operations (ROP) List and Encoding Protocol Specification | |
| MS-OXCRPC | Wire Format Protocol Specification | |
| MS-OXCSTOR | Store Object Protocol Specification | |
| MS-OXCTABL | Table Object Protocol Specification | |
| MS-OXNSPI | Exchange Server Name Service Provider Interface (NSPI) Protocol | |
| Exchange MAPI | MS-OXCMAPIHTTP | Messaging Application Programming Interface (MAPI) Extensions for HTTP |
| MS-OXCNOTIF | Core Notification Protocol | |
| MS-OXORULE | Email Rules Protocol | |
| SharePoint Server Protocols | MS-ADMINS | Administration Web Service Protocol |
| MS-AUTHWS | Authentication Web Service Protocol | |
| MS-COPYS | Copy Web Service Protocol | |
| MS-CPSWS | SharePoint Claim Provider Web Service Protocol | |
| MS-DWSS | Document Workspace Web Service Protocol | |
| MS-LISTSWS | Lists Web Service Protocol | |
| MS-MEETS | Meetings Web Service Protocol | |
| MS-OFFICIALFILE | Official File Web Service Protocol | |
| MS-OUTSPS | Lists Client Sync Protocol | |
| MS-SHDACCWS | Shared Access Web Service Protocol | |
| MS-SITESS | Sites Web Service Protocol | |
| MS-VERSS | Versions Web Service Protocol | |
| MS-VIEWSS | Views Web Service Protocol | |
| MS-WDVMODUU | Office Document Update Utility Extensions | |
| MS-WEBSS | Webs Web Service Protocol | |
| MS-WSSREST | ListData Data Service Protocol | |
| MS-WWSP | Workflow Web Service Protocol | |
| SharePoint File Sync and WOPI Protocols | MS-WOPI | Web Application Open Platform Interface Protocol |
| MS-FSSHTTP | File Synchronization via SOAP over HTTP Protocol | |
| MS-FSSHTTPB | Binary Requests for File Synchronization via SOAP Protocol |
Resources and More Information
The following resources provide more information on Test Suites and Open Specifications.