SwiftTest Customer Case Study
"Realism and performance are why our customers choose us. Without the Microsoft storage protocol documentation, we would have had to guess or learn from observing behavior what the SMB protocols do. The Microsoft protocol documentation, Network Monitor and early access to Microsoft implementations really helped. With the documents, we know how to implement CIFS/SMB protocols to represent accurate, real-world behavior." -Peter Murray, Product Manager, SwiftTest
SwiftTest (renamed Load DynamiX and merged with Virtual Instruments in 2016)
https://www.virtualinstruments.com/enterprise/
Country or region: USA
Industry: Network Storage and Application Testing
The Company
SwiftTest delivers powerful and efficient commercial Internet Protocol (IP) network testing tools that take the burden of IP storage test tool development off of in-house engineers and quality assurance teams.
The appliances, software and templates offered by SwiftTest help storage vendors, service providers and large enterprises test the functionality and scalability of their systems. With deep expertise in IP networks, testing, and the storage industry, SwiftTest has gained traction in a variety of market segments related to storage, applications and networking.
The Business Problem
Growth of the storage industry is accelerating. Equipment vendors, service providers and enterprises are introducing storage devices and technologies at a furious pace. Testers are racing to implement test suites to validate and stress devices and systems in the shortest time possible; in the most diverse environments imaginable. Likewise, support organizations must be able to re-create customer issues repeat-ably and reliably.
Validating storage equipment is challenging. Typically, testing is time and resource intensive. SwiftTest customers had found that their existing testing capabilities weren't fully supporting the depth and breadth of their use cases. Relying on home-grown, open source or vendor-specific test tools, they found that most of these tools were incomplete and required extensive maintenance whenever changes occurred to the protocols.
Tools that embed deep technical knowledge in testing a protocol are invaluable to testers. Most existing tools were not developed for large-scale performance testing nor would they scale appropriately for the cloud. Alternatively, using a multitude of general-purpose computers is expensive, complex and requires a large investment in hardware, software, licenses, development and maintenance.
"We're growing fast, and need test tools beyond what we can do in-house. We found SwiftTest to be better than any other solution we looked at and the SwiftTest 3000 is more than filling our needs. Particularly with the depth and maturity of their CIFS implementation, we can expand into new markets and support bigger customers," said Chris Siakos, CTO of Exinda.
The Product
The SwiftTest 3000 and SwiftTest 5000 appliance and software solution helps customers simulate storage and validate the capabilities of data storage systems, network devices and large file systems. SwiftTest products deliver high-capacity, highly-configurable traffic even for demanding devices.
SwiftTest appliances help users get to the specifics of misbehavior or bottlenecks found during testing with sophisticated, comprehensive reporting. Detailed reports highlight the impact on throughput caused by various combinations of features, shortening test cycles and increasing customer satisfaction.
SwiftTest customers include IBM, Riverbed, F5 Networks, Exinda, BlueArc (Hitachi), Pillar Data Systems (Oracle) and Avere systems. These customers use SwiftTest to simulate large numbers of users accessing storage through a variety of protocol formats ranging from Common Internet File System (CIFS) to Server Message Block (SMB and SMB2), Network File System (NFS), internet Small Computer System Interface (iSCSI) and Hypertext Transfer Protocol (HTTP). They appreciate the product's comprehensive reporting capabilities.
Flexibility through protocols enables customers to test a wide variety of storage products and systems. "With SwiftTest we can trace, track and compare anything — all the protocol-level details we want," said Steve Downer, Director of Test Engineering at Pillar Data Systems.
The products include a graphical interface called the SwiftTest Test Development Environment that enables easy test configuration with sample tests and wizards and real-time reporting which allows results drill-down to identify implementation errors and performance bottlenecks.
"Having SwiftTest has improved my job tenfold. I'm now free to do things way more important to BlueArc and more technically challenging than running around trying to start a bunch of clients simultaneously." -Nigel Stolting, US QA Manager at BlueArc
The Solution
SwiftTest used Microsoft storage protocol specifications that are available to the public on the Open Specifications site. SwiftTest also executed Microsoft's standard patent license agreement for those protocols available through its Open Specifications program on the Dev Center.
SwiftTest appliances were then built to test storage protocol functionality, performance and stress. SwiftTest delivers a commercial-grade storage protocol test tool for less than the cost of the hardware and software required to provide equivalent performance, while eliminating the need for customers to constantly develop and maintain test tools themselves.
Using the SwiftTest tools, customers can test individual protocol versions or multiple versions simultaneously to ensure proper functionality and performance under normal and stressful conditions. Repeatable tests with repeatable results shorten the testing cycle. The SwiftTest Test Development Environment can flexibly configure and run tests from one or many test ports and any port can be configured as client or server. This flexibility is enabled by SwiftTest's careful implementation of protocols.
For Storage Device Testing Environments - Companies simulating client traffic often need tens or even thousands of servers to emulate client load. Software used to generate the necessary load is often difficult to maintain, inflexible, and requires expensive licenses and hardware. With SwiftTest, servers can be replaced, reducing footprint, power and cooling costs, while simulating a variety of load types at high volume.
For Storage-Aware Network Device Testing Lab - SwiftTest appliances work with ports configured as either client or fileserver. Companies testing a storage-aware network device can replace clients or servers with a SwiftTest appliance.
For Service Providers and Enterprise Data Centers - SwiftTest supports flexible simulation of real-world traffic loads to help customers test storage equipment and systems at the most critical times — before purchase, before deployment and before changes. Storage testing services can be leveraged internally across organizational boundaries.
Microsoft Protocol Implementation Details
SwiftTest chose to implement Microsoft storage specifications to achieve accurate, high-performance implementations and get to market quickly. SwiftTest needed to be able to accurately simulate specific activities to identify stress points, such as permissions to open a file, and see how devices perform in high-failure scenarios which can drag server performance.
SwiftTest chose to implement Common Internet File System (CIFS), SMB and SMB2. Unlike other storage protocols, which are primarily described in Requests for Comment (RFCs); CIFS/SMB protocols are defined solely by Microsoft Open Specifications (found publicly on Microsoft Docs). These documents were used as a guideline to implement protocol requests to enable CIFS/SMB testing.
"Using internal research alone, it would have taken years to reverse-engineer the Microsoft protocols, and would still not help catch corner cases. The protocol specification documents were invaluable aids in increasing the accuracy of the products and decreasing time to market. The result is a superior test tool that makes it difficult to test storage protocols without it," stated Ivan Selivanov, Software Developer.
Licensing the Microsoft storage patents gives SwiftTest access to Microsoft key intellectual property for implementing the protocols.
Results
Competitively, SwiftTest is ahead of the market as a result of their protocol adoption strategy; simplifying implementation and shortening development time for themselves and their customers.
"We have been in situations where, if our competitors know we are there, they walk away. We can consistently get tests running on our appliances when our competitors are challenged," said Peter Murray, Product Manager at SwiftTest. "The thoroughness of our implementation has been a competitive advantage for us and people are looking at us to ensure they test properly," Murray adds.
Value to the Business
For anyone building or evaluating storage-aware devices, the value of a comprehensive CIFS/SMB protocol test solution includes:
Increased product quality: Customers seek protocol accuracy in order to test their own products. The documentation examples, tools and access to support engineers ensure that the CIFS/SMB protocols are implemented correctly so customers can test correctly.
Shorter development time: Simplified Microsoft Open Specifications have shortened SwiftTest's development time and led to increased sales.
Expanded target markets: A high-quality implementation helped gain traction and sales throughout the storage vendor community and as SwiftTest became better known by the vendors, they were able to expand sales into the enterprise market.
The SwiftTest development team has been satisfied with Microsoft's Open Specifications Support, stating "The Microsoft Protocol Documentation Team has done an excellent job updating documents to make them accurate and easy for SwiftTest to understand. They have also been invaluable in explaining the specifications and answering our questions."