Security Innovation Security Engineering Study

The Security Innovation Security Engineering study,  Comparing Security in the Application Lifecycle - Microsoft and IBM Development Platforms Compared, is timely, given the emerging industry emphasis on integrating security in the life cycle. 

My favorite quote in the study is "The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found."  I think this reflects the fact we have more than 2,500 pages of security guidance (see Security Guidance, Security Engineering, Threat Modeling, and Improving Web Application Security) , and we've integrated our guidance into MSF/VS 2005 (see MSF/VS 2005 and p&p Integration.) 

The study was available from the MSDN Security DevCenter for a while but seems to have fallen off.  I've summarized the study here for quick reference:

Overview
Security Innovation evaluated the guidance and tools of Microsoft's and IBM's development platforms.  The study compared the support available to a development team via security guidance, documentation and security focused features in the life-cycle tool suites.  Gartner reviewed the approach.

Evaluation Criteria

• Coverage.  How well do the provided tools and guidance cover the key set of security areas?  
• Quality.  How effective and accurate are the tools and guidance?
• Visibility.  How easy is it to find the tools and guidance and then apply it to your security needs?
• Usability.  Are the tools and guidance precise, comprehensive and easy to use?

Ratings

• Outstanding: 81-100%
• Good: 61-80%
• Average: 41-60%
• Below Average: 21-40%
• Poor: 0-20%

Scorecard Categories

• Basic Platform Security.  When used in accordance with its documentation, a platform should be inherently secure.
• Platform Security Services.  A mature platform should include services that make it easier for developers to implement security features in their applications.
• Platform Security Guidance. A secure platform is much less useful if it lacks proper guidance.
• Software Security Engineering Guidance.  It is not possible to develop a secure application unless security is a focus during every phase of the development lifecycle.
• Security Tools.  A secure platform should include tools that make it easier to define, design, implement, test, and deploy a secure application.

Results of the Study

First, here's a couple key points, then the summaries are below:

• Microsoft beat IBM in every category around guidance.
• Microsoft beat IBM in three out of four categories around tools.

IBM

1. Platform Overall
   1. Overall: 36%
   2. Coverage: 62%
   3. Quality: 70%
   4. Visibility: 17%
   5. Usability: 72%
2. Platform Security Guidance
   1. Overall: 50%
   2. Coverage: 81%
   3. Quality: 85%
   4. Visibility: 17%
   5. Usability: 84%
3. Security Engineering Guidance
   1. Overall: 25%
   2. Coverage: 50%
   3. Quality: 64%
   4. Visibility: 17%
   5. Usability: 69%
4. Security Tools
   1. Overall: 32%
   2. Coverage: 55%
   3. Quality: 59%
   4. Visibility: 56%
   5. Usability: 63%

Microsoft

1. Platform Overall
   1. Overall: 67%
   2. Coverage: 88%
   3. Quality: 85%
   4. Visibility: 61%
   5. Usability: 80%
2. Platform Security Guidance
   1. Overall: 76%
   2. Coverage: 93%
   3. Quality: 85%
   4. Visibility: 67%
   5. Usability: 91%
3. Security Engineering Guidance
   1. Overall: 78%
   2. Coverage: 100%
   3. Quality: 89%
   4. Visibility: 67%
   5. Usability: 79%
4. Security Tools
   1. Overall: 47%
   2. Coverage: 71%
   3. Quality: 78%
   4. Visibility: 50%
   5. Usability: 68%

Quotes from the Study

• Microsoft’s overall rating of 67% reflects the impressive level of focus Microsoft has applied to application security in the past several years.
• IBM’s overall score of 36% is the result of a more disjointed approach to security. Security guidance is spread throughout the IBM web site and is difficult to discover.
• The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found.

More Information
For more information, see Comparing Security in the Application Lifecycle -
Microsoft and IBM Development Platforms Compared at Security Innovation's site.  They created four documents that take you through the details and results: Executive Summary, Research Overview, Full Detailed Reports and Results, and Methodology.

Comments

• Anonymous
  December 24, 2007
  Book building is art and science. I've built a few books over the years at patterns & practices.

• Anonymous
  December 24, 2007
  Book building is art and science. I've built a few books over the years at patterns & practices