Practice and Theory of Security Reviews -- Part 3

Problem introduction and disclaimer.

Security Review Heuristics Zoo. 

Part 3 -- Reflections

Or rather a few closing notes...

Can you quantify "product security"?

Usually when people start talking about "X being 23% more secure than Y" I just snort. However, with the notion of features interaction complexity, we can at least try to approach that problem from a quantitative side.

The idea is to assign a numeric security measure L to a product if all features interactions up to the complexity level L for most industry-recognized representations of the product have been analyzed and checked for security. Yes, "most representations" is not very precise, but as in practice there are not so many of them (about 10?), this still may offer a workable yardstick for a quantitative assessment. The scale would look then something like this:

Level 0. No systematic security work has been done even with individual features. Well-recognized security industry standards are likely violated (e.g., credit card numbers are sent over WiFi in clear). A product almost certainly contains glaring security flaws easily discovered by non-professionals.

Level 1. Security coverage has been achieved for isolated features of a product. It is probably free from the most common security failures associated with its domain of operation. No pairwise component interactions have been seriously considered though, so security weaknesses still arise in corner customer scenarios; those would require some basic level of security expertise to recognize them. (e.g., a customer is authenticated at the product's Web site, but critical operations do not verify that the Authentication has not expired yet, so a CSRF is considerable).

Level 2. All elements and their pairwise interactions are covered for most of industry recognized ways to re-factor a product into elements. Theoretically, a well-done Threat Modeling can achieve that at least from the design standpoint. Finding a security hole would require pushing a product very far into an unexpected state. Most pentesing vendors would still be able to discover scattered holes in such a product with enterprise-level funding (e.g., via fuzzing).

Level 3. Exhaustive verification of all three-way interactions is conducted for multiple re-projections. A product is solid against most flavors of nearly all industry-recognized attacks. Finding a new security hole in it costs O(N⁴) effort with respect to the number of elements logically comprising a product. In practice this would require at least billions of attempts. State-sponsored agencies can probably still run effective security offence against products at that security level, with costs expressed in 8-9 digit figures.

Level 4. All 4-way interactions have been verified as safe. I would say this is what a military-grade security *should* look like. Compromising such a product would take computational resources of leading world superpowers and cost significant chunks of their yearly GDPs. Security holes at this level would literally be solitary, unique in nature, and kept under high secrecy as important military assets.

And so on.

 

Security Career Roles

Every company needs "security people", but whom exactly? While most security specialists can wear multiple hats at the same time, they also have their natural preferences. And those preferences could be classified within the definitions of this article as following:

• Security Auditor. They work "horizontally", quickly and effectively scaling across largest products. They typically are familiar with all classes of security attacks, are up-to-date on the most recent news in that space, and easily pass an interview on that subject. They may not always be exceptionally technical, and probably won't spend too much time looking for new unusual attacks. But they are good at grasping a large picture and factoring it into a set of actionable work items, so if you need someone to manage a security compliance program for your product, an Auditor would be an ideal choice. They tend to be process-oriented and naturally plug in into existing workflows.
• Security Reviewer. They prefer to apply more computationally expensive techniques like Threat Modeling or Scenario Fuzzing to discover security holes. They like analyzing complex interactions, looking for the unknown, creatively discovering new classes of attacks. They would have sufficient technical depth to talk to developers in their language, and are good at quickly telling significant security issues from minor ones. They would do great job owning a few critical components within the overall architecture. But they would be bored to death running a "security compliance" program or pushing a team through 25 security checkboxes. They treat security as science rather than a production line and would be outraged by a request "to sign off" on anything they haven't spent at least a day scrutinizing and that has not had at least a month of security work behind the belt. Surprisingly, they may be unaware of some popular security keywords or famous recent attacks. That's because they simply don't care – they can re-invent that stuff on a daily basis as needed :)
• Security SME uses exceptional knowledge of some technology to help other people navigate within it. Need to make a call on the suitability of Elliptic Curve encryption for your product? Wondering what an obscure DACL means? Ask an SME. They will answer all these questions, precisely and probably immediately. Having an SME in the team is really, really great. However, it takes years to grow (or even hire) one.
• Pattern Discoverer. Through years of working with security issues, discovers patterns in them and then completely eliminates whole classes of issues with the (proposed) automatic tools. They may not even be "security experts" per se, as they enjoy solving any classes of challenging problems using the same approach. Alone, they may achieve what dozens of people could've not achieved in years. On the darker side, they may easily spend years without any immediately visible results while the product would be in a need for a regular security attendance at simpler complexity levels.