Project Quant

by Peter Galli on April 15, 2009 05:54pm

I noticed today that my colleague Jeff Jones in the security group is launching a metric project that appears to be leveraging some of the good bits of open techniques. 

I touched base with him briefly and he gave me a little more information about Project Quant, which is being undertaken along with Securosis, an independent security research firm.

Project Quant will be working on the metrics of patch management and is as much an experiment of a new research process as it is one of security metrics, said Securosis founder Rich Mogull in a blog post.

"For this project Jeff wanted to be involved, but also asked for an open, unbiased model that will be useful to community-at-large (in other words, he didn't ask for a sales tool). Rather than us developing something back at the metrics lab, Jeff asked us to lead an open community project with as much involvement from the different corners of the industry as possible," Mogull said.

While he also acknowledged that it is risky for Securosis to allow direct involvement of the sponsor, the company is hoping that the process works the way it thinks it will and which also happens to match Microsoft's project goals.

So, this is what's expected to happen: a project landing site has been set up at Securosis that will contain all material and research as it is developed; every piece of research will be posted for public comment and no comments will be filtered unless they are spam, totally off topic, or personal insults.

All significant contributors will also be acknowledged in the final report, although there will be no financial compensation for contributors and the project itself will retain ownership rights. All material will also be released under a Creative Commons license, with spreadsheets released in both Excel and open formats.

"In short, we are developing all research out in the open, soliciting community involvement at every stage, making all the materials public, acknowledging contributors, and eventually releasing the final results for free and public use. The end goal of the project is to deliver a metrics model for patch management response to help organizations assess their costs, optimize their process, and achieve their business goals. Let us know what you think, even if you think we're just full of it," Mogull said.

For his part, Jones told me that while he has been zealous in past reports about using repeatable methodologies, pointing to his source of public data, and outlining his assumptions step-by-step, he would like to take transparency one step further by developing models and methodologies first, in an open and transparent manner, so that everyone can agree on the pros and cons before the methodologies are applied.

"I think being completely open and transparent will help credibility since, similar to open source, everyone can scrutinize every step of the analysis ... creating open models and potentially getting community involvement just seems to be the right process," he says.

I plan to interview him at greater length in the next few weeks, so look for a follow-up blog then.