[Previous entry: "A bug in fread() could lead to a buffer overflow vulnerability"] [Next entry: "I'm wrong. Supposedly..."]

04/18/2008: "Code Commandos, SDL and Metrics"

Some people are arguing that you shouldn't/can't use publicly disclosed vulnerability counts as a metric to the success of the SDL. The major sticking point is best codified with a quote from Pete Lindstrom's blog:

"Microsoft has systematically hired and/or contracted with every one of their most vocal critics (and most seasoned bugfinders) to do the work behind the scenes and they don't count those vulns!"

This quote suggests a lack on understanding about what really goes on "behind the scenes" so this blog posting should hopefully make things clearer. Firstly, some definitions: a "pre-release product" is a product not yet shipped and at any given moment is under-going some part of the SDL process. A "shipped product" is a product that has undergone the SDL processes and whose code has been finalized. Now, the main aim of Microsoft's Security Development Lifecycle (SDL) is to deliver to their customers shipped products whose code is as robust and secure as possible. As part of this process there is a Final Security Review (FSR). This is where Microsoft hire an external 3rd party that are sent to give the pre-release product and code a beating up - code commandos if you will This FSR serves two important purposes from my point of view. Firstly, for each problem found questions are asked as to how the problem got through as far as it did. The answers to these questions allow the tools and processes behind SDL to be refined and improved meaning that, next time around, instances of the same class of issue shouldn't get so far again. The second purpose of the FSR, and most obvious, is to ensure that there are fewer security problems in the final, shipped product because they are found and fixed. In other words, the use of a FSR, which is part of SDL, means that the final product is more secure (by which I mean has less security flaws) than a hypothetical product that did not undergo the FSR.

Now down to counting vulnerabilities as a metric...

You only start counting bugs when the code has been finalized and the product ships. You don't start counting before - as it's a pre-release product undergoing security review. This should be obvious. So even if during the FSR of a given product a gazillion bugs were found then, with regard to the vulnerability count, this is entirely irrelevant because these bugs were in a pre-release product and not a shipped product and were found as part of the SDL process.

Now what happens if one of Microsoft's hired code commandos finds a flaw in a shipped product? I think this question is the root cause of Pete's and people of his ilk's lack of trust when it comes to vulnerability counting as a metric. His assumption is that these are quietly swept under the carpet and therefore not counted. Let me put him straight. This does not happen.

The reason it doesn't happen is two fold - firstly, NGSSoftware don't want to be or be seen as Microsoft lackeys and secondly Microsoft doesn't want to, or be seen to be, hiring lackeys. In the former case it would affect our independent status and in the latter it would undermine Microsoft's attempts to improve their software's security and convince their customers that this is the case. To that end, NGSSoftware asked for and was whole-heartedly given the right to continue to research and publish flaws in Microsoft products - shipped products - even those we have done work on when they were pre-release. Here are some MS alerts from the past two years that NGS have been credited in:

http://www.microsoft.com/technet/security/bulletin/ms06-nov.mspx
http://www.microsoft.com/technet/security/bulletin/ms06-aug.mspx
http://www.microsoft.com/technet/security/bulletin/ms06-jun.mspx
http://www.microsoft.com/technet/security/bulletin/ms06-mar.mspx
http://www.microsoft.com/technet/security/bulletin/ms06-jan.mspx
http://www.microsoft.com/technet/security/bulletin/ms07-apr.mspx
http://www.microsoft.com/technet/security/bulletin/ms07-jul.mspx
http://www.microsoft.com/technet/security/bulletin/ms07-dec.mspx
http://www.microsoft.com/technet/security/bulletin/ms08-feb.mspx

Furthermore, we don't do any MS sponsored reviews after the product has been shipped so if we find a security flaw at this stage there will be an alert.

Let me state this as clearly as I can. If I found a flaw in SQL Server 2005 tomorrow, I would report it to secure@microsoft.com, it would be fixed and an alert would be publicly issued. If SQL Server 2005 wasn't so secure, I would find one just to prove this - but I can't - because it is secure and SDL, every part of it, is responsible. So, when it comes to using publicly reported vulnerabilities in shipped products as a metric for SDL, rest assured, the results aren't as skewed as some would have you believe. I personally think, and I hope this blog posting will help convince others, that as any measure goes in an imperfect world, this one's "pretty good".