[Previous entry: "Code Commandos, SDL and Metrics"] [Next entry: "Citius, Altius, Fortius!"]

04/20/2008: "I'm wrong. Supposedly..."

I've been wrong in the past on a few occasions but never until last Firday was I wrong so honouredly. For a brief moment in Internet time I was wrong according to Ryan's facebook status. [Thanks, Steve ]

But I wasn't wrong. With all due respect to Ryan he misunderstood my post, and in this case, is the person who is wrong. He thought my post was defending public vulns counts as a true and accurate measure of SDL's success. My post wasn't about this at all and was simply a correction of something Pete said which I think know is not true. Boiling it down to the basics, as part of a much larger post, Pete said that Microsoft have paid the top researchers for their permanent silence which is why he thinks the public vuln count is down. My post said that this wasn't the case and that NGS can and do still release advisories - refer to both posts for the gory details

So I wasn't wrong after all, but anyway, let's tackle the larger issue: using public vuln counts as a metric for the success of SDL - can we or can't we - should we or shouldn't we?


To me the guts of these question are, at release time how many bugs are in the software? More or less than one would expect?

Here's a potentially true or false statement:

(1) Public vuln count is down meaning that there are (2) less bugs in release-day code meaning that (3) SDL works.

For those that think this is false it's because point 2 doesn't necessarily follow from point (1). For Pete, it doesn't follow because he thinks the researchers have been gagged. Even if this were true, which it's not (see my post), there's a problem - remembering that the vuln count is down what happened to the flaws that the researchers found but kept quiet? Either they were fixed or they're still in the code. If the latter, then why haven't they been found by other good researchers who baulk at the very idea of working for Microsoft and would love to see nothing more than Microsoft being embarrased or by made a name for themselves by getting out an advisory or two or sold them to Verisign or Tippingpoint's ZDI? Given that this hasn't happened we must assume the bugs supposedly found by gagged researchers were fixed silently and therefore the bugs are no longer in the code. This bring us to silent fixes. Silent fixes are another reason why some people (Ryan for example) think point (2) above doesn't necessarily follow point (1). We all know that given some patched software and some unpatched software hackers have learned how to find the bugs that have been silently fixed. A really juicy flaw would soon be found and end up in Metasploit or Canvas and I'm sure some modules have come through this route but nowhere near the numbers of flaws we'd need to make up the difference given the current public vuln counts.

Here's another thing about silent fixes and quoting from an email from Ryan: "we know [silent fixes] exist because the SDL requires that SWI [the MS internal security team] looks in adjoining areas for code dependencies and those are fixed but never documented [publicly]."

This is true. Whilst Ryan doesn't think it, to me this doesn't say anything bad about using public vuln counts as a metric for measuring the success of SDL; in fact it says something good about it. If SDL dictates that when a new bug is found externally, SWI looks for similar flaws then this surely leads to more bugs being found internally, meaning less are found externally, meaning the potential public vuln count diminishes. QED

But seriously, and wrapping this up for good (at least from my perspective), I don't claim public vuln counting is an accurate metric for measuring the success of SDL. It may or may not be accurate. However it is a measure nonetheless and shows the direction of the trend. There is an expectation that software developed under SDL will make the code in the final released product more secure and have less security bugs. Given that there are less bugs it is reasonable to assume that the public vuln count will be less, too. Lastly, let's consider this: if the public vuln count was up, even marginally, you could bet that everyone would be screaming from the rooftops that SDL was a failure. Given that most people (even Pete and Ryan) think SDL was a success, why is it so hard to believe the opposite?