Coverity has published the 2008 edition of its Open Source Report. The report uses static code analysis on C, C++, and Java source code to determine the quality of the code. These reports are funded by the US Department of Homeland Security and supported by Stanford University, and are part of the US government’s Open Source Hardening Project. The report is based on over two years’ worth of data from Coverity Scan.55 Million lines of recurring code from 250 open source projects, which lead to 14238 analysis runs, resulting in almost 10 billion lines of code analysed. The core results are neatly summarised in the executive summary:
- The overall quality and security of open source software is improving – Researchers at the Scan site observed a 16% reduction in static analysis defect density over the past two years
- Prevalence of individual defect types – There is a clear distinction between common and uncommon defect types across open source projects
- Code base size and static analysis defect count – Research found a strong, linear relationship between these two variables
- Function length and static analysis defect density – Research indicates static analysis defect density and function length are statistically uncorrelated
- Cyclomatic complexity and Halstead effort – Research indicates these two measures of code complexity are significantly correlated to codebase size
- False positive results – To date, the rate of false positives identified in the Scan databases averages below 14%
The 2006 benchmark data shows that the participating open source project had roughly one static analysis defect per 3333 lines of code. Based on the most recent results, the participating projects now have roughly one static analysis defect per 4000 lines of code, a decrease of 16%. A few projects have even eliminated all static analysis defects: courier-maildir, curl, libvorbis, and vim. Each of these projects consists of 50000 lines of code or more.
How does this tool differ from good old Lint that’s available for C/C++?
From the little details I can gain from that website, it appears to be little more than a Lint-like tool. If that is the case, I do not see the point as there are hundreds of such programs around. See http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis for a list.
First, this tool is not only for C/C++. And they’re actively monitoring open source projects for defects, which is a good thing.
Anyways, you question is not much different from “Why do we need new editors while we have good old Vi?” (Actually vi question is more serious).
Why do we even need new news articles? The old ones are perfectly fine by me.
Coverity is vastly superior to any other static analysis tool on the market. It does inter-procedural analysis (following semantic paths across function and library calls), incremental analysis (only scans paths that changed since the last scan), concurrency checking, locking consistency, enforcement of arbitrary coding standards, and much more.
My favorite aspect of Coverity is the web-based interface that combines a syntax-highlighted, hyperlinked source browser with a step-by-step trace of the path leading to the selected defect inlined right into the code. As someone who has previously worked on integrating far less user-friendly static analysis tools into commercial development projects, Coverity is a thing of beauty.
Edited 2008-05-23 01:11 UTC
Thank you for that informative post. It’s a shame that I can’t mod you up after posting
If it does concurrency checking and locking consistency, that’s a very useful feature in my book.
I’m hoping we will see Haiku, syllable, reactos and other open source OS’s on the coverity scan.
Coverity offers the use of their (full-featured) tool free-of-charge to any open-source project on the condition that any bugs they find include an attribution (i.e. “Found using Coverity”) in their bug tracker.
It’s a pretty sweet deal, since Coverity can easily cost over $1 million USD for proprietary projects, depending on the size of the codebase. All that open-source projects have to do is take advantage of this mutually-beneficial arrangement. There’s no reason not to!
for example see for firefox:
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short…
I worked on an open-source project used extensively in the systems biology field. (Several major international research institutions are involved in its development.) I called Coverity and asked them if we could use their program for our project. They said they only allow a limited number of open-source projects to use their program for free.
Hello Samad,
I was sent a copy of your comment. I would like to know when you called Coverity, and which department, or who you spoke to, if you still have that information.
We don’t have a set limit on the number of projects included in the Scan, so either your call was before some of the project was planned out, or I need to do some internal communication to prevent an incorrect message like ‘a limited number of projects’ from being repeated again.
There are limited resources of course, since we don’t have an infinite number of build machines, but I’ve never turned a project away because of how many projects we have in the Scan already.
There is a backlog of requests for adding new projects, but to get in the queue, submit your project to [email protected], if you have not done so already. I don’t know the name of your project, so I can’t proactively check the queue before sending this reply.
coverity is another reason why the OS’s mentioned in the other post should be open sourced
So far everyone has commented on everything but the point of the article. Open source software is getting better.
Edited 2008-05-23 10:10 UTC
FOSS improving through evolution is inevitable so the only thing too comment on is everything else.
I couldn’t resist. Seriously though, I’m glad too see government funding going into FOSS so publicly. The comparison to previous audits shows improvement and all found bugs are reported back directly driving improvement; what’s not to like?
Not much to say. Generally all software is improving. Without knowing the /difference/ in rate of improvement all we can say is “Well good.”