Has Static Analysis reached its limits?

HP’s acquisition of Fortify this week (which I am sure will make some people at Kleiner Perkins happy) has made me think some more about static analysis and the state of the technology.

I'm not a technical expert in static analysis, and I have only a superficial understanding of the computer science behind static analysis and the ongoing research in this area. So this is based more on a customer’s view of developments in static analysis over the past few years.

One of the static analysis tools that we rely on is Findbugs, an open source static analyzer for Java. I’ve said it before and I’ll say it again: if you are writing code in Java and not using Findbugs, you are a fool. It’s free, it’s easy to install and setup, and it works. However, the Findbugs detection engine was last updated almost a year ago (Aug 2009) and the updates leading up to that were mostly minor improvements to checkers. Bill Pugh, the University of Maryland professor who leads the development of Findbugs, admitted back in 2009 at JavaOne that the analysis engine “continues to improve, but only incrementally”. At that time, he said that the focus for future development work was to provide improved management and workflow capabilities and tools integration for large-scale software development using Findbugs. This was driven by work that Professor Pugh did while a visiting scientist at Google, a major user of the technology.

Anyways, these improvements look to be about a year late (originally to be available in the fall of 2009) and there hasn’t been any updates on product plans on the findbugs mailing list for a while. But the direction (less work on the engine, more on tools) seems to be consistent with other static analysis suppliers. In the meantime, Findbugs continues to do the job – and hey, it’s free, who am I going to complain to even if I wanted to?

Coverity, another tool that we have used for a while, has reduced its investment in the core static analysis engine over the past couple of years. As I noted in a post last year, after receiving venture capital investment, Coverity went on a buying spree to fill out its product portfolio, and since then has focused on integrating these acquisitions and repackaging and re-branding everything. In the last major release (5.0) of its technology earlier this year, Coverity put a lot of effort into packaging, improving management and workflow capabilities and documentation – but there were only minor tweaks to the core static analysis engine. All nice and ready for the enterprise, all dressed up and ready for somebody to make them an offer. But with IBM and HP already matched up, who’s left to buy them? Symantec maybe?

Over the last year, Fortify has been filling out its product portfolio and working on integration with HP’s product suite leading up to the acquisition (much foreshadowing) and building an On Demand / Cloud / SaaS offering. Over the past year, Klocwork has integrated its static analysis tool with collaborative peer review and code refactoring tools (Klocwork Insight Pro) and improved its reporting – like some of the other tool providers, mapping problems against the CWE and so on. And Parasoft has been about providing an integrated toolset for a long time now – static analysis is only one small part of what they offer (if you can get it to work).

I haven’t done anything with Microsoft tools in a long time, so I don’t know what advances they are making with their static analysis tools like PREfast and FxCop, although they do have active research projects in static analysis and on debugging concurrent programs, all of which might be cool if we developed in .NET.

Now that Ounce Labs has become Rational Appscan something-or-other at IBM it will be difficult for anyone except IBM product specialists to keep up with its development, and most of IBM’s time and money for the next while will likely be spent on assimilating Ounce and getting its sales teams skilled up. IBM Research labs have been doing some advanced work on static analysis, mostly in the Java world, although it’s not clear how or if this work will make it into IBM’s product line, especially after the Ounce acquisition.

The most interesting development in the static analysis area recently is the O2 work that OWASP’s Dinis Cruz started while at Ounce Labs, to build more sophisticated security analysis, scripting and tracing capabilities on top of Ounce and other static (and dynamic) analysis engines. I haven’t had a chance to look at O2 yet in any depth - I am still not sure I entirely understand what it does and how it does what it does, and I am not sure it is intended to be used by mere mortals, but it looks like it is almost ready for prime time.

Even O2 isn’t about offering new advances in static analysis – at least from what I understand so far. It is about leveraging existing underlying static analysis technology and getting more out of it, building a technical security analyst’s workbench and tool platform.

All of this: the absence of innovative product announcements in static analysis; the investments made instead to fill out integrated product suites and create better management tools for larger companies, and the emphasis on selling value-add consulting services; are clear signs that the market has matured; that the underlying technology has reached, or is reaching, its limits.

Our experience has been that there are benefits to using static analysis tools, once you get through the initial adoption – which admittedly can be confusing and harder than it should be, and which I guess is one of the drivers for vendors to offer better management tools and reporting, to make it easier for the next guy. The tools that we use today continue to catch a handful of problems each week: a little sloppy coding here, a lack of defensive coding there, and a couple of simple, small mistakes. And every once in a while a real honest-to-goodness slap-yourself-in-the-head bug that made it though unit testing and maybe even code reviews. So as I have said before, the returns from using static analysis tools are real, but modest. And given where suppliers are spending their time and money now, it looks like we won’t be able to expect much more than modest returns in the future.