Static Analysis isn’t Development Testing

I constantly get emails from Static Analysis vendors telling me why I need to buy their technology. Recently I’ve been receiving emails explaining how my team can use static analysis tools to do impressive things like “test millions of complex lines of codes [sic] in minutes”.

Hold on now, pardner. Running static analysis tools against your code to enforce good coding practices and check for common coding mistakes and violations of security coding rules is a darn good idea. But this isn’t testing, and it’s not a substitute for testing.

Static Analysis tools like Coverity, Klocwork Insight or Klocwork Solo for Java Developers,Findbugs,HP Fortify,IBM’s Software Analyzer, Microsoft’s FXCop and Veracode’s on-demand static analysis checking are a step up from what compilers and IDEs check for you (although IDEs like IntelliJ IDEA include a good set of built-in static analysis code checkers). These tools should be part of your development and continuous integration environment.

They are fast and efficient at finding common coding mistakes and inconsistencies in large code bases. The kinds of mistakes that are easy to miss and hard to test for, and that are usually easy to fix. This makes code reviews more efficient and effective, because it lets reviewers focus on higher-level problems rather than low-level correctness. And they can identify areas in the code that need to be refactored and simplified, and areas of the code that may need more testing or deeper review – especially if the tool finds problems in code that is already “working”.

But static analysis isn’t the same as, or as good as, testing

Static analysis tools can’t test the code to validate that it does what it is supposed to do. That the features that the customer requested are all there and that the business logic is correct (unit testing and functional testing and acceptance testing). They can’t check that the interfaces of a module and its behavior is correct (component-level testing), or that the larger pieces of the system work correctly together and with other systems (system testing and integration testing). Or that the system will hold up under load (performance and stress testing and soak testing), or whether it is vulnerable to security attacks (pen testing), or that it people can actually understand how to use it (usability testing). Static analysis tools, even the best ones, are limited to finding “a narrow band of code-related defects”.

I appreciate that static analysis tool vendors want to find new ways to sell their solutions. Maybe this is why Coverity is now calling its static analysis toolset the “Industry’s First Development Testing Platform”. Their message seems to be that developers don’t test their code and won’t fix bugs found by testers “because of organizational silos and conflicting priorities”. So static analysis has to fill this perceived gap.

By improving integration with development tools like Eclipse and Visual Studio and making the static analysis checks run faster, Coverity has made it easier for developers to find and fix more coding bugs earlier – that’s good. But better packaging and workflow and faster performance doesn’t change what the tool actually does.

Development Testing or Developer Testing

Running a static analysis tool isn’t “Development Testing”. Development Testing or Developer Testing is what developers do to test their own code to see if it works: walking through their code in a debugger as it executes and making sure that it does what they expect, profiling it to make sure that performance doesn’t suck, writing automated unit tests and component tests to make sure that the code does what it is actually supposed to do.

Confusing static analysis tools with other automated testing tools is a bad idea. It misrepresents what static analysis tools can do. It could lead people to believe that static analysis is a substitute for testing. Bill Pugh, an expert in static analysis and the father of FindBugs, makes it clear that while static analysis tools definitely work, “testing is far more valuable than static analysis” at finding bugs. And security experts like Jeremiah Grossman at White Hat Security debunk the idea that static analysis is enough to check for security bugs or other kinds of bugs in a system.

Static analysis tools are getting better and easier to use, they’re getting faster and more accurate and they throw off less false positives than they did even a couple of years ago. Everyone should use them if they can find a tool that works well for them and that they can afford. If you’re in a Java shop for example, there’s no excuse not to be using Findbugs at least – it’s free and it works. My team uses multiple different static analysis tools at different stages of development and continuous integration, because different tools find different kinds of problems, and some tools take longer to run. These tools find real bugs and coding problems. But there’s too much hype in this market. In the land of software testing, static analysis tools don’t “reign king”.