Friday, January 6, 2012

Static Analysis isn’t Development Testing

I constantly get emails from Static Analysis vendors telling me why I need to buy their technology. Recently I’ve been receiving emails explaining how my team can use static analysis tools to do impressive things like “test millions of complex lines of codes [sic] in minutes”.

Hold on now, pardner. Running static analysis tools against your code to enforce good coding practices and check for common coding mistakes and violations of security coding rules is a darn good idea. But this isn’t testing, and it’s not a substitute for testing.

Static Analysis tools like Coverity, Klocwork Insight or Klocwork Solo for Java Developers,Findbugs,HP Fortify,IBM’s Software Analyzer, Microsoft’s FXCop and Veracode’s on-demand static analysis checking are a step up from what compilers and IDEs check for you (although IDEs like IntelliJ IDEA include a good set of built-in static analysis code checkers). These tools should be part of your development and continuous integration environment.

They are fast and efficient at finding common coding mistakes and inconsistencies in large code bases. The kinds of mistakes that are easy to miss and hard to test for, and that are usually easy to fix. This makes code reviews more efficient and effective, because it lets reviewers focus on higher-level problems rather than low-level correctness. And they can identify areas in the code that need to be refactored and simplified, and areas of the code that may need more testing or deeper review – especially if the tool finds problems in code that is already “working”.

But static analysis isn’t the same as, or as good as, testing

Static analysis tools can’t test the code to validate that it does what it is supposed to do. That the features that the customer requested are all there and that the business logic is correct (unit testing and functional testing and acceptance testing). They can’t check that the interfaces of a module and its behavior is correct (component-level testing), or that the larger pieces of the system work correctly together and with other systems (system testing and integration testing). Or that the system will hold up under load (performance and stress testing and soak testing), or whether it is vulnerable to security attacks (pen testing), or that it people can actually understand how to use it (usability testing). Static analysis tools, even the best ones, are limited to finding “a narrow band of code-related defects”.

I appreciate that static analysis tool vendors want to find new ways to sell their solutions. Maybe this is why Coverity is now calling its static analysis toolset the “Industry’s First Development Testing Platform”. Their message seems to be that developers don’t test their code and won’t fix bugs found by testers “because of organizational silos and conflicting priorities”. So static analysis has to fill this perceived gap.

By improving integration with development tools like Eclipse and Visual Studio and making the static analysis checks run faster, Coverity has made it easier for developers to find and fix more coding bugs earlier – that’s good. But better packaging and workflow and faster performance doesn’t change what the tool actually does.

Development Testing or Developer Testing

Running a static analysis tool isn’t “Development Testing”. Development Testing or Developer Testing is what developers do to test their own code to see if it works: walking through their code in a debugger as it executes and making sure that it does what they expect, profiling it to make sure that performance doesn’t suck, writing automated unit tests and component tests to make sure that the code does what it is actually supposed to do.

Confusing static analysis tools with other automated testing tools is a bad idea. It misrepresents what static analysis tools can do. It could lead people to believe that static analysis is a substitute for testing. Bill Pugh, an expert in static analysis and the father of FindBugs, makes it clear that while static analysis tools definitely work, “testing is far more valuable than static analysis” at finding bugs. And security experts like Jeremiah Grossman at White Hat Security debunk the idea that static analysis is enough to check for security bugs or other kinds of bugs in a system.

Static analysis tools are getting better and easier to use, they’re getting faster and more accurate and they throw off less false positives than they did even a couple of years ago. Everyone should use them if they can find a tool that works well for them and that they can afford. If you’re in a Java shop for example, there’s no excuse not to be using Findbugs at least – it’s free and it works. My team uses multiple different static analysis tools at different stages of development and continuous integration, because different tools find different kinds of problems, and some tools take longer to run. These tools find real bugs and coding problems. But there’s too much hype in this market. In the land of software testing, static analysis tools don’t “reign king”.


Eric Friese said...


I very much agree that static analysis does not replace all of the different types of testing. That's just crazy. I would argue that static analysis is a type of testing. From my point of view, quality is a part of testing and I think static analysis falls into that category. Maybe I'm biased because I'm one of the no-good-dirty-vendors (HP), but I think static analysis is a form of testing that can complement other types of development testing.

BTW, really enjoy the blog.

Anonymous said...

Good write up. You might as well include XCode's static analysis engine in the mix there. If someone thinks SAST is a panacea and a replacement for DAST or good old external penetration testing of your applications, the might (sooner or later) be in for a surprise.

gorlok said...

Static Analysis is a helper for "white box review". It don' replace "white box review" or any kind of testing neither.

Anonymous said...

Testing is by definition a dynamic activity that needs a portion of code to be executer. Static analysis is not a replacement of testing but it can complements it

Pavan said...

Good article. Based on experience in software development and testing, I can say these static analysis tool really avoid coding mistakes which leads to exceptions.

Andrey Karpov said...

What do you think of on-the-fly mode, in the analyzer PVS-Studio?

Anonymous said...

Hi, completely disagree.

Something you do not find useless does not mean that it is useless.

Here, you are mixing the subjects. Testing has nothing to do with static analysis. No static analysis tool claim that they can replace testing, never heard one.

By testing, you can count the trees but could not see the forest. You even do not know that, nowadays, static analysis tools work multi platform, which means, they detect the problems in different layer interactions. How will you find this by testing? With how many scenarios?

You might choose not to use any static analysis tool but please avoid mixing the subjects..They -software testing and static analysis- are both important in their domain and they are different..

Jim Bird said...

@Anonymous, I am not saying that static analysis tools aren't useful - we use them and they do find problems. And I am not the one that is mixing static analysis and software testing. The point that I was trying to make is that static analysis shouldn't be thought of the same as testing. I am being critical of the vendor marketing hype where a developer running a static analysis tool is being branded "development testing" and statements like "In the land of software testing, static analysis reigns king" which mislead people on the value and use of static analysis tools. Just google the phrases and you'll see what I mean.

Site Meter