Thursday, August 2, 2012

Fixing Bugs – there’s no substitute for experience

We've all heard that the only way to get good at fixing bugs in through experience – the school of hard knocks. Experienced programmers aren’t afraid, because they’ve worked on hard problems before, and they know what to try when they run into another one – what’s worked for them in the past, what hasn’t, what they’ve seen other programmers try, what they learned from them.

They’ve built up their own list of bug patterns and debugging patterns, and checklists and tools and techniques to follow. They know when to try a quick-and-dirty approach, use their gut, and when to be methodical and patient and scientific.They understand how to do binary slicing to reduce the size of the problem set. They know how to read traces and dump files. And they know the language and tools that they are working with.

It takes time and experience to know where to start looking, how to narrow in on a problem; what information is useful, what isn’t and how to tell the difference. And how to do all of this fast. We’re back to knowing where to tap the hammer again.

But how much of a difference does experience really make?

Steve McConnell’s Code Complete is about programmer productivity: what makes some programmers better than others, and what all programmers can do to get better. His research shows that there can be as much as a 10x productivity difference in the quality, amount and speed of work that top programmers can do compared to programmers who don't know what they are doing.

Debugging is one of the areas that really show this difference, that separates the men from the boys and the women from the girls. Studies have found a 20-to-1 or even 25-to-1 difference in the time it takes experienced programmers to find the same set of defects found by inexperienced programmers. That’s not all. The best programmers also find significantly more defects and make far fewer mistakes when putting in fixes.

What’s more important: experience or good tools?

In Applied Software Measurement, Capers Jones looks at 4 different factors that affect the productivity of programmers finding and fixing bugs:

  1. Experience in debugging and maintenance
  2. How good – or bad – the code structure is
  3. The language and platform
  4. Whether the programmers have good code management and debugging tools – and know how to use them.

Jones measures the debugging and bug fixing ability of a programmer by measuring assignment scope - the average amount of code that one programmer can maintain in a year. He says that the average programmer can maintain somewhere around 1,000 function points per year – about 50,000 lines of Java code.

Let’s look at some of this data to understand how much of a difference experience makes in fixing bugs.

Inexperienced staff, poor structure, high-level-language, no maintenance tools

Worst Average Best
150 300 500

Experienced staff, poor structure, high-level language, no maintenance tools

Worst Average Best
1150 1850 2800

This data shows a roughly 20:1 difference between experienced programmers and inexperienced programmers, on teams working with badly structured code and without good maintenance tools. Now let’s look at the difference good tools can make:

Inexperienced staff, poor structure, high-level language, good tools

Worst Average Best
900 1400 2100

Experienced staff, poor structure, high-level language, good tools

Worst Average Best
2100 2800 4500

Using good tools for code navigating and refactoring, reverse engineering, profiling and debugging can help to level the playing field between novice programmers and experts.

You’d have to be an idiot to ignore your tools (debuggers are for losers? Seriously?). But even with today’s good tools, an experienced programmer will still win out – 2x more efficient on average, 5x from best to worst case.

The difference can be effectively infinite in some cases. There are some bugs that an inexperienced programmer can’t solve at all – they have no idea where to look or what to do. They just don’t understand the language or the platform or the code or the problem well enough to be of any use. And they are more likely to make things worse by introducing new bugs trying to fix something than they are to fix the bug in the first place. There’s no point in even asking them to try.

You can learn a lot about debugging from a good book like Debug It! or Code Complete. But when it comes to fixing bugs, there’s no substitute for experience.


Lukas Eder said...

I'm always surprised by this tremendous difference in productivity, myself. However, I wouldn't blame it only on the fact whether a programmer is "experienced" or not. There is a certain amount of cognitive/reflective capability required for understanding the nature of bugs or code in general. Not everyone has it to the same extent, and it cannot really be trained. Having this capability, of course, will lead then to faster acquisition of experience.

Unknown said...

When I see the symptoms I can often tell what kind of bugs there is.
When I know the code base I can often tell where to start looking.
When I know the programmer working on a feature I can often tell what problems you will find before they appear.

There is a point when experience and intuition are one and the same. This is true of anyone in any field that dedicated his life to something.

Great article!

Site Meter