According to Steve McConnell in Code Complete (data from 1975-1992) most bugs don’t take long to fix. About 85% of errors can be fixed in less than a few hours. Some more can be fixed in a few hours to a few days. But the rest take longer, sometimes much longer – as I talked about in an earlier post.
Given all of these factors and uncertainty, how to you estimate a bug fix? Or should you bother?
Block out some time for bug fixing
Some teams don’t estimate bug fixes upfront. Instead they allocate a block of time, some kind of buffer for bug fixing as a regular part of the team’s work, especially if they are working in time boxes. Developers come back with an estimate only if it looks like the fix will require a substantial change – after they’ve dug into the code and found out that the fix isn’t going to be easy, that it may require a redesign or require changes to complex or critical code that needs careful review and testing.
Use a rule of thumb placeholder for each bug fix
Another approach is to use a rough rule of thumb, a standard place holder for every bug fix. Estimate ½ day of development work for each bug, for example. According to this post on Stack Overflow the ½ day suggestion comes from Jeff Sutherland, one of the inventors of Scrum.
This place holder should work for most bugs. If it takes a developer more than ½ day to come up with a fix, then they probably need help and people need to know anyways. Pick a place holder and use it for a while. If it seems too small or too big, change it. Iterate. You will always have bugs to fix. You might get better at fixing them over time, or they might get harder to find and fix once you’ve got past the obvious ones.
Or you could use the data earlier from Capers Jones on how long it takes to fix a bug by the type of bug. A day or half day works well on average, especially since most bugs are coding bugs (on average 3 hours) or data bugs (6.5 hours). Even design bugs on average only take little more than a day to resolve.
Collect some data – and use it
Steve McConnell, In Software Estimation: Demystifying the Black Art says that it’s always better to use data than to guess. He suggests collecting time data for as little as a few weeks or maybe a couple of months on how long on average it takes to fix a bug, and use this as a guide for estimating bug fixes going forward.
If you have enough defect data, you can be smarter about how to use it. If you are tracking bugs in a bug database like Jira, and if programmers are tracking how much time they spend on fixing each bug for billing or time accounting purposes (which you can also do in Jira), then you can mine the bug database for similar bugs and see how long they took to fix – and maybe get some ideas on how to fix the bug that you are working on by reviewing what other people did on other bugs before you. You can group different bugs into buckets (by size – small, medium, large, x-large – or type) and then come up with an average estimate, and maybe even a best case, worst case and most likely for each type.
Use Benchmarks
For a maintenance team (a sustaining engineering or break/fix team responsible for software repairs only), you could use industry productivity benchmarks to project how many bugs your team can handle. Capers Jones in Estimating Software Costs says that the average programmer (in the US, in 2009), can fix 8-10 bugs per month (of course, if you’re an above-average programmer working in Canada in 2012, you’ll have to set these numbers much higher). Inexperienced programmers can be expected to fix 6 a month, while experienced developers using good tools can fix up to 20 per month.
If you’re focusing on fixing security vulnerabilities reported by a pen tester or a scan, check out the remediation statistical data that Denim Group has started to collect, to get an idea on how long it might take to fix a SQL injection bug or an XSS vulnerability.
So, do you estimate bug fixes, or not?
Because you can’t estimate how long it will take to fix a bug until you’ve figured out what’s wrong, and most of the work in fixing a bug involves figuring out what’s wrong, it doesn’t make sense to try to do an in-depth estimate of how long it will take to fix each bug as they come up.
Using simple historical data, a benchmark, or even a rough guess place holder as a rule-of-thumb all seem to work just as well. Whatever you do, do it in the simplest and most efficient way possible, don’t waste time trying to get it perfect – and realize that you won’t always be able to depend on it.
Remember the 10x rule – some outlier bugs can take up to 10x as long to find and fix than an average bug. And some bugs can’t be found or fixed at all – or at least not with the information that you have today. When you’re wrong (and sometimes you’re going to be wrong), you can be really wrong, and even careful estimating isn’t going to help. So stick with a simple, efficient approach, and be prepared when you hit a hard problem, because it's gonna happen.
Interesting post. If bug estimation is constantly difficult then I believe that there is a more fundamental issue in process or architecture
ReplyDeleteMaybe requirements capture is not effective and there is lots of rework required or the architecture makes it easy to make mistakes or hard to fix bugs, or both.
I think most teams are too quick to accept large amounts of bugs or simply blame the engineers because improvement on a larger scale is hard.
@Andy, I looked at different factors that can affect how long it takes to fix a bug. Two of the most important factors are the experience of the developer, and the structure/quality of the code. So architecture definitely comes into play here. If you're fixing a bug in a simple app that you're still working on, you should be able to tell quickly if it is trivial or it's something more fundamental that you missed in understanding the requirements or in architecture or design. The trivial ones are easy to estimate, or just to fix. The others, where you need to go back and re-design: not so much. As the situation gets more complicated, say a large enterprise system that is being used by lots of customers and hooked into lots of other systems, it can be much harder to understand what needs to be fixed and how long it might take.
ReplyDeleteHi Jim,
ReplyDeleteHow can you estimate bug fixing up front when you don't know what your bugs are and how many of them you have?
In any case, bug fixing is very tricky and it has some very bad effects on the overall schedule, especially if you have a sloppy team.
We have published something on this issue here: http://www.pmhut.com/a-bug-fix-cycle-at-project-end-is-good-for-everyone . Hope you'll get the chance to read it.
Very interesting. I'm actually thinking about such an approach for user stories also: look at your history and the distribution of what matters (like cycle time), and identify buckets to store your stories. Fitting stories into these buckets, ideally limited to 1, is estimating. Exceptions are exceptions, they're not such a big deal.
ReplyDelete