Zero Bug Tolerance Intolerance

It sounds good to say that you shall not and will not release code with bugs – that your team has “zero bug tolerance”. It makes for a nice sound bite. It sounds responsible. And it sounds right. But let’s look carefully at what this really means.

First, there are the logical arguments as to whether it is possible to build a perfect system of any level of complexity, and whether you can prove that a piece of nontrivial code is bug free. These are interesting questions, but I think the more important question is whether you should really try to.

I was guilty earlier in my career of pushing feature-set and schedule over quality, leaving too many bugs too late, and then having to deal with the aftermath. When I first started managing development projects a long time ago, I didn’t understand how “slowing down” to fix bugs would help get the project done faster. But I have learned, and I know (there is a difference between learning something and knowing something, really knowing it deep down) that fixing bugs helps keep costs down, and that it is possible to build good software quickly.

In Software Quality at Top Speed, Steve McConnell makes the case that short-changing design and writing bad code is stupid and will bite you in the ass, and that doing a responsible job on design and writing good code gets you to the end faster. At somewhere around the 90% defect removal rate you reach an optimal point:

“the point at which projects achieve the shortest schedules, least effort, and highest levels of user satisfaction” Capers Jones, Applied Software Measurement: Assuring Productivity and Quality, 1991.

Most teams don’t get close to the optimal point. But aiming beyond this, towards 100% perfection, causes costs to skyrocket; you quickly reach the point of diminishing returns. Diminishing returns in the pursuit of perfect software is explored further by Andy Boothe in The Economics of Perfect Software:

“For example, imagine a program has 100 bugs, and we know it will take 100 units of effort to find and fix all 100 of those bugs. The Law of Diminishing Returns tells us that the first 40 units of effort would find the first 70 bugs, the next 30 units of effort would find the next 20 bugs, and the next 30 units of effort would find the last 10 bugs. This means that the first 70 bugs (the shallow bugs) are cheap to find and squash at only 40 / 70 = 0.571 units of work per per bug (on average). The next 20 bugs (the deep bugs) are significantly more expensive at 30 / 20 = 1.5 units of effort per bug, and the final 10 bugs (the really deep bugs) are astronomically expensive at 30 / 10 = 3 units of effort per bug. The last 10 bugs are more than 5 times more time- and capital-intensive to eliminate per bug than the first 70 bugs. In terms of effort, the difference between eliminating most bugs (say 70%-90%) and all bugs is huge, to the tune of a 2x difference in effort and cost.

And in real life it’s actually worse than that. Because you don’t know when you’ve killed the last bug — there’s no countdown sign, like we had in our example — you have to keep looking for more bugs even when they’re all dead just to make sure they’re all dead. If you really want to kill all the bugs, you have to plan for that cost too.”

There's a cost to building good software

There’s a cost to put in place the necessary controls and practices, the checks and balances, and to build the team’s focus and commitment and discipline, and keep this up over time. To build the right culture, the right skills, and the right level of oversight. And there’s a cost to saying no to the customer: to cutting back on features, or asking for more time upfront, or delaying a release because of technical risks.

You also have to account for opportunity costs. Kent Beck and Martin Fowler have built their careers on writing high quality software and teaching other people how to do this. In Planning Extreme Programming they make it clear that it is important to write good software, but:

“For most software, however, we don’t actually want zero bugs. Any defect, once it is in there, takes time and effort to remove. That time and effort will take away from effort spent putting in features. So you have to decide what to do.”

That’s why I am concerned by right-sounding technical demands for zero bug tolerance. This isn’t a technical decision that can be made by developers or testers or project managers…or consultants. It’s bigger than all of them. It’s not just a technical decision – it’s also a business decision.

Long Tail of Bugs

Like many other problem spaces, the idea of the Long Tail also applies to bugs. There are bugs that need to be fixed and can be fixed right now. But there are other bugs that may never need to be fixed, or bugs that the customer may never see: minor bugs in parts of the system that aren’t used often, or problems that occur in unlikely configurations or unusual strings of events, or only under extreme stress testing. Bugs in code that is being rewritten anyways, or in a part of the system that is going to be decommissioned soon. Small cosmetic issues: if there was nothing better to do, sure you would clean this up, but you have something better to do, so you do this something better instead.

There are bugs that you aren’t sure are actually bugs, where you can’t agree on what the proper behavior should be. And there are bugs that can be expensive and time consuming to track down and reproduce and fix. Race conditions that take a long time to find – and may take a redesign to really fix. Intermittent heisenbugs that disappear when you look at them, non-deterministic problems that you don’t have the information or time to fix right now. And WTF bugs that you don’t understand and don’t know how to fix yet, or the only fix you can think of is scarier than the situation that you are already in.

Then there are bugs that you can’t fix yourself, bugs in underlying technology or third party libraries or partner systems that you have to live with or work around for now. And there are bugs that you haven’t found yet and won’t find unless you keep looking. And at some point you have to stop looking.

All of these bugs are sources of risk. But that doesn’t necessarily mean that they have to be fixed right now. As Eric Sink explains in My Life as a Code Economist, there are different questions that need to be answered to determine whether a bug should be fixed:

First there are the customer questions, basic questions about the importance of a bug to the business:

• Severity: when this bug happens, how bad is the impact? Is it visible to a large number of customers? Could we lose data, or lose service to important customers or partners? What are the downstream effects: what other systems or partners could be impacted, and how quickly could the problem be contained or repaired? Could this violate service levels, or regulatory or compliance requirements?


• Frequency: how often could this bug happen in production?

Then there are the developer questions, the technical questions about what needs to be done to fix the bug:

• Cost: how much work is required to reproduce, fix and test this bug - including regression testing? What about RCA, how much work should we do in digging deeper and fixing the root cause?


• Risk: what is the technical risk of making things worse by trying to fix it? How well do I understand the code? How much refactoring do I need to do to make sure that the code, and the fix, is clear? Is the code protected by a good set of tests?

For some bugs, the decision is dead easy: simple, stupid mistakes that you or somebody else just made as part of a change or another fix, mistakes that are found right away in testing or review, and should be fixed right away. You know what to do, you don’t waste time: you fix it and you move on.

But for other bugs, especially bugs discovered in existing code, it’s sometimes not so easy. Zero bug tolerance naively assumes that it is always good, it’s always right, to fix a bug. But fixing a bug is not always the right thing to do, because with any fix you run the risk of introducing new problems:

“the bugs you know are better than introducing new bugs”

Fred Brooks first pointed the regression problem out in The Mythical Man Month:

“…fixing a defect has a substantial (20-50%) chance of introducing another. So the whole process is two steps forward and one step back.”

The risk of introducing a bug when trying to fix another bug may have gone down since Fred Brooks’ time. In Geriatric Issues of Aging Software Capers Jones provides some less frightening numbers

“Roughly 7 percent of all defect repairs will contain a new defect that was not there before. For very complex and poorly structured applications, these bad fix injections have topped 20 percent”.

But the cost and risks are still real, and need to be accounted for.

Bars and Broken Windows

You, the team, the customer all need to agree on how high to set the bar, what kind of bugs and risks can be accepted. And then you have to figure out what controls, checks, practices, tools, skills, and how much more time you need to consistently hit that bar. How much it is going to cost. If you’re building safety-critical systems like the command software for the space shuttle
(what are we going to use as an example when the space shuttles stop flying?) the bar is extremely high, and so of course are the costs. If you’re a social media Internet startup, short on cash and time and big on opportunity, then the bar is as low as you can afford it. For the rest of us the bar is somewhere in between.

I get what’s behind Zero Bug Tolerance. I understand the “No Broken Windows” principle in software development: that if we let one bug through then what’s to stop the next one, and the next one and the next one after that. But it’s not that simple. No Broken Windows is about having the discipline to do a professional job. It’s about not being sloppy or careless or irresponsible. And there is nothing irresponsible about making tough and informed decisions about what can and should be fixed now.

Knowing when to stop fixing bugs, when you’ve reached the point of diminishing returns, when you should focus on more important work, isn’t easy. Knowing which bugs to fix and which ones not too, or which ones you can’t or shouldn’t fix now, isn’t easy. And you will be wrong sometimes. Some small problem that you didn’t think was important enough to look into further, some bug that you couldn’t justify the time to chase down, may come back and bite you. You’ll learn and hopefully make better decisions in the future.

That’s real life. In real life we have to make these kinds of hard decisions all of the time. Unfortunately, we can't rely on simple, 100% answers to real problems.

Sad State of Secure Software Maintenance

This is sad. No, it's not sad, it's sick. I'm looking for ideas and clear thinking about secure software maintenance. But I can't find anything beyond a couple of articles on Software Security in Legacy Systems by Craig Miller and Carl Weber at Cigital. I met Craig, he did some consulting work at a startup that I was running. He's a smart guy for sure. These papers offer some good advice to enterprises looking for where to start, how to get a handle on securing legacy systems and COTS packages. They are worth reading. But this is all I can find anywhere. And that's not good enough.

Most of us who make a career in software development will spend most of our careers maintaining and supporting software. If we're lucky, we will work on software that we had a hand in designing and writing; if we're not so lucky, software that we inherited from somebody else. Software that we don't understand and that we need to get control of.

Software maintenance is a risk management game. Understanding what's important to the business, trading off today's priorities with the long term view. Dealing with work that has to be done right now, what's needed for this customer, how fast can that change be done, what do we need to do to fix this bug. How much are we spending, and where can we save. And making sure that we're not sacrificing tomorrow: keeping the team together, keeping them focused and motivated, helping them moving forward. Keeping technical debt under control: taking on debt where it makes sense, paying it off when we can. And making sure that we're are always dealing with what's important: service levels to customers, reliability, security: protecting customer data.

There's more to secure software maintenance than running static analysis checks on the code and an occasional vulnerability scan and application pen test. And most teams aren't even doing this.

There's not enough smart people taking on the problems of how to manage software maintenance properly. And there's definitely not enough people thinking about software security and maintenance. Where to start, how much to spend, why, what's important, what the next steps should be, where do you get the most return. This has to change. It's too important to too many people. There's too much money being spent and wasted on doing a poor job at too many companies. There's too much at stake.

There's more to Managing Software Debt

If you work with a big system over time, you have to learn how to recognize and deal with technical debt, and how to keep debt under control. Like a married couple with 2 kids in college, paying off a mortgage and a couple of car loans, debt is a fact of life. You have to understand it, come to terms with it.

Debt can be a problem, but it is also a tool. As Steve McConnell explains in Managing Technical Debt there are perfectly valid reasons to take debt on intentionally and strategically. Some things should be done, but they don’t need to be done right now, and as long as you understand the costs and risks involved, you can make a trade off decision. But you will also take on debt unintentionally through honest mistakes (you didn’t or couldn’t know any better) and carelessness (you did know better, or you should have). This is inevitable.

There is a lot to successfully managing technical debt. I have to give props to Chris Sterling for trying to take on this problem on in his book Managing Software Debt.

Sterling looks at different aspects of what he calls software debt:

• Technical Debt: taking short cuts, folding under pressure, deferring work, duplication – the copy-and-paste problem


• Quality Debt: not enough attention to quality, not enough tests, not taking responsibility for fixing bugs


• Configuration Management Debt: mistakes and inefficient practices in version management, build, deployment and release practices


• Design Debt: under-designing and over-designing – expect to get the design right the third time


• Platform Experience Debt: not sharing information, over-specialization

I like the idea of broadening the discussion of debt from code and design considerations to include testing and configuration management, and team considerations as well.

Sterling is an Agile Coach and Trainer, so he predictably looks for answers to managing software debt problems in the context of Agile development. The book is 220 pages long, but only the first 30 pages take on software debt directly; the rest of the book is about how to follow good practices in Agile development, and how this will help you minimize the amount of debt that you take on. It’s more of a primer on Agile development management and design than an analysis of software debt.

If you are building new software and want to take a responsible approach to managing debt for the future, and if you want to know more about how Agile practices and ideas can help you do this, then you may want to read this book. If you want to just get an overview of the book, most of the ideas are introduced here.

I wanted more than a review of good Agile practices. I want more on how to responsibly deal with debt. I want to know more about how to measure and asses the direct and indirect costs of debt, and how to recognize when a team has taken on too much – how to qualify and quantify it. And I want to understand more about how and when to make the case for paying off debt.

Technical debt is about making tradeoffs in cost and risk. I expected more numbers, more analysis on the cost of decisions, the time value of money. And I also expected more war stories, to see more evidence from real life. Everyone is dealing with software debt. There must be more success stories and stories of failures, concrete examples to learn from, more data to chew on and spit out. Instead, there is a couple of short case studies, and a handful of dubious graphs that attempt to quantify the cost of debt, and the return on paying debt back. But on closer review these aren’t graphs, they are pictures: there’s no data to back them up.

In the beginning, Sterling introduces a case study of a company that appears to have run into a wall. They are maintaining a 15-year-old C application, and while they are “doing everything right”, they are facing serious quality problems and can’t meet their delivery commitments. Sterling implies that they have taken on too much technical debt, and now they have to pay the price. But the case isn’t convincing. They are certainly doing something wrong. Yes their code base sounds like it is a mess, the kind of mess you might expect in a big 15-year-old C application, so it’s clear that they have some technical debt problems. But it’s also clear that their planning and technical practices, and their risk management controls aren’t good enough either – or they wouldn’t be in the mess that they are, and they wouldn’t be surprised by it.

There’s not enough to this story to make the case. And the story is left unfinished. The team had a problem and asked for some training. What happened next? Did they find out how much technical debt was holding them back, where the costs were coming from? Did they find a way to pay down their debt and get back on track? What action did they take, what worked, what didn’t? That’s what this book should have been about.

There’s another story later on about a team that had too many open issues (thousands of them), and were now having to spend too much on fixing bugs. Again, not a lot of context, but the lesson was clearer this time –leaving thousands of bugs and problems open for years is going to bite you in the ass eventually. That’s obvious. But there’s not enough here to draw good conclusions from or learn much from.

My other concern is that Sterling’s analysis focuses on a narrow part of the debt problem: preventing yourself from taking on too much unintentional debt by following good practices and disciplined development. doesn’t explore how to make intentional cost/risk trade offs, how to use debt tactically or strategically. It doesn’t help you deal with debt that you have created, or inherited. It doesn’t provide guidance on how to pay debt back, where to start, where you can get the most return, or the fastest return.

There’s more to managing software debt than good clean living, even good clean Agile living. Than doing it right the third time. If you are dealing with technical debt problems, if you have inherited a lot of debt and need to find a way to pay it off, if you want to understand what to do with debt and where to start paying it off, you will get a lot more out of Michael Feathers’ book Working Effectively with Legacy Code.

SWEBOK V3 and Software Security

Steve Tockey at Construx Software gave me some good news recently: the new revision of the IEEE’s Software Engineering Body of Knowledge (SWEBOK) will include software security as a fundamental concern in software engineering.

The SWEBOK is intended to document a common understanding of software engineering, and to act as a map to everything that anybody who designs, builds and tests software should know and understand. This is part of the IEEE’s attempt to establish basic accreditation for software engineers: a Certified Software Development Professional (CSDP) designation, similar to certified project managers (PMP) and certified IT security professionals (CISSP) and so on.

The CSDP has not gained much traction in the software development community, although the accreditation initiative has the support of companies like Boeing and Lockheed Martin. Most developers that I know haven't heard of the SWEBOK or know about CSDP certification. The only real-life CSDPs I have met have been instructors from Construx, which also sponsored some of the SWEBOK work.

But the SWEBOK has become a reference for some university programs, and a handful of universities now offer entry-level CSDA certification training as part of their software engineering programs - similar to PMI's CAPM associate certification. So the SWEBOK has the potential to influence future software development.

The security updates in V3 of the SWEBOK look like they will wire security into requirements, design, construction, testing, maintenance, configuration management, software engineering management and processes, tools and methods, and software quality. Everywhere really. This is exactly right.

We have to be realistic. It will take a while for these and other changes to be made and reviewed and approved, and for the new SWEBOK to be published. (It looks like the latest revision is already some months behind schedule). And it will take years after that before the new version will be adopted. Adding software security in the SWEBOK isn’t going to change how people design and build software in the real world soon. But I am still glad to see it. It’s a small step, in the right direction.

What I like (and don't like) about DevOps

I’ve spent a lot of time in my career working on problems that cross the lines between development and operations. That’s why I am interested in the emerging DevOps community: a bunch of smart people who are trying to bring development and operations closer together, and applying ideas and practices from Agile development to improving system and application operations. I try to keep up as much as I can with what DevOps is about, what people are doing and thinking, and understanding what it means to me. I went to Velocity last year, I’ve been reading the DevOps blogs, following the DevOps discussion group, I read and enjoyed the Web Operations book.

So, from a software development manager and CTO perspective, this is what I like (and don’t like) about DevOps so far.

What I like

The DevOps community is real: it's about people solving real problems, hard problems that a lot of us face in managing large systems. It's about people working together to come up with new ideas on how to do a better job. It's about taking ownership of problems and the solutions to these problems.

It is not an idea that has been crafted outside and that is being sold to people because it is good for them – not like Rugged Software for example. DevOps has authentic buy-in, it is being built from the ground up, and I can see things actually happening.

And there is no right or wrong, there’s no Manifesto (please please keep it that way), there’s no fundamentalism or religious wars – so far at least. DevOps is still in that early place and time where things are fluid and searching. People can disagree, and as long as they are smart and pragmatic and fair, they are listened to. Nobody is worried about contradicting one of the Creators (unlike in the Scrum community, or the the software Kanban community). Most of the people involved in DevOps today seem to be practical and open minded, and focused on finding answers. I hope it stays this way, forever and ever and ever.

DevOps has strong support from vendors. There’s configuration management and deployment automation technology like Puppet and Chef of course, and companies selling Continuous Delivery services and technology, the stuff needed for Infrastructure as Code, for effectively integrating application and infrastructure changes. And there is an important need in DevOps for highly scalable but lightweight data management technology, network management and monitoring platforms, tools to collect and analyze system and application metrics and logs. And then there's the Cloud.

Yes, there has to be a role for vendors: somebody has to act as a sponsor, to help sustain momentum, and people need technology to solve DevOps problems, to make deployment and management and operations of large-scale systems possible. But DevOps is not vendor-led, or consultant-led. Big Blue and HP haven’t figured out what DevOps means to them, or how to turn it to their advantage. Most of the technology is Open Source, making it easier for the community to get involved in solving their own problems.

The people in DevOps who are worth listening to are techies or hands-on managers who have real day-to-day responsibilities and solve real problems and who are sharing what has worked for them, and who are actively trying to find ways to do their jobs better and are interested in answers. Most of the thinking and writing in DevOps is being done by the doers, not by people trying to sell something, trying to convince you that you have a problem and that you need their help to solve it.

DevOps is about solving some interesting and challenging problems, in some cases extreme problems. I don’t have to manage the technology for a platform with thousands or tens of thousands of servers or millions of online customers, but I can learn from people who do. It’s fun and it’s worth paying attention to.

What I Don’t like

There’s a lot of enthusiasm in the DevOps community, people excited because they are onto something that is working, creating new tools, finding new solutions to problems. There are times when those of us who have been around for a few years recognize old ideas, even when they’ve been given cool new names. Listening to someone excitedly describing “dark launching” and “feature flippers” and pretending that this is a new idea is tiresome… But you can get over it, you can still enjoy the enthusiasm and wait to see if there are any new lessons to be learned.

Not surprisingly, there is much talk by DevOpsers about Values and Culture and Collaboration and Trust and other Capitalized Important People Stuff. This is all necessary when you are trying to get people to work together in new ways, but it has the potential to slide into pop psychology and New Age philosophizing like too much of what passes for thinking in the Agile development community today by consultants and "coaches". Hopefully, DevOpsers will stick to solving problems and getting things done in an open way, and not wandering down this path.

DevOps hasn’t come to terms with ITIL, and it needs to. ITIL is used as a straw man by some of the DevOps thinkers, in the way that Waterfall development is used by some Agile development enthusiasts: as an outmoded way of thinking and working, a collection of antipatterns. This naive approach is not fair or practical, it alienates a large community of people working in enterprise organizations and government IT organizations who have adopted more structured frameworks to get their work under control. There’s a lot to like in DevOps, but there’s a lot that DevOps can learn and has to learn from ITIL as well.

More fundamentally concerning to me is that DevOps has failed to take responsibility for security. They don’t seem to understand or care about what’s involved in deploying and operating secure systems, or if they do, they don’t explain how it fits. This will limit DevOps adoption, because most operations and development teams who want to take advantage of the ideas and practices and technologies coming out of DevOps, to become more efficient and more agile, also need to do this in a secure way. And this is reflected in the ongoing security problems that some of the organizations that are pioneering DevOps ideas continue to face. Would you trust your email to Facebook, would you really?

There’s a lot of interesting ideas coming out of large-scale online social media companies like Facebook and Twitter, and Flickr and Etsy, and some of the online multi-player Internet game platforms. But there’s a gap in trying to apply their lessons to the problems and requirements that the rest of us deal with: smaller-scale, but just as hard, just as real. Secure design and deployment and operations, all kinds of compliance and auditing, real data privacy, real transaction reliability, B2B integration, protecting compatibility with customers and legacy systems. Ideas taken to extremes like Continuous Deployment are exciting and challenging, they force you to think about how you can get better. But they aren’t ready for the enterprise yet.

DevOps doesn’t represent or meet the needs and priorities of most of us working on enterprise systems. But that doesn’t mean that they shouldn’t keep going. And that doesn’t mean that the rest of us shouldn’t keep following them – and trying to keep up.

Software Security and the Long Tail challenge

Secure and Resilient Software Development, a new book by Mark Merkow and Laksh Raghavan, explains the problems that developers face in designing and building secure software, and where to find resources to help deal with these problems.

The book covers basic principles of secure software, and how to include security in analysis and design and coding and testing. It does a good job of mapping security problems and requirements to available solutions: where you can find training and certification, commercial software and services, and open source tools and checklists, especially tools and checklists from OWASP. It walks through the OWASP Top 10 risk list in detail, and introduces how to use OWASP’s open source Enterprise Security API (ESAPI) library to build secure apps.

The authors lay out a high-level secure SDLC explaining where checks and controls are needed. Like most secure SDLCs, it’s serial, waterfall-based, with gates between analysis and design and coding and so on. There’s no guidance on how to adapt these ideas to Agile methods like Scrum, or to maintenance, but they point to another (inactive) OWASP project CLASP which can help you integrate security controls into the way that you build software.

The book also covers publicly-available software security maturity models: BSIMM and OWASP’s SAMM, used to assess an organization’s software security program and identify gaps and weaknesses. I’ve used SAMM before with my own team as a management scorecard: it helped me structure my thinking and planning on how to improve our software security program. BSIMM is a much bigger beast, intended for enterprise-scale software security programs.

The Enterprise

And there’s the rub. Maturity assessment frameworks, and review-heavy waterfall-based secure SDLCs, are targeted to big companies - like the expensive commercial security testing and checking tools that are also covered in this book. Too much of software security is about the enterprise. There is a lot of money being spent (and a lot of money to be made) here, and big companies have more people and time to invest in software security work as part of their broader risk-management and compliance programs. Software security has made the big time, which is why IBM and HP have bought their way in, and why the remaining independent technology vendors are tooling up for the enterprise (and raising their prices).

Enterprise customers are an important market and this market demands and deserves to be well-served.

The Long Tail

But it’s important to remember the Long Tail.

Sure, a lot of software is built and used by enterprises. But most application software developers will spend their careers working in small software companies or in internal IT groups in small and medium-sized businesses. Small teams like these are looking for simpler and easier ways to build software cheaper and faster and better. Because they can, and because they have to. Agile methods to break projects down into 1-week or 2-week chunks and deliver features to the customer faster. Open source tools and frameworks that work. Practices like unit testing and refactoring that help developers get control over development. Continuous integration, and continuous delivery, even continuous deployment. Ideas applied from Lean Manufacturing to strip out waste. Whatever is practical, whatever helps them to get the job done.

And these ideas and approaches are making it into the enterprise. Teams in enterprise businesses, and in the enterprise software vendors and offshore companies that service big businesses, are adopting lighter-weight project management and development practices because they help them cut costs and deliver faster, and because they work.

There aren’t many small teams that are going to bother with a software security maturity assessment framework, or understand it. BSIMM has 109 different measurement areas: that's around 100 more than most small teams would be able to deal with. And asking small teams who follow Agile practices to measure themselves against a comprehensive process model and work their way up a maturity ladder doesn’t compute. Agile methods were created as a reaction against this kind of thinking and the costs and inefficiencies involved.

It’s important and necessary to find solutions and ideas that will help smaller, faster-moving software organizations – and bigger organizations that are also trying to find ways to move faster. We need practices and ideas that fit with the way these teams work. And that’s not going to be easy.

At OWASP’s AppSec USA 2010 conference, Adrian Lane critically reviewed the security challenges that Agile teams face and the security problems that they create, in his presentation Agile + Security = Fail. As is clear from the title, there are more problems than solutions. Agile teams move too fast for heavyweight security models, they don’t leave time for exhaustive systemic tests and reviews, they don’t have time to do everything right the first time – and they are trying to move even faster.

OWASP tools like ESAPI and the Top 10, and OWASP's secure development and testing guides and secure coding checklists are important and useful resources for developers trying to build secure software. It's good to have a book that points out where to get help like this. And Microsoft's SDL-Agile shows how a company with Microsoft-level capabilities and resources can build secure software using Agile methods.

But we need more. We need more tools and ideas that work for smaller development and maintenance teams, that we can fit in with lightweight methods and that work under pressure. We need patterns and practices that are simple to understand, easy to follow and practical. We need improved languages and frameworks that make it harder to write bad code; and simple and affordable tools that make it easier to find and fix vulnerabilities, because people will still write bad code. And we need the leaders of the development community and the Open Source communities to take security seriously and include security in the way that we build and delivery systems, and in the Open Source code that we share and rely on. We need more solutions that small teams can understand, can afford, and will use.

Because the challenges and problems of writing secure software are both bigger than, and smaller than, the enterprise.

Everything I can find about Software Maintenance

Given that most of us will spend most of our careers maintaining and supporting software and not building something new from scratch, there is a surprising and disappointing lack of useful information on software maintenance, on the problems that we all face in software maintenance. I’ve tried to put together here an index to useful books, articles and blog posts and white papers on software maintenance.

Last updated: November 13, 2013

Blog Posts and Articles on Software Maintenance

Jeff Atwood’s The Noble Art of Maintenance Programming is a thoughtful post on how maintenance programmers are unjustifiably unappreciated: how maintenance is much harder and more important than most people, even the people doing the work, understand.

Steve Kilner has an interesting blog called Notes on Software Maintenance. His technical focus is in maintaining RPG code on IBM iSeries computers, but he looks into a lot of basic problems and open questions in software maintenance and software maintenance management, including how to handle estimates and how to effectively manage risk and complexity in maintenance. You can get a good introduction to these problems in an article that he wrote for IBM Systems Journal, Why Good Software Goes Bad and How to Stop It and in this slideshow on Unsolved Problems in Software Maintenance.

In Sustained engineering Idiocy, Microsoft’s Eric Brechner explores how to organize your maintenance team, and the pros and cons of juggling maintenance and support responsibilities with new development work. Although some of his writing is internally Microsoft-focused, his blog is worth following for general good ideas on software development management and maintenance.

Robert Glass wrote an insightful essay on the challenges in software maintenance, "Software Maintenance is a Solution, not a Problem", in his book Software Conflict 2.0: The Art and Science of Software Engineering. You can read a copy of the essay here.

Greg Jorgensen, the Typical Programmer, shares a positive perspective on The Joys of Maintenance Programming, describing how maintenance programming can be a satisfying and successful career choice.

Steve Shaw at Intelliware wrote a short paper on his team's experience following Agile methods in software maintenance; what practices worked (short iterations, testing, prioritization, common code ownership) and what didn't (pairing, stories and story cards, stand-ups, metaphor). There is an interesting case study on Using Extreme Programming in a Maintenance Environment and another on problems with introducing Agile methods into an organization with a complex code base. A study on Introducing Agile into a large-scale maintenance organization can be downloaded from IEEE here for $19 (but $30 elsewhere at IEEE - don't ask me why).

Naresh Jain put together this slide show on why Maintenance Sucks and how to apply Agile methods to software maintenance when he was at ThoughtWorks.

The Journal of Software Maintenance and Evolution: Research and Practice regularly publishes (highly) academic papers on software maintenance and related software engineering problems.

The Good Books

There are some excellent books on writing and maintaining code, starting with Code Complete 2 by Steve McConnell: the definitive book on how to write good code, maintain it and debug it.

Legacy code means different things to different people. For some programmers, legacy code is code is any code that they didn't write. For others, it is code that stills runs on abandoned or out-of-date platforms and technologies. But everyone agrees that it is code that is difficult to understand, difficult to support and difficult to change. And in maintenance you are going to spend a lot of time dealing with it. Working Effectively with Legacy Code by Michael Feathers lays out how to get a legacy code base under control, and the steps on how to make changes safely and properly. You can follow Michael Feathers' latest work on mining source code and "brutal refactoring" on his blog.

The Passionate Programmer by Chad Fowler offers some good advice to programmers (and managers) looking for meaning in software development and maintenance.

If you are confronted with the problem of reengineering a legacy system, check out Object Oriented Reengineering Patterns, which is available free online. The pattern approach is awkward at times, but this book offers some clear guidance on when and how to reengineer a system.

More on Legacy Code

There are also a couple of good online presentations from the smart people at ThoughtWorks on how they have dealt with legacy code problems. A presentation by Matt Simons and Jonny LeRoy covers how to get a handle on existing code, mapping the code and identifying risk areas, creating dashboards to visualize the state of the code base using open source tools like Panopticode and Codecity (which offers a cool 3D cityscape view of your code, making metrics fun for everyone), or Sonar (actually, they didn’t mention Sonar, but they should have). Then they outline different strategies for incrementally re-writing large applications, emphasizing Martin Fowler’s Strangler pattern, and some examples of how they have followed these ideas with different customers.

Another good presentation from Josh Graham at ThoughtWorks on Brownfield Software walks through an interesting case study where his team came in to help with a major overhaul of a large, 10-year-old core business system.

A Team, A System, Some Legacy... And You is an intelligent and practical presentation by Eoin Woods on how an architect can make a legacy system better. How to to an architectural assessment of a legacy system, and how to improve the reliability, scalability and maintainability of the system through just enough software architecture, safe step evolution, improving testing and deployment, engaging in production, working with stakeholders and understanding the capability of the team.

Refactoring is a foundational discipline for maintenance – Martin Fowler put a name and a disciplined structure to the work that we all do, or should do, when changing code. Seriously, is there anybody who hasn’t read this book by now, who doesn’t follow some/most of these ideas in maintaining code?

Debug It! by Paul Butcher is a narrowly-focused but useful book, about debugging, problem-solving techniques, handling bugs, and writing code that is easy to debug. Especially useful for programmers new to maintenance.

Managing Maintenance

There are some great books on managing software development projects, like Steve McConnell's Rapid Development, and Scott Berkun's Making Things Happen. Many of the ideas in these books will help you in managing software maintenance work.

There are only a couple of books that take on managing software maintenance:

The best of these by far is Software Maintenance Success Recipes by Donald Reifer, a thoughtful analysis of current software maintenance practices (2012) based on the study of several large software maintenance organizations. It hilights the importance of testing in maintenance (up to 60-70% of time is spent testing and retesting changes to check for regressions, testing to reproduce bugs then testing to verify that bugs were fixed correctly...), and examines models and metrics for effectively managing maintenance.

Alain April and some other researchers have put together a Software Maintenance Maturity Model, a la the CMMI. It's obviously process-heavy and bureaucratic, but it does a thorough job of mapping out all of the work involved in software maintenance, and it's interesting in an abstract way. If you find this model useful, you can go through it in detail in the book Software Maintenance Management, although it is written by and for academics, and focuses more in defining the model than on how to deal with practical problems.

If you're managing a large enterprise maintenance team and working through a lot of structure and standards, then Thomas Pigoski's Practical Software Maintenance offers advice based on real experience - the author ran a large maintenance organization for the US Navy, and is now the CEO of TechSoft, an outsourcing firm that handles maintenance for other companies. Unfortunately this book is out of date (published 1996) and predates modern Agile and Lean methods and practices and new technologies. Pigoski also wrote the chapter on software maintenance in the IEEE SWEBOK, an academic definition of software engineering.

Another academic book which I can't recommend for practitioners is Software Maintenance: Concepts and Practice by Penny Grubb and Armstrong Takang. If you're already working in maintenance, you won't find anything here that you don't already know or that will change the way that you think or work. It's more of an introductory text for undergrad Comp Sci courses.

Studies, Statistics and Lies

Dr. Jussi Koskinen, a Comp Sci professor in Finland, maintains a useful summary of software maintenance cost data and sources.

If I want any kind of data on software development or software maintenance, the first person that I go to is Capers Jones. His book on Estimating Software Costs was last updated in 2007, and includes extensive data on software cost information and statistics and trends on software maintenance and support. Or you can read Applied Software Measurement or Software Engineering Best Practices - all of these books are built on the same comprehensive research. You can spend days going through this stuff - I have, and I keep finding interesting things.

Some of his research specific to software maintenance and legacy software is captured in an excellent white paper called Geriatric Issues of Aging Software. A couple of different versions of this paper can be found on the web, the best of them is in CrossTalk: The Journal of Defense Software Engineering dated December 2007.

One of the foundational studies on software maintenance is more than 30 years old now: Lientz and Swanson's study of software maintenance at 487 IT organizations. This study is widely referenced, and still the most comprehensive analysis of what work makes up maintenance (or at least what did in 1978). You can read a follow-up study on software maintenance management problems (1995) here. A newer (1998) but smaller study by Janice Singer looks at bugs and bug tracking, and how unimportant documentation actually is in maintenance.

Building, releasing and deploying code

And there are some good books on building, deploying and delivering code, on Continuous Integration and source code management, all of which are important in maintaining and releasing code. The best of these is the new book on Continuous Delivery by Jez Humble and David Farley. It’s full of practical advice on how to improve your code build and deployment practices; and on data migration and test automation strategies; and more far-out ideas on how far-out you can push automation in build and deployment.

Lean and Kanban and Maintenance

If you are interested in how Lean Manufacturing and Kanban ideas can be applied to software maintenance, and quite a few people are, a place to start is the XIT case study which explains how Kanban was first used to speed up and simplify management of maintenance work by a small cross-application team at Microsoft. I think there’s too much hype and religious debate around what is a conceptually simple idea (visualizing and limiting work in progress), but Kanban offers a natural framework for getting control over maintenance and support work, helping to deal with interruptions and constantly changing priorities.

Operations and DevOps

Developers need to understand how their code runs in production, need to care about and know about operations and performance and how to handle failures. This becomes even more important, more necessary, in maintenance: it’s a reality that maintenance teams face all day each day, rather than just another set of requirements for the future. Some smart people in the DevOps community are trying to come up with a more collaborative and agile way for operations and development teams to work together. Most of their work focuses on large-scale Web Operations problems and operations toolchains, but there is some good thinking here on how to improve deployment, on metrics and monitoring, and on dealing with architectural problems like scaling, and failure handling and recovery.

While helpful, most of the thinking in DevOps is being done from the operations point of view (maybe it should be called OpsDev instead…). The best book on operations problems by and for developers and software architects is still Release It! by Michael Nygard. He makes clear how important it is to, and how to, design scalable and resilient software architectures, and how and why to build an operations view into your system.

If you are working with Ops (especially in an enterprise shop) you should understand the basics of ITIL, and the cost and risks of change in production, and how to to get a production system under control. The best place to start on this is The Visible Ops Handbook, a tiny primer on intelligent IT change management.

Dealing with Data

One of the biggest problems in software maintenance is handling changes to data and data models. Scott Ambler's book Refactoring Databases walks through an incremental approach to making structural changes to database models, and how to test and deploy these changes.

Secure Software Maintenance

Dealing with security risks for legacy systems is another wormcan, and you're not going to find a lot of information to help you understand where to start, other than this set of white papers on legacy systems and COTS from Cigital, available on the US Department of Homeland Security’s Build Security In portal.

Forums and Discussion Groups

I can't find any discussion groups on software maintenance or sustaining engineering.

Stack Overflow has some interesting threads on maintenance and refactoring and other maintenance problems. It’s one of the mainstay resources for developers looking for advice on maintenance issues or anything else for that matter.

And finally...

You can learn a lot from antipatterns. How to Write Unmaintainable Code by Roedy Green is thorough, thought-provoking and fun.

I would like to hear from people about any other useful books, blogs, articles or forums that focus on software maintenance and support problems - what resources you trust, what you have found helpful.