Last week I attended the OWASP AppSec USA conference in Minneapolis. It was my first time at an OWASP event, and it was an impressive show. More than 600 attendees, some high-quality speakers, lots of vendor representation.
There were several different tracks over the two days of the conference: attacks and defenses, security issues in Mobile computing and Cloud computing, thought leadership, OWASP tools, security patterns and secure SDLCs. And there were lots of opportunities to talk to vendors and smart people. Here were the hilights for me:
Jim Manico showed that there’s hope for developers to write secure code, at least when it comes to protecting web apps from XSS attacks, if they use common good sense and new frameworks and libraries like Google’s Caja, the technology behind OWASP’s Java HTML Sanitizer.
There is even some hope for writing secure Mobile apps, as long as developers are careful and informed. This is a more reassuring story than what I was hearing from security experts earlier in the year, when everyone was focused on how pathetically insecure the platforms and tools were for writing Mobile apps.
It's also possible to be secure in the Cloud as long as you take the time to understand the Cloud services that you are using and how they are implemented and what assumptions they make, and if you think through your design. Which a lot of people probably aren’t bothering to do.
There's too much code already out there
We’re still trying to find ways to build software that is secure. But a bigger problem for many organizations is that there is already too much code out there that is insecure. Jeff Williams at Aspect Security guesstimates that there are more than 1 trillion lines of code in the world, and around 15 million developers around the world writing more of it. A lot of this legacy code contains serious security problems – but it is too expensive to find these and fix these problems. We need better tools to do this, tools that scale up to enterprises and down to small companies.
Many organizations, especially big enterprises with thousands of applications, are relying heavily on scanning tools to find security vulnerabilities. But it takes too long and costs too much to work through all of the results and to find real risks from the long lists of findings and false positives – to figure out what problems need to be fixed and how to fix them.
HP thinks that they have an answer by correlating dynamic scanning and static code analysis results. The idea is to scan the source code, then run dynamic scans against an application under a run-time monitor which captures problems found by the dynamic scanner and records where the problems occurred in the code. By matching up these results, you can see which static analysis findings are exploitable, and it makes it easier to find and fix problems found in run-time testing. And combining these techniques also makes it possible to find new types of vulnerabilities that can’t be found using just one or other of the approaches. I heard about this idea earlier in the year when Brian Chess presented it at SANS Appsec, but it is interesting to see how far it has come since then. Hybrid testing is limited to HP technology for now (SPI Dynamics and Fortify) and it’s limited by what problems the scanners can find, but it looks like a good step forward in the test tool space.
Good Pen Testing and Bad Pen Testing
It’s clear talking to vendors that there are good pen tests and bad pen tests. In most cases, good pen tests are what they offer, and bad pen tests are what you would get from a competitor. Comparing one vendor to another isn't easy, since an application pen test means different things to different people, and the vendors all had different stories and offerings. Some stressed the quality of their tools, others stressed the experience and ingenuity of their testers. Some offered automated scans with some interpretation, which could work fine for plain-vanilla web apps. Some do combined source code review and scanning. Some don’t want help understanding the code or scoping an engagement – they have a fixed-price menu running from running scans to manual pen tests using scanners and fuzzers and attack proxies. Others expect to spend a lot of time scoping and understanding what to test, to make sure that they focus on what's important.
SSL is broken
Moxie Marlinspike’s keynote on the problems with SSL and the CA model was entertaining but scary. He showed that the circle of trust model that underlies SSL – I trust the host because my browser trusts the CA and the CA trusts the host – is fundamentally broken now that CAs like Comodo and DigiNotar (and maybe others) have been compromised.
His solution is Convergence, an alternative way to establish trust between parties, where the client is responsible for choosing trust providers (notaries) rather than being locked into a single trust provider selected by the host.
Metrics and numbers for fun
I like numbers and statistics, here are a few that I found especially interesting. Some of them were thrown out quickly, I hope that I got them right.
At a panel on secure development, one expert estimated that the tax for introducing a secure SDLC, including training people, getting tools and getting people to use them, was about 15% of the total development costs. Although he qualified this number by saying that it was from a pilot, another panelist agreed that it was reasonable, so it seems a good rule of thumb to use for now.
John Steven at Cigital says they have found that an inexperienced team can expect to have 38-40 security vulnerabilities per KLOC. A mature team, following good security hygiene, can get this down to 7-10 vulnerabilities per KLOC.
Chris Wysopal at Veracode presented some interesting numbers in his presentation on Application Security Debt – you can read more on his ideas on applying the technical debt metaphor to application security and building economic models for the costs of this debt on his blog. He referenced findings from the 2010 Verizon Breach study and tried to correlate these findings with analysis work that Veracode has done using data from testing customer applications. This showed that the most common application-related breach was through Backdoors or hidden Command Channels which I was surprised at, and so was he, since Veracode’s own analysis shows that application Backdoors are not that common. I assumed from this analysis that application Backdoors were so dangerous because they were so easily found and exploited by attackers. However, when I read the Verizon report again, what Verizon calls a Backdoor is a command channel opened up by malware, which is something quite different to a Backdoor left in an application, so the comparison doesn’t stand up. This leaves SQL Injection to be the number one application vulnerability, something that shouldn't surprise anyone.
The “A Word”
Most of the speakers and panelists at the conference were from enterprise companies or from vendors trying to sell to the enterprise. Not surprisingly, as a result there was little said about the needs and constraints of small companies, and at the sessions I attended, the speakers had not come to terms with the reality that most companies are adopting or have already adopted more incremental agile development practices. At one panel, Agile development was referred to as the “A Word”.
We urgently need tools and practices that small, fast-moving teams can use, because these teams are building and maintaining a lot of the world's software. It's time for OWASP and ISC2 and the rest of the application security community to accept this and understand what it means and offer help where it's really needed.
Dumbest thing said at the conference
This award goes to one vendor on a panel on security testing who said that it is irresponsible for developers to work with new technologies where they don't understand the security risks, that developers who work on the bleeding edge are putting their customers and businesses at unnecessary risk. Besides being blindingly obvious, it shows a complete lack of connection to the realities of software development and the businesses that rely on software (which is pretty much every business today). Dinis Cruz pointed out the absurdity of this statement, by admitting that we don’t fully understand the security risks of the common platforms that we use today like Java and .NET. This means that nobody should be writing software at all - or maybe it would be ok if we all went back to writing batch programs in COBOL and FORTRAN.
Impressed and Humbled
Overall I learned some good stuff, and I enjoyed the conference and the opportunity to meet some of the people behind OWASP. I was impressed and humbled by the commitment and accomplishments of the many OWASP contributors. Seeing how much they have done, talking to them, understanding how much they care, gives me some hope that we can all find a way to build better software together.
Monday, September 26, 2011
Wednesday, September 7, 2011
Standups – take ‘em or leave ‘em
We left ‘em.
Standup meetings are a core practice in Agile methods like Scrum and XP. Each day the team meets briefly to answer 3 questions: What did I get done yesterday? What am I going to do today? What is getting in my way?
Standups offer a quick check on what’s happening, what’s changed, who ‘s working on what, who needs help. The meeting is supposed to be short and sweet, no more than 15 minutes a day. Martin Fowler lists some good reasons to hold standups:
- Share commitment
- Communicate status
- Identify obstacles to be solved
- Set direction and focus
- Help to build a team
However, not everyone finds that standups are necessary and some people have started to question the value of standups over time and are looking for substitutes. A number of people have suffered through poorly-run standup meetings. And some people just plain don’t like them.
Our team has been successful following Scrum and XP practices as we transitioned from delivering phased releases (a release every 2-3 months) to 2-3 small releases a month. We looked closely at how other teams worked, spent time learning about incremental and iterative development methods, and took what made sense to us and tried it and adapted it. But one of the Agile practices that we did not take up was standups.
When we introduced the idea of standups to the team a few years ago, we were surprised by the response. The team was roughly split. Some people, especially people who were new to the team, or people who had worked on successful Scrum teams or XP teams before, liked the idea. But other people who had been on the team from the beginning were opposed, some of them strongly opposed. And they had some good reasons:
We were already running an operational business and building and delivering new software at the same time. Team members were helping to support the system, helping with day-to-day operations, and working closely with the business side. Some people were working late and weekends to get changes in or to do after-hours testing, others were working early to help with startup issues or to coordinate with partners, some people were working in different timezones, some people were working at home, others were at the beck and call of integration partners and important clients. It would not be possible to schedule a daily meeting, even a short one, which would work well for everyone on a sustained basis. This is one example (and there many others), where Agile development ideas which work well in the controlled, artificial reality of a development project need to be rethought and adapted to fit day-to-day operational demands and constraints.
Some of the team had their hands full with important changes or problems that had to be taken care of urgently. They met regularly with other team members to work through design issues and reviewed each other’s code. They got help from other team members or managers when they needed it – there was no need to wait for a daily meeting to bring up blockers or get whatever information they needed.
And there were a couple of introverts who hated all meetings and didn’t understand why a meeting that they had to go to everyday and standup at would be any better than any other meeting. They saw it as a half hour wasted out of every day – because they knew that a 15-minute meeting, when you include the time taken to save what you were working on, go to the meeting, standup, get out of the meeting and get back to working productively, would mean at least 30 minutes if not more time every day from what they were doing.
Rather than try to fight with the team to implement a practice that we weren’t sure was really needed, we decided to find other ways to get the same results.
Alternatives to standups
There are other ways to share status within (and outside of) the team. Because we have people working together in different countries and different groups, we rely heavily on our bug tracking system to manage the backlog of work, to schedule and plan work, and to radiate status information. The bug tracking system is used by and shared with everyone: developers, testers, support, business operations, systems engineering, IT, compliance and management. Everything to do with the software and systems operations is tracked here, including new features and bug fixes and security vulnerabilities and problems found in testing, and operational and infrastructural changes, compliance reviews and new clients being onboarded. Using the system’s dashboards and notification feeds everyone can easily tell what is happening and what will be happening and when. The team doesn’t need standups to know what is happening, who is working on what, what the schedule is or what needs to be worked on next.
Managers who need to know what’s going on and who want to make sure that everyone is on track can get all of this through MBWA. And regular one-on-one meetings help reinforce priorities and make sure that everyone gets face time and a chance to ask questions without wasting the rest of the team’s time.
Standups help at the start
I can see that standups are more useful in the early stages of a project, when the team is starting to come together and everyone is working through the design and learning the domain and the technology and feeling out each other, finding out each other’s strengths and quirks.
Without standups, it is harder for new people joining the team to understand what is going on and what’s important, and figure out where they fit in. Pairing up new team members with someone more experienced helps, but if you have a couple or more people joining a team, it makes sense to hold standups for a while at least to help them get up to speed.
Declining value over time
But when people have been working together for a while, standups offer declining value. Once the team has gelled and people know each other and can depend on each other, they don’t need to meet in a room every morning to talk about what they are working on. Like some other practices that are important and useful in the beginning, the team grows up or grows out of them.
And as you move more into maintenance and support work, when people on the team are working on smaller individual changes and fixes and the work tends to be more interrupt-driven, standups are a waste. What one person is working on doesn’t have much if anything to do with what somebody else is doing. They don’t need to, or want to, listen to each other talking about what they did and what they’re going to do, because if it was important they would already know about it anyways.
Stuffed Pigs, Nerf Balls and other Silly Games
One of the things I don’t like about standups is that they are fundamentally paternalistic – you’re treating the team like children, forcing them to get together and stand up in a room every day (they have to stand up because they can’t be trusted to sit down) and making them speak in turn for only a few minutes (but only of course if they are a pig, not a chicken). And on some teams, people go so far as to hand around a stuffed pig or a Nerf ball or follow some other awkward ritual to make sure that people don’t speak out of turn. If this sounds silly and childish, it is. You have a room full of children: team members who can’t ask questions, they can’t solve problems, they have to stand there and follow silly rituals. Day after day, week after week, month after month, year after year…
Rather than treating adults like little children, why not let people meet when they need to or want to – to solve problems, to review requirements or designs, to respond to incidents (Root Cause Analysis), to plan, and to make sure that people know when requirements or priorities are changing in a significant way.
Would we have had different results if we had implemented standups? Probably. Would the results have been better for the team and for the company? I’m not sure. We found other good ways for the team, and the rest of the company, to track what was going on and what needed to be done next. We’ve had some misunderstandings, and some people did get off track – standups might have helped prevent this. And standups would have helped new people joining the team, to come up to speed. But the rest of the team gelled quickly during our crazy startup days, and built up a high level of trust and openness and shared commitment without standups. Can you build high-performing, collaborative teams without standups? Of course you can.
Standups have a place, especially early on with people who don’t know each other, where everyone is learning and uncertain about what to do and about each other. But don’t do something, or keep doing something if you don’t think that you need to, just because somebody read it in a book or learned about it in a 3 day SCM class. It’s your team, and you can find your own way to succeed.
Standup meetings are a core practice in Agile methods like Scrum and XP. Each day the team meets briefly to answer 3 questions: What did I get done yesterday? What am I going to do today? What is getting in my way?
Standups offer a quick check on what’s happening, what’s changed, who ‘s working on what, who needs help. The meeting is supposed to be short and sweet, no more than 15 minutes a day. Martin Fowler lists some good reasons to hold standups:
- Share commitment
- Communicate status
- Identify obstacles to be solved
- Set direction and focus
- Help to build a team
However, not everyone finds that standups are necessary and some people have started to question the value of standups over time and are looking for substitutes. A number of people have suffered through poorly-run standup meetings. And some people just plain don’t like them.
Our team has been successful following Scrum and XP practices as we transitioned from delivering phased releases (a release every 2-3 months) to 2-3 small releases a month. We looked closely at how other teams worked, spent time learning about incremental and iterative development methods, and took what made sense to us and tried it and adapted it. But one of the Agile practices that we did not take up was standups.
When we introduced the idea of standups to the team a few years ago, we were surprised by the response. The team was roughly split. Some people, especially people who were new to the team, or people who had worked on successful Scrum teams or XP teams before, liked the idea. But other people who had been on the team from the beginning were opposed, some of them strongly opposed. And they had some good reasons:
We were already running an operational business and building and delivering new software at the same time. Team members were helping to support the system, helping with day-to-day operations, and working closely with the business side. Some people were working late and weekends to get changes in or to do after-hours testing, others were working early to help with startup issues or to coordinate with partners, some people were working in different timezones, some people were working at home, others were at the beck and call of integration partners and important clients. It would not be possible to schedule a daily meeting, even a short one, which would work well for everyone on a sustained basis. This is one example (and there many others), where Agile development ideas which work well in the controlled, artificial reality of a development project need to be rethought and adapted to fit day-to-day operational demands and constraints.
Some of the team had their hands full with important changes or problems that had to be taken care of urgently. They met regularly with other team members to work through design issues and reviewed each other’s code. They got help from other team members or managers when they needed it – there was no need to wait for a daily meeting to bring up blockers or get whatever information they needed.
And there were a couple of introverts who hated all meetings and didn’t understand why a meeting that they had to go to everyday and standup at would be any better than any other meeting. They saw it as a half hour wasted out of every day – because they knew that a 15-minute meeting, when you include the time taken to save what you were working on, go to the meeting, standup, get out of the meeting and get back to working productively, would mean at least 30 minutes if not more time every day from what they were doing.
Rather than try to fight with the team to implement a practice that we weren’t sure was really needed, we decided to find other ways to get the same results.
Alternatives to standups
There are other ways to share status within (and outside of) the team. Because we have people working together in different countries and different groups, we rely heavily on our bug tracking system to manage the backlog of work, to schedule and plan work, and to radiate status information. The bug tracking system is used by and shared with everyone: developers, testers, support, business operations, systems engineering, IT, compliance and management. Everything to do with the software and systems operations is tracked here, including new features and bug fixes and security vulnerabilities and problems found in testing, and operational and infrastructural changes, compliance reviews and new clients being onboarded. Using the system’s dashboards and notification feeds everyone can easily tell what is happening and what will be happening and when. The team doesn’t need standups to know what is happening, who is working on what, what the schedule is or what needs to be worked on next.
Managers who need to know what’s going on and who want to make sure that everyone is on track can get all of this through MBWA. And regular one-on-one meetings help reinforce priorities and make sure that everyone gets face time and a chance to ask questions without wasting the rest of the team’s time.
Standups help at the start
I can see that standups are more useful in the early stages of a project, when the team is starting to come together and everyone is working through the design and learning the domain and the technology and feeling out each other, finding out each other’s strengths and quirks.
Without standups, it is harder for new people joining the team to understand what is going on and what’s important, and figure out where they fit in. Pairing up new team members with someone more experienced helps, but if you have a couple or more people joining a team, it makes sense to hold standups for a while at least to help them get up to speed.
Declining value over time
But when people have been working together for a while, standups offer declining value. Once the team has gelled and people know each other and can depend on each other, they don’t need to meet in a room every morning to talk about what they are working on. Like some other practices that are important and useful in the beginning, the team grows up or grows out of them.
And as you move more into maintenance and support work, when people on the team are working on smaller individual changes and fixes and the work tends to be more interrupt-driven, standups are a waste. What one person is working on doesn’t have much if anything to do with what somebody else is doing. They don’t need to, or want to, listen to each other talking about what they did and what they’re going to do, because if it was important they would already know about it anyways.
Stuffed Pigs, Nerf Balls and other Silly Games
One of the things I don’t like about standups is that they are fundamentally paternalistic – you’re treating the team like children, forcing them to get together and stand up in a room every day (they have to stand up because they can’t be trusted to sit down) and making them speak in turn for only a few minutes (but only of course if they are a pig, not a chicken). And on some teams, people go so far as to hand around a stuffed pig or a Nerf ball or follow some other awkward ritual to make sure that people don’t speak out of turn. If this sounds silly and childish, it is. You have a room full of children: team members who can’t ask questions, they can’t solve problems, they have to stand there and follow silly rituals. Day after day, week after week, month after month, year after year…
Rather than treating adults like little children, why not let people meet when they need to or want to – to solve problems, to review requirements or designs, to respond to incidents (Root Cause Analysis), to plan, and to make sure that people know when requirements or priorities are changing in a significant way.
Would we have had different results if we had implemented standups? Probably. Would the results have been better for the team and for the company? I’m not sure. We found other good ways for the team, and the rest of the company, to track what was going on and what needed to be done next. We’ve had some misunderstandings, and some people did get off track – standups might have helped prevent this. And standups would have helped new people joining the team, to come up to speed. But the rest of the team gelled quickly during our crazy startup days, and built up a high level of trust and openness and shared commitment without standups. Can you build high-performing, collaborative teams without standups? Of course you can.
Standups have a place, especially early on with people who don’t know each other, where everyone is learning and uncertain about what to do and about each other. But don’t do something, or keep doing something if you don’t think that you need to, just because somebody read it in a book or learned about it in a 3 day SCM class. It’s your team, and you can find your own way to succeed.
Wednesday, August 24, 2011
Bugs and Numbers: How many bugs do you have in your code?
If you follow Zero Bug Tolerance of course you’re not supposed to have any bugs to fix after the code is done. But let’s get real. Is there any way to know how many bugs you're missing and will have to fix later, and how many bugs you might already have in your code? Are there any industry measures of code quality that you can use as a starting point?
For questions like these, the first place I look is one of the books by Capers Jones: arguably the leading expert in all things to do with software development metrics. There’s Applied Software Measurement, or Estimating Software Costs, or Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies. These books offer different views into a fascinating set of data that Capers Jones has collected over decades from thousands of different projects. It’s easy to spend hours getting lost in this data set, and reading through and questioning the findings that he draws from it.
So what can we learn from Capers Jones about bugs and defect potentials and defect density rates? A lot, actually.
On average 85% of bugs introduced in design and development are caught before the code is released (this is the average in the US as of 2009). His research shows that this defect removal rate has stayed roughly the same over 20 years, which is disappointing given the advances in tools and methods over that time.
We introduce 5 bugs per Function Point (Capers Jones is awfully fond of measuring everything by Function Points, which are an abstract way of measuring code size), depending on the type of system being built. Web systems are a bit lower surprisingly, at 4 bugs per Function Point; other internal business systems are 5, military systems average around 7. Using backfiring (a crude technique to convert function points back into LOC measures) you can equivalence 1 Function Point to about 50-55 lines of Java code.
For the sake of simplicity, let’s use 1 Function Point = 50 LOC, and keep in mind that all of these numbers are really rough, and that using backfiring techniques to translate Function Points to source code statements introduces a probability of error, but it’s a lot easier than trying to think in Function Points. And all I want here is a rough indicator of how much trouble a team might be in.
If 85% of bugs are hopefully found and fixed before the code is released, this leaves 0.75 bugs per Function Point unfound (and obviously unfixed) in the code when it gets to production. Which means that for a small application of 1,000 Function Points (50,000 or so lines of Java code), you could expect around 750 defects at release .
And this is only accounting for the bugs that you don’t already know about: a lot of code is released with a list of known bugs that the development team hasn’t had a chance to fix, or doesn’t think is worth fixing, or doesn’t know how to fix. And, this is just your code: it doesn’t account for bugs in the technology stack that the application depends on: the frameworks and application platform, database and messaging middleware, and any open source libraries or COTS that you take advantage of.
Of these 750+ bugs around 25% will be severity 1 show stoppers – real production problems that cause something significant to break.
Ouch – no wonder most teams will spend a lot of time on support and fixing bugs after releasing a big system. Of course, if you’re building and releasing software incrementally, you’ll find and fix more of these bugs as you go along, but you’ll still be fixing a lot of bugs in production.
Remember that these are rough averages. And remember (especially the other guys out there), we can’t all be above average, no matter how much we would like to be. For risk management purposes, it might be best to stick with averages, or even consider yourself below the bar.
Also keep in mind that defect potentials increase with the size of the system – big apps have more bugs on average. Not only is there a higher potential to write buggy code in bigger systems, but as the code base gets bigger and more complex it’s also harder to find and fix bugs. So big systems get released with even more bugs, and really big apps with a lot more bugs.
All of this gets worse in maintenance
In maintenance, the average defect potential for making changes is higher than in development, about 6 bugs per Function Point instead of 5. And the chance of finding and fixing mistakes in your changes is lower (83%). This is all because it’s harder to work with legacy code that you didn’t write and don’t understand all that well. So you should expect to release 1.08 bugs per Function Point when changing code in maintenance, instead of 0.75 bugs per Function Point.
And maintenance teams still have to deal with the latent bugs in the system, some of which may hide in the code for years, or forever. This includes heisenbugs and ghosts and weird timing issues and concurrency problems that disappear when you try to debug them. On average, 50% of residual latent defects are found each calendar year. The more people using your code, the faster that these bugs will be found.
Of course, once you find these bugs, you still have to fix them. The average maintenance programmer can be expected to fix around 10 bugs per month – and maybe implement some small enhancements too. That's not a great return on investment.
Then there’s the problem of bug re-injections, or regressions – when a programmer breaks something accidentally as a side-effect of making a fix. On average, programmers fixing a bug will introduce a new bug 7% of the time – and this can run as high as 20% for complex, poorly-structured code. Trying to fix these bad fixes is even worse – programmers trying to fix these mistakes have a 15% chance of still messing up the fix, and a 30% chance of introducing yet another bug as a side effect! It's better to roll-back the fix and start again.
Unfortunately, all of this gets worse over time. Unless you are doing a perfect job of refactoring and continuously simplifying the code, you can expect code complexity to increase an average of between 1% and 3% per year. And most systems get bigger over time, as you add more features and copy-and-paste code (of course you don't do that): the code base for a system under maintenance increases between 5-10% per year. As the code gets bigger and more complex, the chance for more bugs also increases each year.
But what if we’re not average? What if we’re best in class?
What if you are doing an almost perfect job, if you are truly best in class? Capers Jones finds that best in class teams create half as many bugs as average teams (2.5 or fewer defects per Function Point instead of 5), and they find and fix 95% or more of these bugs before the code is released. That sounds impressive - it means only 0.125 bugs per Function Point. But for a 50,000 LOC system, that’s still somewhere around 125 bugs on delivery.
And as for zero bugs? In his analysis of 13,000 projects over a period of more than 40 years, there were 2 projects with no defects reported within a year of release. So you can aspire to it. But don’t depend on it.
For questions like these, the first place I look is one of the books by Capers Jones: arguably the leading expert in all things to do with software development metrics. There’s Applied Software Measurement, or Estimating Software Costs, or Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies. These books offer different views into a fascinating set of data that Capers Jones has collected over decades from thousands of different projects. It’s easy to spend hours getting lost in this data set, and reading through and questioning the findings that he draws from it.
So what can we learn from Capers Jones about bugs and defect potentials and defect density rates? A lot, actually.
On average 85% of bugs introduced in design and development are caught before the code is released (this is the average in the US as of 2009). His research shows that this defect removal rate has stayed roughly the same over 20 years, which is disappointing given the advances in tools and methods over that time.
We introduce 5 bugs per Function Point (Capers Jones is awfully fond of measuring everything by Function Points, which are an abstract way of measuring code size), depending on the type of system being built. Web systems are a bit lower surprisingly, at 4 bugs per Function Point; other internal business systems are 5, military systems average around 7. Using backfiring (a crude technique to convert function points back into LOC measures) you can equivalence 1 Function Point to about 50-55 lines of Java code.
For the sake of simplicity, let’s use 1 Function Point = 50 LOC, and keep in mind that all of these numbers are really rough, and that using backfiring techniques to translate Function Points to source code statements introduces a probability of error, but it’s a lot easier than trying to think in Function Points. And all I want here is a rough indicator of how much trouble a team might be in.
If 85% of bugs are hopefully found and fixed before the code is released, this leaves 0.75 bugs per Function Point unfound (and obviously unfixed) in the code when it gets to production. Which means that for a small application of 1,000 Function Points (50,000 or so lines of Java code), you could expect around 750 defects at release .
And this is only accounting for the bugs that you don’t already know about: a lot of code is released with a list of known bugs that the development team hasn’t had a chance to fix, or doesn’t think is worth fixing, or doesn’t know how to fix. And, this is just your code: it doesn’t account for bugs in the technology stack that the application depends on: the frameworks and application platform, database and messaging middleware, and any open source libraries or COTS that you take advantage of.
Of these 750+ bugs around 25% will be severity 1 show stoppers – real production problems that cause something significant to break.
Ouch – no wonder most teams will spend a lot of time on support and fixing bugs after releasing a big system. Of course, if you’re building and releasing software incrementally, you’ll find and fix more of these bugs as you go along, but you’ll still be fixing a lot of bugs in production.
Remember that these are rough averages. And remember (especially the other guys out there), we can’t all be above average, no matter how much we would like to be. For risk management purposes, it might be best to stick with averages, or even consider yourself below the bar.
Also keep in mind that defect potentials increase with the size of the system – big apps have more bugs on average. Not only is there a higher potential to write buggy code in bigger systems, but as the code base gets bigger and more complex it’s also harder to find and fix bugs. So big systems get released with even more bugs, and really big apps with a lot more bugs.
All of this gets worse in maintenance
In maintenance, the average defect potential for making changes is higher than in development, about 6 bugs per Function Point instead of 5. And the chance of finding and fixing mistakes in your changes is lower (83%). This is all because it’s harder to work with legacy code that you didn’t write and don’t understand all that well. So you should expect to release 1.08 bugs per Function Point when changing code in maintenance, instead of 0.75 bugs per Function Point.
And maintenance teams still have to deal with the latent bugs in the system, some of which may hide in the code for years, or forever. This includes heisenbugs and ghosts and weird timing issues and concurrency problems that disappear when you try to debug them. On average, 50% of residual latent defects are found each calendar year. The more people using your code, the faster that these bugs will be found.
Of course, once you find these bugs, you still have to fix them. The average maintenance programmer can be expected to fix around 10 bugs per month – and maybe implement some small enhancements too. That's not a great return on investment.
Then there’s the problem of bug re-injections, or regressions – when a programmer breaks something accidentally as a side-effect of making a fix. On average, programmers fixing a bug will introduce a new bug 7% of the time – and this can run as high as 20% for complex, poorly-structured code. Trying to fix these bad fixes is even worse – programmers trying to fix these mistakes have a 15% chance of still messing up the fix, and a 30% chance of introducing yet another bug as a side effect! It's better to roll-back the fix and start again.
Unfortunately, all of this gets worse over time. Unless you are doing a perfect job of refactoring and continuously simplifying the code, you can expect code complexity to increase an average of between 1% and 3% per year. And most systems get bigger over time, as you add more features and copy-and-paste code (of course you don't do that): the code base for a system under maintenance increases between 5-10% per year. As the code gets bigger and more complex, the chance for more bugs also increases each year.
But what if we’re not average? What if we’re best in class?
What if you are doing an almost perfect job, if you are truly best in class? Capers Jones finds that best in class teams create half as many bugs as average teams (2.5 or fewer defects per Function Point instead of 5), and they find and fix 95% or more of these bugs before the code is released. That sounds impressive - it means only 0.125 bugs per Function Point. But for a 50,000 LOC system, that’s still somewhere around 125 bugs on delivery.
And as for zero bugs? In his analysis of 13,000 projects over a period of more than 40 years, there were 2 projects with no defects reported within a year of release. So you can aspire to it. But don’t depend on it.
Monday, August 15, 2011
The C14N challenge
Failing to properly validate input data is behind at least half of all application security problems. In order to properly validate input data, you have to start by first ensuring that all data is in the same standard, simple, consistent format – a canonical form. This is because of all the wonderful flexibility in internationalization and data formatting and encoding that modern platforms and especially the Web offer. Wonderful capabilities that attackers can take advantage of to hide malicious code inside data in all sorts of sneaky ways.
Canonicalization is a conceptually simple idea: take data inputs, and convert all of it into a single, simple, consistent normalized internal format before you do anything else with it. But how exactly do you do this, and how do you know that it has been done properly? What are the steps that programmers need to take to properly canonicalize data? And how do you test for it? This is where things get fuzzy as hell.
To read my latest post on canonicalization problems (and the search for solutions), go to the SANS Application Security blog.
Monday, July 11, 2011
Developing and Testing in the Cloud
There’s a lot of hype around “the Cloud” and what it can do.
One of the things that I am interested in is Cloud solutions that can help small software companies, and especially to kickstart software startups. Good tools that development teams can take advantage of to build and test their own stuff, without all of the hassle and expense of internal IT, buying and provisioning their own gear, setting up the tools and systems and networks and finding someone who understands all of it well enough to do it right and to keep it running. I want to find Cloud-based technology that can do for software teams what Salesforce.com does for customer-service businesses, so that software teams can get started off quickly, and doing things right from the start.
Software teams need a core set of shared tools and capabilities:
Managing and Building Code in the Cloud
I keep running into people using GitHub to host their source code repositories. Not just people working on Open Source projects, but companies using GitHub to manage commercial projects in hosted private repositories, a service used by a number of startups. In GitHub, developers have a platform for managing code with the Git distributed version control system, and they get access to a good set of tools for a development team, especially distributed teams: admin functions to control permissioning, wikis, a bug tracking system and an online code review tool.
One alternative to GitHub is BitBucket which uses the Mercurial DVCS. Like GitHub, BitBucket can be used to manage Open Source and private projects, and it looks like it offers a similar set of management capabilities and tools. GitHub is free for Open Source projects and cheap for small teams while BitBucket’s pricing is based on the number of users, small teams (up to 5 users) are free for open or closed source.
Another On Demand SCM platform built on Mercurial is Fog Creek Software’s Kiln which integrates nicely with Fog Creek’s bug tracking and planning system FogBugz.
Atlassian, which bought BitBucket last year also offers Jira Studio a comprehensive hosted development platform centered on a Subversion code repository integrated with the rest of Atlaissian’s strong development toolset: Jira bug tracking, FishEye for code searching, Confluence wiki, Greenhopper for Agile team planning, Elastic Bamboo for build and Continuous Integration on Amazon EC2 and Crucible for online code review. That’s almost everything that a team needs (all that is missing is static analysis and a functional and load testing platform). Presumably the same integrated development tool support will extend to BitBucket over time, although there are already some overlaps between BitBucket and Jira Studio.
And there’s CloudBees Dev@cloud which offers a choice of secure Git, SVN or Maven repositories, and Continuous Integration through a hosted Jenkins server.
Spring Source’s Code2Cloud might be a good option especially for Java / Spring developers when it is ready – the details are fuzzy at the moment.
Google Code is cool, but for now is only available for Open Source projects.
For enterprises, IBM recently announced Smart Business Development and Test Cloud and IBM Smart Business Development and Test on the IBM Cloud (also known as SmartCloud Enterprise - I’m not sure what the difference is between them, I got too worn out reading through the marketing speak) , with all sorts of IBM technology and partner tools. This stuff isn’t for the faint of heart, or the small of wallet.
Continuous Integration in the Cloud
Continuous Integration is a problem that development teams need to solve before they get too far along, and setting up a CI server and keeping it running isn’t trivial. It’s another good fit for the elastic On Demand model of the Cloud, paying for more infrastructure only when you need it.
Besides Atlassian’s hosted end-to-end platform, and CloudBees’ Jenkins as a Service which can be used to build code hosted on a repository accessible through the Internet, there are a few options for CI in the Cloud (courtesy of Pascal Thivent on Stack Overflow):
For Open Source projects there is CodeBetter based on the TeamCity build server. For proprietary projects there is Mike CI which is hosted on Amazon EC2, and includes interfaces to Subversion, Git and Mercurial repositories; and CI Foundry. However, I don’t know how real these services are. Mike CI was down the last couple of times I went to check on it, and CI Foundry’s home page includes this not-very-convincing testimonial:
Bug Tracking in the Cloud
There are some decent and reasonably priced Cloud-based bug tracking systems suitable for small teams, including Fog Creek Software’s FogBugz and Atlassian’s Jira Hosted option both of which are also included in their hosted code management suites.
Testing in the Cloud
The Cloud has a lot of promise for app testing, especially for large-scale performance and load testing. You’re building a cool new Web platform that needs to support 10,000 or 100,000 concurrent sessions according to your business plan. Before your investors put more money in, they want to see just how real your software is, how far away you are from delivering something that could work. Of course you don’t have the money or time or IT skills to build a large-scale test center of your own at this point, and there’s a lot of waste in doing this – you need a lot of gear, but you won’t need to use all of it very often.
You can spin up a test farm on EC2 surprisingly quickly, even a big one – and as long as you don’t need to run the tests for a long time and you don’t ask for too high a service level, it won’t cost you much at all. You pay for what you need when you need it.
To run stress tests you need more than just the app deployed in a farm. You also need a scalable stress test harness to drive load scenarios and to measure the results. There are some viable options available in the Cloud today: SOASTA with its CloudTest solution for load testing and extreme stress testing, scaling from smallish sites to really big; HP Load Runner in the Cloud running on EC2; JMeter in the Cloud; LoadStorm and BrowserMob.
There are also some other interesting test capabilities, including technology from Sauce Labs which lets you run Selenium tests on Web apps in the Cloud or manually test your app against different Cloud-based instantly-available browsers, all with video records of failed tests and other problems found. Simple, useful and cool.
A different take on Cloud-based testing that could work for startups and smaller companies is crowd-sourced testing services like uTest, where you get people “in the Cloud” to test your app (web apps, desktop apps and mobile apps), including functional coverage and regression testing (you give them tests, they run them and maybe help improve them), exploratory testing, usability testing, load testing and test planning and management. Like other Cloud-based On Demand solutions, you pay for work when you need it. uTest has a community of something like 40,000 testers available, although it’s not clear how many of them are much good.
Mob4Hire is another crowd-sourced solution specifically for mobile apps – they offer functional testing and usability testing and other services like market research.
Static Analysis in the Cloud
Some of the leading static analysis suppliers have also gone to the Cloud. HP offers Fortify on Demand to scan code for security vulnerabilities, and IBM has Rational Appscan OnDemand which includes assistance from an experienced team to help you interpret the results and plan remediation. Both of these solutions are targeted more towards the enterprise than small teams.
Veracode, which runs binary code analysis for security vulnerabilities and now also offers web vulnerability scanning, is only available through the Cloud. They offer a free trial scan of small Java apps for XSS and SQL Injection flaws (two of the most common and fatal web application vulnerabilities).
For startups that care about building reliable and secure apps (and most of them should, especially Web 2.0 startups and mobile app developers) it probably makes more sense to look at simpler developer IDE-based tools like Findbugs and Google’s CodePro Analytix (both free) or Klocwork Solo for Java which is priced per user.
The Tradeoffs
Managing your code, building it and testing it in the Cloud has a number of clear advantages:
The concern with using a Cloud platform like this comes down to some fundamental points:
1. It’s a shared platform, used by a lot of companies. The more successful the platform is, the more customers and interesting and valuable data / IP that they manage, the more attractive a target they are to bad guys. A small company by itself is an easy target, but not especially interesting or easy to find. But a platform that holds data for a lot of small, innovative companies? Yummy….[pronounced in a deep, growly voice with a foreign accent]
2. You take on some important risks any time that you outsource something that is core and critical to your business. Your data / code / customer base is more important to you than it is to whoever you outsource this responsibility to. If something goes badly wrong, if you can’t get an important change or an important bug fix out to a customer because the Cloud service isn’t working, or if the integrity or confidentiality of your code is compromised, they will be sad, and they will lose a customer (you)... but you may go out of business.
You can ask for, and pay for, good SLAs - but Amazon’s major EC2 outage earlier this year showed that even the best providers can’t meet their SLA commitments. While Amazon did a lot of things right in handling and recovering from that outage, it affected a lot of customers for too long.
3. There’s more to a secure online platform than using SSL and network firewalls and running in a good data center - pretending otherwise is at best naïve. As a customer, you need to be confident in how the Cloud services provider designed and built the software architecture and platform to protect the confidentiality and integrity of your IP and data, in their multi-tenant partitioning scheme and software security controls, and in their SDLC. Security vulnerabilities in application code are a serious source of risk to businesses on the Web, and most companies still do a poor job of building apps in a secure way.
DropBox, a Cloud-based document storage facility is an example of a platform that seemed to be secure and confidential, until somebody started to look deeper into it, and they continue to run into problems with basic security issues Unfortunately, none of the hosted platforms provide any kind of statement on their secure SDLC that I could find, on what steps they take to ensure that the platform is designed and implemented safely.
Testing in the Cloud, especially load testing for Web apps, looks to be a no-brainer – the business case is too compelling to ignore. Many of the other tools have a lot of promise: something like Atlassian’s end-to-end hosted studio would save a lot of time and trouble for a small company, and the upfront cost savings are real.
It comes down to how much trust you can put into your Cloud provider and their SLAs for availability and support, and whether in the end you can afford to trust your core IP to somebody else. 37 Signals, which builds and operates its own Cloud solutions for project management and document sharing, doesn’t think it is a good idea:
One of the things that I am interested in is Cloud solutions that can help small software companies, and especially to kickstart software startups. Good tools that development teams can take advantage of to build and test their own stuff, without all of the hassle and expense of internal IT, buying and provisioning their own gear, setting up the tools and systems and networks and finding someone who understands all of it well enough to do it right and to keep it running. I want to find Cloud-based technology that can do for software teams what Salesforce.com does for customer-service businesses, so that software teams can get started off quickly, and doing things right from the start.
Software teams need a core set of shared tools and capabilities:
- Source code management / version control
- Bug tracking
- Collaboration and shared documentationBuild and Continuous Integration
- Source code scanning and static analysis
- Unit and functional testing
- System, load and stress testing
- Code deployment.
Managing and Building Code in the Cloud
I keep running into people using GitHub to host their source code repositories. Not just people working on Open Source projects, but companies using GitHub to manage commercial projects in hosted private repositories, a service used by a number of startups. In GitHub, developers have a platform for managing code with the Git distributed version control system, and they get access to a good set of tools for a development team, especially distributed teams: admin functions to control permissioning, wikis, a bug tracking system and an online code review tool.
One alternative to GitHub is BitBucket which uses the Mercurial DVCS. Like GitHub, BitBucket can be used to manage Open Source and private projects, and it looks like it offers a similar set of management capabilities and tools. GitHub is free for Open Source projects and cheap for small teams while BitBucket’s pricing is based on the number of users, small teams (up to 5 users) are free for open or closed source.
Another On Demand SCM platform built on Mercurial is Fog Creek Software’s Kiln which integrates nicely with Fog Creek’s bug tracking and planning system FogBugz.
Atlassian, which bought BitBucket last year also offers Jira Studio a comprehensive hosted development platform centered on a Subversion code repository integrated with the rest of Atlaissian’s strong development toolset: Jira bug tracking, FishEye for code searching, Confluence wiki, Greenhopper for Agile team planning, Elastic Bamboo for build and Continuous Integration on Amazon EC2 and Crucible for online code review. That’s almost everything that a team needs (all that is missing is static analysis and a functional and load testing platform). Presumably the same integrated development tool support will extend to BitBucket over time, although there are already some overlaps between BitBucket and Jira Studio.
And there’s CloudBees Dev@cloud which offers a choice of secure Git, SVN or Maven repositories, and Continuous Integration through a hosted Jenkins server.
Spring Source’s Code2Cloud might be a good option especially for Java / Spring developers when it is ready – the details are fuzzy at the moment.
Google Code is cool, but for now is only available for Open Source projects.
For enterprises, IBM recently announced Smart Business Development and Test Cloud and IBM Smart Business Development and Test on the IBM Cloud (also known as SmartCloud Enterprise - I’m not sure what the difference is between them, I got too worn out reading through the marketing speak) , with all sorts of IBM technology and partner tools. This stuff isn’t for the faint of heart, or the small of wallet.
Continuous Integration in the Cloud
Continuous Integration is a problem that development teams need to solve before they get too far along, and setting up a CI server and keeping it running isn’t trivial. It’s another good fit for the elastic On Demand model of the Cloud, paying for more infrastructure only when you need it.
Besides Atlassian’s hosted end-to-end platform, and CloudBees’ Jenkins as a Service which can be used to build code hosted on a repository accessible through the Internet, there are a few options for CI in the Cloud (courtesy of Pascal Thivent on Stack Overflow):
For Open Source projects there is CodeBetter based on the TeamCity build server. For proprietary projects there is Mike CI which is hosted on Amazon EC2, and includes interfaces to Subversion, Git and Mercurial repositories; and CI Foundry. However, I don’t know how real these services are. Mike CI was down the last couple of times I went to check on it, and CI Foundry’s home page includes this not-very-convincing testimonial:
What people are saying...Thivent suggests that you can roll-your- own CI solution on Amazon EC2 or maybe use something like “CI in a box”. Using an On Demand platform like EC2 makes sense for CI, rather than provisioning your own gear: it’s a bit more work for you to do the setup, but you can rely on Amazon’s infrastructure to reduce costs and simplify operations.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris quis ligula vel ligula varius cursus tristique eget sem. Mauris sagittis, ante id imperdiet auctor, sem mi condimentum neque, quis feugiat quam dui dictum risus.
- Chris Read
Continuous Deployment expert
Bug Tracking in the Cloud
There are some decent and reasonably priced Cloud-based bug tracking systems suitable for small teams, including Fog Creek Software’s FogBugz and Atlassian’s Jira Hosted option both of which are also included in their hosted code management suites.
Testing in the Cloud
The Cloud has a lot of promise for app testing, especially for large-scale performance and load testing. You’re building a cool new Web platform that needs to support 10,000 or 100,000 concurrent sessions according to your business plan. Before your investors put more money in, they want to see just how real your software is, how far away you are from delivering something that could work. Of course you don’t have the money or time or IT skills to build a large-scale test center of your own at this point, and there’s a lot of waste in doing this – you need a lot of gear, but you won’t need to use all of it very often.
You can spin up a test farm on EC2 surprisingly quickly, even a big one – and as long as you don’t need to run the tests for a long time and you don’t ask for too high a service level, it won’t cost you much at all. You pay for what you need when you need it.
To run stress tests you need more than just the app deployed in a farm. You also need a scalable stress test harness to drive load scenarios and to measure the results. There are some viable options available in the Cloud today: SOASTA with its CloudTest solution for load testing and extreme stress testing, scaling from smallish sites to really big; HP Load Runner in the Cloud running on EC2; JMeter in the Cloud; LoadStorm and BrowserMob.
There are also some other interesting test capabilities, including technology from Sauce Labs which lets you run Selenium tests on Web apps in the Cloud or manually test your app against different Cloud-based instantly-available browsers, all with video records of failed tests and other problems found. Simple, useful and cool.
A different take on Cloud-based testing that could work for startups and smaller companies is crowd-sourced testing services like uTest, where you get people “in the Cloud” to test your app (web apps, desktop apps and mobile apps), including functional coverage and regression testing (you give them tests, they run them and maybe help improve them), exploratory testing, usability testing, load testing and test planning and management. Like other Cloud-based On Demand solutions, you pay for work when you need it. uTest has a community of something like 40,000 testers available, although it’s not clear how many of them are much good.
Mob4Hire is another crowd-sourced solution specifically for mobile apps – they offer functional testing and usability testing and other services like market research.
Static Analysis in the Cloud
Some of the leading static analysis suppliers have also gone to the Cloud. HP offers Fortify on Demand to scan code for security vulnerabilities, and IBM has Rational Appscan OnDemand which includes assistance from an experienced team to help you interpret the results and plan remediation. Both of these solutions are targeted more towards the enterprise than small teams.
Veracode, which runs binary code analysis for security vulnerabilities and now also offers web vulnerability scanning, is only available through the Cloud. They offer a free trial scan of small Java apps for XSS and SQL Injection flaws (two of the most common and fatal web application vulnerabilities).
For startups that care about building reliable and secure apps (and most of them should, especially Web 2.0 startups and mobile app developers) it probably makes more sense to look at simpler developer IDE-based tools like Findbugs and Google’s CodePro Analytix (both free) or Klocwork Solo for Java which is priced per user.
The Tradeoffs
Managing your code, building it and testing it in the Cloud has a number of clear advantages:
- Faster time to market – you can get access to the infrastructure and tools quickly, with minimal hassle and setup time.
- Savings on cost and Capex – especially for load testing – you don’t have to pay upfront, you only have to pay for what you use when you use it, so you can better manage your cash flow.
- Convenience – you don’t have to provision and operate the gear yourself and add to your operational headaches.
- Access to specialist skills – your Cloud provider will know more about these technologies and how to solve a specific problem much better than you can afford to, so you don’t need to contract or hire gurus to help solve problems or make sure everything is setup properly, or waste time figuring it out yourself.
- Service levels – they’ll have more people to help keep things running, to make sure code and data are backed up, systems are cleaned up and monitored and patched. They’re less likely to lose your code and data than you are.
The concern with using a Cloud platform like this comes down to some fundamental points:
1. It’s a shared platform, used by a lot of companies. The more successful the platform is, the more customers and interesting and valuable data / IP that they manage, the more attractive a target they are to bad guys. A small company by itself is an easy target, but not especially interesting or easy to find. But a platform that holds data for a lot of small, innovative companies? Yummy….[pronounced in a deep, growly voice with a foreign accent]
2. You take on some important risks any time that you outsource something that is core and critical to your business. Your data / code / customer base is more important to you than it is to whoever you outsource this responsibility to. If something goes badly wrong, if you can’t get an important change or an important bug fix out to a customer because the Cloud service isn’t working, or if the integrity or confidentiality of your code is compromised, they will be sad, and they will lose a customer (you)... but you may go out of business.
You can ask for, and pay for, good SLAs - but Amazon’s major EC2 outage earlier this year showed that even the best providers can’t meet their SLA commitments. While Amazon did a lot of things right in handling and recovering from that outage, it affected a lot of customers for too long.
3. There’s more to a secure online platform than using SSL and network firewalls and running in a good data center - pretending otherwise is at best naïve. As a customer, you need to be confident in how the Cloud services provider designed and built the software architecture and platform to protect the confidentiality and integrity of your IP and data, in their multi-tenant partitioning scheme and software security controls, and in their SDLC. Security vulnerabilities in application code are a serious source of risk to businesses on the Web, and most companies still do a poor job of building apps in a secure way.
DropBox, a Cloud-based document storage facility is an example of a platform that seemed to be secure and confidential, until somebody started to look deeper into it, and they continue to run into problems with basic security issues Unfortunately, none of the hosted platforms provide any kind of statement on their secure SDLC that I could find, on what steps they take to ensure that the platform is designed and implemented safely.
Testing in the Cloud, especially load testing for Web apps, looks to be a no-brainer – the business case is too compelling to ignore. Many of the other tools have a lot of promise: something like Atlassian’s end-to-end hosted studio would save a lot of time and trouble for a small company, and the upfront cost savings are real.
It comes down to how much trust you can put into your Cloud provider and their SLAs for availability and support, and whether in the end you can afford to trust your core IP to somebody else. 37 Signals, which builds and operates its own Cloud solutions for project management and document sharing, doesn’t think it is a good idea:
We host all the source code for our applications internally for obvious security reasons. That’s not to say Github’s private repository hosting isn’t a good option, especially if you want a hassle-free setup. It’s just not for us.For some companies, especially startups, IP embodied in their code is critically important – it may be all that they have. Ironically, it’s these companies that are most likely to use a Cloud platform like GitHub or Jira Studio or BitBucket. Deciding whether to trust your future to the Cloud is not an easy decision to make. But as the technology continues to get better, it’s a question that is worth asking.
JH 22 Aug 08
Friday, July 1, 2011
Please, no more Manifestos
For some reason, people involved in software development have a thing for Manifestos (always with a Capital M). It all started with the Agile Manifesto. Then came the Software Craftsmanship Manifesto, signed by serious programmers big and small, except ironically the original author of Software Craftsmanship.
Then there have been other, less practical and less successful manifestos, like the Rugged Software Manifesto which as I have explained before was proclaimed with the naïve understanding that reciting feel-good phrases will somehow make developers write better software.
What brought this all up is that some people involved in DevOps still feel incomplete without a Manifesto of their very own.
This is a shame, because I like following what’s happening in DevOps and learning from the best (and worst) of it, and I want them to keep doing what they’re doing.
I don’t see the point in manifestos. They don’t move me or change the way that I think or work. I can get through each day without having to refer to a manifesto. I want tools and concrete ideas that I can use to get things done, to do a better job. Not motherhood or bullshit. Patterns and anti-patterns and recipes and best practices (and worst practices) – these are useful. But Manifestos? Useless, or at their worst, dangerous:
Manifestos s are a way to keep people from thinking and asking questions.
Or maybe Andrew Shafer is right
Then there have been other, less practical and less successful manifestos, like the Rugged Software Manifesto which as I have explained before was proclaimed with the naïve understanding that reciting feel-good phrases will somehow make developers write better software.
“I am rugged…”And the even-less-practical but much-more fun-to-read manifesto for truly hard-core programmers (there really is no problem that you can’t program your way out of) and of course the cynical Half-Arsed Agile Manifesto.
What brought this all up is that some people involved in DevOps still feel incomplete without a Manifesto of their very own.
This is a shame, because I like following what’s happening in DevOps and learning from the best (and worst) of it, and I want them to keep doing what they’re doing.
I don’t see the point in manifestos. They don’t move me or change the way that I think or work. I can get through each day without having to refer to a manifesto. I want tools and concrete ideas that I can use to get things done, to do a better job. Not motherhood or bullshit. Patterns and anti-patterns and recipes and best practices (and worst practices) – these are useful. But Manifestos? Useless, or at their worst, dangerous:
Workers of the world unite….Look how that one worked out.
Manifestos s are a way to keep people from thinking and asking questions.
Oh no, people have more than one way of thinking about a problem, or what something means!This is not a bad thing. This is a GOOD THING! This is how we move forward, this is how we get better. What we’re doing isn’t simple, or right, or wrong, so let’s stop pretending.
Or maybe Andrew Shafer is right
A manifesto, by definition, is essentially a marketing document.Either way, we don’t need more manifestos. What we need is people to keep thinking and asking questions, and getting things done.
“Re: Time for a DevOps Manifesto?”
Monday, June 27, 2011
Moving forward from Failure
System failures at scale are inescapable, as I have talked about before in the context of designing systems for failure in high-scale computing and how to apply these ideas to enterprise systems. Failures are wasted if you don’t learn enough from them; if the way that you design and deliver application systems, and the way that you deploy and run these systems, don’t get better as a result. You have to find lessons in each failure and constantly move forward, and make the system, and the team, more resilient.
I’ve read and heard a lot of good stuff over the last year or so from the DevOps community on handling system failures: how to minimize the risk of failures through deployment planning, how complex systems fail, how to communicate failures effectively to stakeholders so that you don’t destroy trust, and how to understand failures through postmortem analysis, including Jacob Loomis’ essay “How to Make Failure Beautiful” in the Web Operations book and John Allspaw’s keynote on Advanced Postmortem Fu and Human Error 101 at this year’s Velocity conference.
Many of these ideas fit with my own experience, and fill in some important gaps – they extend beyond the concerns of operating high-volume Web sites to broader, general problems in system operations and systems engineering, and deserve a wider audience outside of the Web operations and Web startup worlds.
Postmortems and Root Cause Analysis
The keys to successful postmortems and Root Cause Analysis are deceptively simple, and therefore difficult to get right:
Root Cause Analysis is important, but it is just the first step. Once you’ve reviewed a failure as a team and found your way to the root causes or at least identified some real problems, now you have to do something about it, beyond the immediate fixes and workarounds that the team made recovering from the incident.
There are straightforward things that you can do now – detective and corrective actions, quick fixes, low cost and low risk – so you do them. Fix a bug, plug a hole, add defensive coding, add some tests. Better logging and diagnostics and alerting, better error handling, better metrics – you can always do this stuff better – so that ops can see the problem coming or at least recognize it when it happens again. Better troubleshooting tools and training for ops.
Don’t stop with corrective actions either
But it’s still not enough to recover from a failure and patch the holes and tighten things up, or even build a better incident management capability. You have to make sure to find the lessons in each failure, really learn about why the failure happened, how it relates to other failures or problems that you have; and about how you dealt with the failure when it happened, and what you need to do to do a better job the next time. Then you have to act to reduce the risk and cost of failures, take steps to make sure that failures like this one, and even failures that aren’t like this one, don’t happen again. And that when the next failure does happen, that the team is more prepared for it, that you can recover faster and with less stress and impact to the business.
Some of these preventative actions are straightforward: training for developers and testers and ops, and better communications in and especially between teams, checklists, health checks and dependency checks, more disciplined change controls and deployment controls, and other safeties.
But most preventative changes, the ones that make a real difference, are deeper, more fundamental: fixing architecture, culture, organization, or management weaknesses. Fixing these kinds of problems takes longer, involves more people, and costs more. Making changes to the architecture – switching out a core technology or implementing a partitioning strategy to contain failures and to scale up – can take months to work out and implement. Organizational and management changes are hard because this directly impacts people’s jobs. Changing culture is even harder and takes longer, especially to make the changes stick.
You can’t improve what you can’t see – you need data
These problems are also hard to see and understand. It’s naïve to think that you can recognize or prove the need for fundamental changes to architecture or organization or management controls or culture from a postmortem meeting – it takes time and perspective and experience with more than one failure to see this kind of problem. And because the fixes are fundamental and expensive, they can be hard to justify, hard to get management and the business and your own people to buy-in.
You need data to help you understand what you’re doing wrong, what you need to change or what you need to stop doing – and what you’re doing right, what you need to keep doing, and to build a case for change. At last year’s Velocity conference, John Allspaw explained how to use metrics to find patterns and trends in failures, to understand what’s hurting you the most, or hurting you the most often, what’s working well and what’s not. To find out what changes are safe, and what changes aren’t; what failures are cheap and easy to recover from, and what failures can take the company down.
Metrics are important to help you see problems, and to help show if you are learning and getting better. Track the types and severity of failures, frequency of failures, your response to failures – time to detect and time to recover from failures; and the frequency and type and size of changes, and the correlation between changes and failures (by type and size of changes, by type and severity of failures). And make sure to track regressions – the number of times that you take a step backward.
Deciding what to fix, and what not to
Metrics can also help you to decide what you can fix and change, and what’s not worth fixing or changing – at least not for now. You need to recognize when a problem is a true outlier, an isolated case that is unlikely to happen again and that you can accept and move forward. It’s important that you can see the difference between a problem that is a one-of, and a first-of –an indicator of something deeply wrong, a fundamental weakness in the way that the system was built or the way that you work. You don’t want to over-react to outliers but you also don’t can’t treat every problem as unique and miss seeing the patterns and connections, the underlying root causes.
There is diminishing returns in preventing (or even reducing the probability) of some problems, trying to stop the unstoppable. Some problems are too rare, or too expensive or difficult to prevent – and trying to prevent these problems can introduce new complexities and new risks into the system. I agree with John Allspaw that there are situations when it makes more sense to focus on how to react and recover faster, on improving MTTR rather than MTBF, but you need to know when to make that case.
Moving forward from Failures is real work and needs to be managed
People don’t want to think about failures for long – they want to put the mistakes and confusion and badness behind them and move on. And the business and management want priorities back on day-to-day delivery as soon as possible. When failures happen, you need to act fast, get the most that you can out of the moment, make decisions and commitments and reinforce changes while the pain is still fresh.
But fixing real problems in the system or the way that people are organized or the way that they work or the way that they think, can’t be done right away. Building reliability and resilience and operability into how you work and how you think and how you build and test software and how you solve problems takes time and commitment and continuous reinforcement. Management, and the team, have to be made accountable for longer-term preventative actions, for making bigger changes and improvements.
This work needs to be recognized by the business and management and the team and built in to the backlog, and actively managed. You need to remind people of the risks when you see them missing steps or trying to cut corners, or falling back into old patterns of thinking and acting. You’ll need to use metrics and cost data to drive behaviour and to drive change, and to decide how much to push and how often: are you changing too much too often, running too loose; or is change costing you too much, are you overcompensating?
A Resilient system is a DevOps problem
Development can’t solve all of these problems on its own, and neither can Operations. It needs the commitment of development (and by development, I mean everyone responsible for designing, developing, testing, securing and releasing the software) and operations (everyone responsible for building and securing and running the infrastructure, deploying the code, and running and monitoring the system). This is a real DevOps problem: development and operations have to be aligned and work together, communicating and collaborating, sharing responsibilities and technology. This is one of the places where I see DevOps adding real value in any organization.
I’ve read and heard a lot of good stuff over the last year or so from the DevOps community on handling system failures: how to minimize the risk of failures through deployment planning, how complex systems fail, how to communicate failures effectively to stakeholders so that you don’t destroy trust, and how to understand failures through postmortem analysis, including Jacob Loomis’ essay “How to Make Failure Beautiful” in the Web Operations book and John Allspaw’s keynote on Advanced Postmortem Fu and Human Error 101 at this year’s Velocity conference.
Many of these ideas fit with my own experience, and fill in some important gaps – they extend beyond the concerns of operating high-volume Web sites to broader, general problems in system operations and systems engineering, and deserve a wider audience outside of the Web operations and Web startup worlds.
Postmortems and Root Cause Analysis
The keys to successful postmortems and Root Cause Analysis are deceptively simple, and therefore difficult to get right:
- get the right people together in a room.
- create the right environment for blameless problem solving: make it safe for people to be open and honest, don’t point fingers, focus on facts and solving problems and how to get better.
- postmortems are expensive and painful, and people want to get out of them as soon as possible. Don’t stop too soon, before the team has really understood the problems, and the solutions.
- don’t be satisfied with a single root cause – complex system failures usually have multiple causes if you dig deep enough.
- human error isn’t a root cause – it’s a symptom of something else that you’ve done wrong.
Root Cause Analysis is important, but it is just the first step. Once you’ve reviewed a failure as a team and found your way to the root causes or at least identified some real problems, now you have to do something about it, beyond the immediate fixes and workarounds that the team made recovering from the incident.
There are straightforward things that you can do now – detective and corrective actions, quick fixes, low cost and low risk – so you do them. Fix a bug, plug a hole, add defensive coding, add some tests. Better logging and diagnostics and alerting, better error handling, better metrics – you can always do this stuff better – so that ops can see the problem coming or at least recognize it when it happens again. Better troubleshooting tools and training for ops.
Don’t stop with corrective actions either
But it’s still not enough to recover from a failure and patch the holes and tighten things up, or even build a better incident management capability. You have to make sure to find the lessons in each failure, really learn about why the failure happened, how it relates to other failures or problems that you have; and about how you dealt with the failure when it happened, and what you need to do to do a better job the next time. Then you have to act to reduce the risk and cost of failures, take steps to make sure that failures like this one, and even failures that aren’t like this one, don’t happen again. And that when the next failure does happen, that the team is more prepared for it, that you can recover faster and with less stress and impact to the business.
Some of these preventative actions are straightforward: training for developers and testers and ops, and better communications in and especially between teams, checklists, health checks and dependency checks, more disciplined change controls and deployment controls, and other safeties.
But most preventative changes, the ones that make a real difference, are deeper, more fundamental: fixing architecture, culture, organization, or management weaknesses. Fixing these kinds of problems takes longer, involves more people, and costs more. Making changes to the architecture – switching out a core technology or implementing a partitioning strategy to contain failures and to scale up – can take months to work out and implement. Organizational and management changes are hard because this directly impacts people’s jobs. Changing culture is even harder and takes longer, especially to make the changes stick.
You can’t improve what you can’t see – you need data
These problems are also hard to see and understand. It’s naïve to think that you can recognize or prove the need for fundamental changes to architecture or organization or management controls or culture from a postmortem meeting – it takes time and perspective and experience with more than one failure to see this kind of problem. And because the fixes are fundamental and expensive, they can be hard to justify, hard to get management and the business and your own people to buy-in.
You need data to help you understand what you’re doing wrong, what you need to change or what you need to stop doing – and what you’re doing right, what you need to keep doing, and to build a case for change. At last year’s Velocity conference, John Allspaw explained how to use metrics to find patterns and trends in failures, to understand what’s hurting you the most, or hurting you the most often, what’s working well and what’s not. To find out what changes are safe, and what changes aren’t; what failures are cheap and easy to recover from, and what failures can take the company down.
Metrics are important to help you see problems, and to help show if you are learning and getting better. Track the types and severity of failures, frequency of failures, your response to failures – time to detect and time to recover from failures; and the frequency and type and size of changes, and the correlation between changes and failures (by type and size of changes, by type and severity of failures). And make sure to track regressions – the number of times that you take a step backward.
Deciding what to fix, and what not to
Metrics can also help you to decide what you can fix and change, and what’s not worth fixing or changing – at least not for now. You need to recognize when a problem is a true outlier, an isolated case that is unlikely to happen again and that you can accept and move forward. It’s important that you can see the difference between a problem that is a one-of, and a first-of –an indicator of something deeply wrong, a fundamental weakness in the way that the system was built or the way that you work. You don’t want to over-react to outliers but you also don’t can’t treat every problem as unique and miss seeing the patterns and connections, the underlying root causes.
There is diminishing returns in preventing (or even reducing the probability) of some problems, trying to stop the unstoppable. Some problems are too rare, or too expensive or difficult to prevent – and trying to prevent these problems can introduce new complexities and new risks into the system. I agree with John Allspaw that there are situations when it makes more sense to focus on how to react and recover faster, on improving MTTR rather than MTBF, but you need to know when to make that case.
Moving forward from Failures is real work and needs to be managed
People don’t want to think about failures for long – they want to put the mistakes and confusion and badness behind them and move on. And the business and management want priorities back on day-to-day delivery as soon as possible. When failures happen, you need to act fast, get the most that you can out of the moment, make decisions and commitments and reinforce changes while the pain is still fresh.
But fixing real problems in the system or the way that people are organized or the way that they work or the way that they think, can’t be done right away. Building reliability and resilience and operability into how you work and how you think and how you build and test software and how you solve problems takes time and commitment and continuous reinforcement. Management, and the team, have to be made accountable for longer-term preventative actions, for making bigger changes and improvements.
This work needs to be recognized by the business and management and the team and built in to the backlog, and actively managed. You need to remind people of the risks when you see them missing steps or trying to cut corners, or falling back into old patterns of thinking and acting. You’ll need to use metrics and cost data to drive behaviour and to drive change, and to decide how much to push and how often: are you changing too much too often, running too loose; or is change costing you too much, are you overcompensating?
A Resilient system is a DevOps problem
Development can’t solve all of these problems on its own, and neither can Operations. It needs the commitment of development (and by development, I mean everyone responsible for designing, developing, testing, securing and releasing the software) and operations (everyone responsible for building and securing and running the infrastructure, deploying the code, and running and monitoring the system). This is a real DevOps problem: development and operations have to be aligned and work together, communicating and collaborating, sharing responsibilities and technology. This is one of the places where I see DevOps adding real value in any organization.
Subscribe to:
Posts (Atom)
