Rolling Forward and other Deployment Myths

There is more and more writing on Devops lately, which is good and bad. There still remains a small core of thoughtful people that are worth listening to and learning from. There’s more and more marketing from vendors and consultants jumping on the Devops bandwagon. There’s some naïve silliness (“Hire wicked smart people and give them all access to root.”) which can probably be safely ignored. And then there’s stuff that is half-right and half-wrong, too dangerous to be ignored. Like this recent post on Rollbacks and Other Deployment Myths, in which John Vincent lists 5 “myths” about system deployment, which I want to take some time to respond to here.

Change is change?

The author tries to make the point that “Change is neither good or bad. It’s just change.” Therefore we do not need to be afraid of making changes.

I don’t agree. This attitude to change ignores the fact that once a system is up and running and customers are relying on it to conduct their business, whatever change you are making to the system is almost never as important as making sure that the system keeps running properly. Unfortunately, changes often lead to problems. We know from Visible Ops that based on studies of hundreds of companies, 80% of operational failures are caused by mistakes made during changes. This is where heavyweight process control frameworks like ITIL and COBIT, and detective change control tools like Tripwire came from. To help companies get control over IT change, because people had to find some way to stop shit from breaking.

Yes, I get the point that in IT we tend to over-compensate, and I agree that calling in the Release Police and trying to put up a wall around all changes isn’t sustainable. People don’t have to work this way. But trivializing change, pretending that changes don't lead to problems, is dangerous.

Deploys are not risky?

You can be smart and careful and break changes down into small steps and try to automate code pushes and configuration changes, and plan ahead and stage and review and test all your changes, and after all of this you can still mess up the deploy. Even if you make frequent small changes and simplify the work and practice it a lot.

For systems like Facebook and online games and a small number of other cases, maybe deployment really is a non-issue. I don’t care if Facebook deploys in the middle of the day – I can usually tell when they are doing a “zero downtime” deploy (or maybe they are “transparently” recovering from a failure) because data disappears temporarily or shows up in the wrong order, functions aren’t accessible for a while, forms don’t resolve properly, and other weird shit happens, and then things come back later or they don’t. As a customer, do I care? No. It’s an inconvenience, and it’s occasionally unsettling ("WTF just happened?"), but I get used to it and so do millions of others. That’s because most of us don’t use Facebook or systems like this for anything important.

For business-critical systems handling thousands of transactions a second that are tied into hundreds of other company’s systems (the world that I work in) this doesn’t cut it. Maybe I spend too much time at this extreme, where even small problems with compatibility that only affect a small number of customers, or slight and temporary performance slow downs, are a big deal. But most people I work with and talk to in software development and maintenance and system operations agree that deployment is a big deal and needs to be done with care and attention, no matter how simple and small the changes are and no matter how clean and simple and automated the deployment process is.

Rollbacks are a myth?

Vincent wants us to “understand that it’s typically more risky to rollback than rolling forward. Always be rolling forward.”

Not even the Continuous Deployment advocates (who are often some of the most radical – and I think some of the most irresponsible – voices in the Devops community) agree with this – they still roll back if they find problems with changes.

“Rollbacks are a myth” is an echo of the “real men fail forward” crap I heard at Velocity last year and it is where I draw the line. It's one thing to state an extreme position for argument's sake or put up a straw man – but this is just plain wrong.

If you're going to deploy, you have to anticipate roll back and think about it when you make changes and you have to test rolling back to make sure that it works. All of this is hard. But without a working roll back you have no choice other than to fail forward (whatever this means, because nobody who talks about it actually explains how to do it), and this is putting your customers and their business at unnecessary risk. It’s not another valid way of thinking. It’s irresponsible.

James Hamilton wrote an excellent paper on Designing and Delivering Internet-Scale Services when he was at Microsoft (now he’s a an executive and Distinguished Engineer at Amazon). Hamilton’s paper remains one of the smartest things that anyone has written about how to deal with deployment and operational problems at scale. Everyone who designs, builds, maintains or operates an online system should read it. His position on roll back is simple and obvious and right:

Reverting to the previous version is a rip cord that should always be available on any deployment.

Everything fails. Embrace failure.

I agree that everything can and will fail some day, that we can’t pretend that we can prevent failures in any system. But I don’t agree with embracing failure, at least in business-critical enterprise systems, where recovering from a failure means lost business and requires unraveling chains of transactions between different upstream and downstream systems and different companies, messing up other companies' businesses as well as your own and dealing the follow-on compliance problems. Failures in these kinds of systems, and a lot of other systems, are ugly and serious, and they should be treated seriously.

We do whatever we can to make sure that failures are controlled and isolated, and we make sure that we can recover quickly if something goes wrong (which includes being able to roll back!). But we also do everything that we can to prevent failures. Embracing failure is fine for online consumer web site startups – let’s leave it to them.

SLAs

I wanted to respond to the points about SLAs, but it’s not clear to me what the author was trying to say. SLAs are not about servers. Umm, yes that’s right of course...

SLAs are important to set business expectations with your customers (the people who are using the system) and with your partners and suppliers. So that your partners and suppliers know what you need from them and what you are paying them for, and so that you know if you can depend on them when you have to. So that your customers know what they are paying you for. And SLAs (not just high-level uptime SLAs, but SLAs for Recovery Time and Recovery Point goals and incident response and escalation) are important so that your team understands the constraints that they need to work under, what trade-offs to make in design and implementation and in operations.

Under-compensating is worse than Over-compensating

I spent more time than I thought I would responding to this post, because some of what the author says is right – especially in the second part of his post, Deploy All the Things where he provides some good practical advice on how to reduce risk in deployment. He’s right that Operations main purpose isn’t to stop change – it can’t be. We have to be able to keep changing, and developers and operations have to work together to do this in safe and efficient ways. But trivializing the problems and risks of change and over-simplifying how to deal with these risks and how to deal with failures, isn’t the way to do this. There has to be a middle way between the ITIL and COBIT world of controls and paper and process, and cool Web startups failing forward, a way that can really work for the rest of us.

You can’t be Agile in Maintenance?

I’ve been going over a couple of posts by Steve Kilner that question whether Agile methods can be used effectively in software maintenance. It’s a surprising question really. There are a lot of maintenance teams who have had success following Agile methods like Scrum and Extreme Programming (XP) for some time now. We’ve been doing it for almost 5 years, enhancing and maintaining and supporting enterprise systems, and I know that it works.

Agile development naturally leads into maintenance – the goal of incremental Agile development is to get working software out to customers as soon as possible, and get customers using it. At some point, when customers are relying on the software to get real business done and need support and help to keep the system running, teams cross from development over to maintenance. But there’s no reason for Agile development teams to fundamentally change the way that they work when this happens.

It is harder to introduce Agile practices into a legacy maintenance team – there are a lot of technical requirements and some cultural changes that need to be made. But most maintenance teams have little to lose and lots to gain from borrowing from what Agile development teams are doing. Agile methods are designed to help small teams deal with a lot of change and uncertainty, and to deliver software quickly – all things that are at least as important in maintenance as they are in development. Technical practices in Extreme Programming especially help ensure that the code is always working – which is even more important in maintenance than it is in development, because the code has to work the first time in production.

Agile methods have to be adapted to maintenance, but most teams have found it necessary to adapt these methods to fit their situations anyways. Let’s look at what works and what has to be changed to make Agile methods like Scrum and XP work in maintenance.

What works well and what doesn’t

Planning Game

Managing maintenance isn’t the same as managing a development project – even an Agile development project. Although Agile development teams expect to deal with ambiguity and constant change, maintenance teams need to be even more flexible and responsive, to manage conflicts and unpredictable resourcing problems. Work has to be continuously reviewed and prioritized as it comes in – the customer can’t wait for 2 weeks for you to look at a production bug. The team needs a fast path for urgent changes and especially for hot fixes.

You have to be prepared for support demands and interruptions. Structure the team so that some people can take care of second-level support, firefighting and emergency bug fixing and the rest of the team can keep moving forward and get something done. Build slack into schedules to allow for last-minute changes and support escalation.

You will also have to be more careful in planning out maintenance work, to take into account technical and operational dependencies and constraints and risks. You’re working in the real world now, not the virtual reality of a project.

Standups

Standups play an important role in Agile projects to help teams come up to speed and bond. But most maintenance teams work fine without standups – since a lot of maintenance work can be done by one person working on their own, team members don’t need to listen to each other each morning talking about what they did yesterday and what they’re going to do – unless the team is working together on major changes. If someone has a question or runs into a problem, they can ask for help without waiting until the next day.

Small releases

Most changes and fixes that maintenance teams need to make are small, and there is almost always pressure from the business to get the code out as soon as it is ready, so an Agile approach with small and frequent releases makes a lot of sense. If the time boxes are short enough, the customer is less likely to interrupt and re-prioritize work in progress – most businesses can wait a few days or a couple of weeks to get something changed.

Time boxing gives teams a way to control and structure their work, an opportunity to batch up related work to reduce development and testing costs, and natural opportunities to add in security controls and reviews and other gates. It also makes maintenance work more like a project, giving the team a chance to set goals and to see something get done. But time boxing comes with overhead – the planning and setup at the start, then deployment and reviews at the end – all of which adds up over time. Maintenance teams need to be ruthless with ceremonies and meetings, pare them down, keep only what’s necessary and what works.

It’s even more important in maintenance than in development to remember that the goal is to deliver working code at the end of each time box. If some code is not working, or you’re not sure if it is working, then extend the deadline, back some of the changes out, or pull the plug on this release and start over. Don’t risk a production failure in order to hit an arbitrary deadline. If the team is having problems fitting work into time boxes, then stop and figure out what you’re doing wrong – the team is trying to do too much too fast, or the code is too unstable, or people don’t understand the code enough – and fix it and move on.

Reviews and Retrospectives

Retrospectives are important in maintenance to keep the team moving forward, to find better ways of working, and to solve problems. But like many practices, regular reviews reach a point of diminishing returns over time – people end up going through the motions. Once the team is setup, reviews don’t need to be done in each iteration unless the team runs into problems.

Schedule reviews when you or the team need them. Collect data on how the team is working, on cycle time and bug report/fix ratios, correlate problems in production with changes, and get the team together to review if the numbers move off track. If the team runs into a serious problem like a major production failure, then get to the bottom of it through Root Cause Analysis.

Sustainable pace / 40-hour week

It’s not always possible to work a 40-hour week in maintenance. There are times when the team will be pushed to make urgent changes, spend late nights firefighting, releasing after hours and testing on weekends. But if this happens too often or goes on too long the team will burn out. It’s critical to establish a sustainable pace over the long term, to treat people fairly and give them a chance to do a good job.

Pairing

Pairing is hard to do in small teams where people are working on many different things. Pairing does make sense in some cases – people naturally pair-up when trying to debug a nasty problem or walking through a complicated change – but it’s not necessary to force it on people, and there are good reasons not to.

Some teams (like mine) rely more on code reviews instead of pairing, or try to get developers to pair when first looking at a problem or change, and at the end again to review the code and tests. The important thing is to ensure that changes get looked at by at least one other person if possible, however this gets done.

Collective Code Ownership

Because maintenance teams are usually small and have to deal with a lot of different kinds of work, sooner or later different people will end up working on different parts of the code. It’s necessary, and it’s a good thing because people get a chance to learn more about the system and work with different technologies and on different problems.

But there’s still a place for specialists in maintenance. You want the people who know the code the best to make emergency fixes or high-risk changes – or at least have them review the changes – because it has to work the first time. And sometimes you have no choice – sometimes there is only one person who understands a framework or language or technical problem well enough to get something done.

Coding Guidelines – follow the rules

Getting the team to follow coding guidelines is important in maintenance to help ensure the consistency and integrity of the code base over time – and to help ensure software security. Of course teams may have to compromise on coding standards and style conventions, depending on what they have inherited in the code base; and teams that maintain multiple systems will have to follow different guidelines for each system.

Metaphor

In XP, teams are supposed to share a Metaphor: a simple high-level expression of the system architecture (the system is a production line, or a bill of materials) and common names and patterns that can be used to describe the system. It’s a fuzzy concept at best, a weak substitute for more detailed architecture or design, and it’s not of much practical value in maintenance. Maintenance teams have to work with the architecture and patterns that are already in place in the system.

What is important is making sure that the team has a common understanding of these patterns and the basic architecture so that the integrity isn’t lost – if it hasn’t been lost already. Getting the team together and reviewing the architecture, or reverse-engineering it, making sure that they all agree on it and documenting it in a simple way is important especially when taking over maintenance of a new system and when you are planning major changes.

Simple Design

Agile development teams start with simple designs and try to keep them simple. Maintenance teams have to work with whatever design and architecture that they inherit, which can be overwhelmingly complex, especially in bigger and older systems. But the driving principle should still be to design changes and new features as simple as the existing system lets you – and to simplify the system’s design further whenever you can.

Especially when making small changes, simple, just-enough design is good – it means less documentation and less time and less cost. But maintenance teams need to be more risk adverse than development teams – even small mistakes can break compatibility or cause a run-time failure or open a security hole. This means that maintainers can’t be as iterative and free to take chances, and they need to spend more time upfront doing analysis, understanding the existing design and working through dependencies, as well as reviewing and testing their changes for regressions afterwards.

Refactoring

Refactoring takes on a lot of importance in maintenance. Every time a developer makes a change or fix they should consider how much refactoring work they should do and can do to make the code and design clearer and simpler, and to pay off technical debt. What and how much to refactor depends on what kind of work they are doing (making a well-thought-out isolated change, or doing shotgun surgery, or pushing out an emergency hot fix) and the time and risks involved, how well they understand the code, how good their tools are (development IDEs for Java and .NET at least have good built-in tools that make many refactorings simple and safe) and what kind of safety net they have in place to catch mistakes – automated tests, code reviews, static analysis.

Some maintenance teams don’t refactor because they are too afraid of making mistakes. It’s a vicious circle – over time the code will get harder and harder to understand and change, and they will have more reasons to be more afraid. Others claim that a maintenance team is not working correctly if they don’t spend at least 50% of their time refactoring.

The real answer is somewhere in between – enough refactoring to make changes and fixes safe. There are cases where extensive refactoring, restructuring or rewriting code is the right thing to do. Some code is too dangerous to change or too full of bugs to leave the way it is – studies show that in most systems, especially big systems, 80% of the bugs can cluster in 20% of the code. Restructuring or rewriting this code can pay off quickly, reducing problems in production, and significantly reducing the time needed to make changes and test them as you go forward.

Continuous Testing

Testing is even more important and necessary in maintenance than it is in development. And it’s a major part of maintenance costs. Most maintenance teams rely on developers to test their own changes and fixes by hand to make sure that the change worked and that they didn’t break anything as a side effect. Of course this makes testing expensive and inefficient and it limits how much work the team can do. In order to move fast, to make incremental changes and refactoring safe, the team needs a better safety net, by automating unit and functional tests and acceptance tests.

It can take a long time to put in test scaffolding and tools and write a good set of automated tests. But even a simple test framework and a small set of core fat tests can pay back quickly in maintenance, because a lot changes (and bugs) tend to be concentrated in the same parts of the code – the same features, framework code and APIs get changed over and over again, and will need to be tested over and over again. You can start small, get these tests running quickly and reliably and get the team to rely on them, fill in the gaps with manual tests and reviews, and then fill out the tests over time. Once you have a basic test framework in place, developers can take advantage of TFD/TDD especially for bug fixes – the fix has to be tested anyways, so why not write the test first and make sure that you fixed what you were supposed to?

Continuous Integration

To get Continuous Testing to work, you need a Continuous Integration environment. Understanding, automating and streamlining the build and getting the CI server up and running and wiring in tests and static analysis checks and reporting can take a lot of work in an enterprise system, especially if you have to deal with multiple languages and platforms and dependencies between systems. But doing this work is also the foundation for simplifying release and deployment – frequent short releases means that release and deployment has to be made as simple as possible.

Onsite Customer / Product Owner

Working closely with the customer to make sure that the team is delivering what the customer needs when the customer needs it is as important in maintenance as it is in developing a new system. Getting a talented and committed Customer engaged is hard enough on a high-profile development project – but it’s even harder in maintenance. You may end up with too many customers with conflicting agendas competing for the team’s attention, or nobody who has the time or ability to answer questions and make decisions. Maintenance teams often have to make compromises and help fill in this role on their own.

But it doesn’t all fit….

Kilner’s main point of concern isn’t really with Agile methods in maintenance. It’s with incremental design and development in general – that some work doesn’t fit nicely into short time boxes. Short iterations might work ok for bug fixes and small enhancements (they do), but sometimes you need to make bigger changes that have lots of dependencies. He argues that while Agile teams building new systems can stub out incomplete work and keep going in steps, maintenance teams have to get everything working all at once – it’s all or nothing.

It’s not easy to see how big changes can be broken down into small steps that can be fit into short time boxes. I agree that this is harder in maintenance because you have to be more careful in understanding and untangling dependencies before you make changes, and you have to be more careful not to break things. The code and design will sometimes fight the kinds of changes that you need to make, because you need to do something that was never anticipated in the original design, or whatever design there was has been lost over time and any kind of change is hard to make.

It’s not easy – but teams solve these problems all the time. You can use tools to figure out how much of a dependency mess you have in the code and what kind of changes you need to make to get out of this mess. If you are going to spend “weeks, months, or even years” to make changes to a system, then it makes sense to take time upfront to understand and break down build dependencies and isolate run-time dependencies, and put in test scaffolding and tests to protect the team from making mistakes as they go along. All of this can be done in time boxed steps. Just because you are following time boxes and simple, incremental design doesn’t mean that you start making changes without thinking them through.

Read Working With Legacy Code – Michael Feathers walks through how to deal with these problems in detail, in both object oriented and procedural languages. What to do if it takes forever to make a change. How to break dependencies. How to find interception points and pinch points. How to find structure in the design and the code. What tests to write and how to get automated tests to work.

Changing data in a production system, especially data shared with other systems, isn’t easy either. You need to plan out API changes and data structure changes as carefully as possible, but you can still make data and database changes in small, structured steps.

To make code changes in steps you can use Branching by Abstraction where it makes sense (like making back-end changes) and you can protect customers from changes through Feature Flags and Dark Launching like Facebook and Twitter and Flickr do to continuously roll out changes – although you need to be careful, because if taken too far these practices can make code more fragile and harder to work with.

Agile development teams follow incremental design and development to help them discover an optimal solution through trial-and-error. Maintenance teams work this way for a different reason – to manage technical risks by breaking big changes down and making small bets instead of big ones.

Working this way means that you have to put in scaffolding (and remember to take it out afterwards) and plan out intermediate steps and review and test everything as you make each change. Sometimes it might feel like you are running in place, that it is taking longer and costing more. But getting there in small steps is much safer, and gives you a lot more control.
Teams working on large legacy code bases and old technology platforms will have a harder time taking on these ideas and succeeding with them. But that doesn’t mean that they won’t work. Yes, you can be Agile in maintenance.

Dealing with security vulnerabilities ... er... bugs

A serious problem in many organizations is that development teams (and their business sponsors) don't take ownership for understanding and managing software security risks, and often try to ignore vulnerabilities or hide them. Without real pressure from the top, it's hard to convince developers and management that dealing with security vulnerabilities is a priority because vulnerabilities aren't requirements or real problems — they are potential problems and risks that can be put off until later.

This is wrong. Vulnerabilities found in pen testing and reviews and scans are either bugs — real problems in the code that should be fixed — or they are noise — false positives or motherhood that can be ignored. Treating them as something different and distinct and managing them in a different way is a mistake.

Read my latest post at the SANS AppSec Street Fighter blog on Dealing with Security Vulnerabilities.

Takeaways from OWASP AppSec USA 2011

Last week I attended the OWASP AppSec USA conference in Minneapolis. It was my first time at an OWASP event, and it was an impressive show. More than 600 attendees, some high-quality speakers, lots of vendor representation.

There were several different tracks over the two days of the conference: attacks and defenses, security issues in Mobile computing and Cloud computing, thought leadership, OWASP tools, security patterns and secure SDLCs. And there were lots of opportunities to talk to vendors and smart people. Here were the hilights for me:

Jim Manico showed that there’s hope for developers to write secure code, at least when it comes to protecting web apps from XSS attacks, if they use common good sense and new frameworks and libraries like Google’s Caja, the technology behind OWASP’s Java HTML Sanitizer.

There is even some hope for writing secure Mobile apps, as long as developers are careful and informed. This is a more reassuring story than what I was hearing from security experts earlier in the year, when everyone was focused on how pathetically insecure the platforms and tools were for writing Mobile apps.

It's also possible to be secure in the Cloud as long as you take the time to understand the Cloud services that you are using and how they are implemented and what assumptions they make, and if you think through your design. Which a lot of people probably aren’t bothering to do.

There's too much code already out there

We’re still trying to find ways to build software that is secure. But a bigger problem for many organizations is that there is already too much code out there that is insecure. Jeff Williams at Aspect Security guesstimates that there are more than 1 trillion lines of code in the world, and around 15 million developers around the world writing more of it. A lot of this legacy code contains serious security problems – but it is too expensive to find these and fix these problems. We need better tools to do this, tools that scale up to enterprises and down to small companies.

Many organizations, especially big enterprises with thousands of applications, are relying heavily on scanning tools to find security vulnerabilities. But it takes too long and costs too much to work through all of the results and to find real risks from the long lists of findings and false positives – to figure out what problems need to be fixed and how to fix them.

HP thinks that they have an answer by correlating dynamic scanning and static code analysis results. The idea is to scan the source code, then run dynamic scans against an application under a run-time monitor which captures problems found by the dynamic scanner and records where the problems occurred in the code. By matching up these results, you can see which static analysis findings are exploitable, and it makes it easier to find and fix problems found in run-time testing. And combining these techniques also makes it possible to find new types of vulnerabilities that can’t be found using just one or other of the approaches. I heard about this idea earlier in the year when Brian Chess presented it at SANS Appsec, but it is interesting to see how far it has come since then. Hybrid testing is limited to HP technology for now (SPI Dynamics and Fortify) and it’s limited by what problems the scanners can find, but it looks like a good step forward in the test tool space.

Good Pen Testing and Bad Pen Testing

It’s clear talking to vendors that there are good pen tests and bad pen tests. In most cases, good pen tests are what they offer, and bad pen tests are what you would get from a competitor. Comparing one vendor to another isn't easy, since an application pen test means different things to different people, and the vendors all had different stories and offerings. Some stressed the quality of their tools, others stressed the experience and ingenuity of their testers. Some offered automated scans with some interpretation, which could work fine for plain-vanilla web apps. Some do combined source code review and scanning. Some don’t want help understanding the code or scoping an engagement – they have a fixed-price menu running from running scans to manual pen tests using scanners and fuzzers and attack proxies. Others expect to spend a lot of time scoping and understanding what to test, to make sure that they focus on what's important.

SSL is broken

Moxie Marlinspike’s keynote on the problems with SSL and the CA model was entertaining but scary. He showed that the circle of trust model that underlies SSL – I trust the host because my browser trusts the CA and the CA trusts the host – is fundamentally broken now that CAs like Comodo and DigiNotar (and maybe others) have been compromised.

His solution is Convergence, an alternative way to establish trust between parties, where the client is responsible for choosing trust providers (notaries) rather than being locked into a single trust provider selected by the host.

Metrics and numbers for fun

I like numbers and statistics, here are a few that I found especially interesting. Some of them were thrown out quickly, I hope that I got them right.

At a panel on secure development, one expert estimated that the tax for introducing a secure SDLC, including training people, getting tools and getting people to use them, was about 15% of the total development costs. Although he qualified this number by saying that it was from a pilot, another panelist agreed that it was reasonable, so it seems a good rule of thumb to use for now.

John Steven at Cigital says they have found that an inexperienced team can expect to have 38-40 security vulnerabilities per KLOC. A mature team, following good security hygiene, can get this down to 7-10 vulnerabilities per KLOC.

Chris Wysopal at Veracode presented some interesting numbers in his presentation on Application Security Debt – you can read more on his ideas on applying the technical debt metaphor to application security and building economic models for the costs of this debt on his blog. He referenced findings from the 2010 Verizon Breach study and tried to correlate these findings with analysis work that Veracode has done using data from testing customer applications. This showed that the most common application-related breach was through Backdoors or hidden Command Channels which I was surprised at, and so was he, since Veracode’s own analysis shows that application Backdoors are not that common. I assumed from this analysis that application Backdoors were so dangerous because they were so easily found and exploited by attackers. However, when I read the Verizon report again, what Verizon calls a Backdoor is a command channel opened up by malware, which is something quite different to a Backdoor left in an application, so the comparison doesn’t stand up. This leaves SQL Injection to be the number one application vulnerability, something that shouldn't surprise anyone.

The “A Word”

Most of the speakers and panelists at the conference were from enterprise companies or from vendors trying to sell to the enterprise. Not surprisingly, as a result there was little said about the needs and constraints of small companies, and at the sessions I attended, the speakers had not come to terms with the reality that most companies are adopting or have already adopted more incremental agile development practices. At one panel, Agile development was referred to as the “A Word”.

We urgently need tools and practices that small, fast-moving teams can use, because these teams are building and maintaining a lot of the world's software. It's time for OWASP and ISC2 and the rest of the application security community to accept this and understand what it means and offer help where it's really needed.

Dumbest thing said at the conference

This award goes to one vendor on a panel on security testing who said that it is irresponsible for developers to work with new technologies where they don't understand the security risks, that developers who work on the bleeding edge are putting their customers and businesses at unnecessary risk. Besides being blindingly obvious, it shows a complete lack of connection to the realities of software development and the businesses that rely on software (which is pretty much every business today). Dinis Cruz pointed out the absurdity of this statement, by admitting that we don’t fully understand the security risks of the common platforms that we use today like Java and .NET. This means that nobody should be writing software at all - or maybe it would be ok if we all went back to writing batch programs in COBOL and FORTRAN.

Impressed and Humbled

Overall I learned some good stuff, and I enjoyed the conference and the opportunity to meet some of the people behind OWASP. I was impressed and humbled by the commitment and accomplishments of the many OWASP contributors. Seeing how much they have done, talking to them, understanding how much they care, gives me some hope that we can all find a way to build better software together.

Standups – take ‘em or leave ‘em

We left ‘em.

Standup meetings are a core practice in Agile methods like Scrum and XP. Each day the team meets briefly to answer 3 questions: What did I get done yesterday? What am I going to do today? What is getting in my way?

Standups offer a quick check on what’s happening, what’s changed, who ‘s working on what, who needs help. The meeting is supposed to be short and sweet, no more than 15 minutes a day. Martin Fowler lists some good reasons to hold standups:
- Share commitment
- Communicate status
- Identify obstacles to be solved
- Set direction and focus
- Help to build a team

However, not everyone finds that standups are necessary and some people have started to question the value of standups over time and are looking for substitutes. A number of people have suffered through poorly-run standup meetings. And some people just plain don’t like them.

Our team has been successful following Scrum and XP practices as we transitioned from delivering phased releases (a release every 2-3 months) to 2-3 small releases a month. We looked closely at how other teams worked, spent time learning about incremental and iterative development methods, and took what made sense to us and tried it and adapted it. But one of the Agile practices that we did not take up was standups.

When we introduced the idea of standups to the team a few years ago, we were surprised by the response. The team was roughly split. Some people, especially people who were new to the team, or people who had worked on successful Scrum teams or XP teams before, liked the idea. But other people who had been on the team from the beginning were opposed, some of them strongly opposed. And they had some good reasons:

We were already running an operational business and building and delivering new software at the same time. Team members were helping to support the system, helping with day-to-day operations, and working closely with the business side. Some people were working late and weekends to get changes in or to do after-hours testing, others were working early to help with startup issues or to coordinate with partners, some people were working in different timezones, some people were working at home, others were at the beck and call of integration partners and important clients. It would not be possible to schedule a daily meeting, even a short one, which would work well for everyone on a sustained basis. This is one example (and there many others), where Agile development ideas which work well in the controlled, artificial reality of a development project need to be rethought and adapted to fit day-to-day operational demands and constraints.

Some of the team had their hands full with important changes or problems that had to be taken care of urgently. They met regularly with other team members to work through design issues and reviewed each other’s code. They got help from other team members or managers when they needed it – there was no need to wait for a daily meeting to bring up blockers or get whatever information they needed.

And there were a couple of introverts who hated all meetings and didn’t understand why a meeting that they had to go to everyday and standup at would be any better than any other meeting. They saw it as a half hour wasted out of every day – because they knew that a 15-minute meeting, when you include the time taken to save what you were working on, go to the meeting, standup, get out of the meeting and get back to working productively, would mean at least 30 minutes if not more time every day from what they were doing.

Rather than try to fight with the team to implement a practice that we weren’t sure was really needed, we decided to find other ways to get the same results.

Alternatives to standups

There are other ways to share status within (and outside of) the team. Because we have people working together in different countries and different groups, we rely heavily on our bug tracking system to manage the backlog of work, to schedule and plan work, and to radiate status information. The bug tracking system is used by and shared with everyone: developers, testers, support, business operations, systems engineering, IT, compliance and management. Everything to do with the software and systems operations is tracked here, including new features and bug fixes and security vulnerabilities and problems found in testing, and operational and infrastructural changes, compliance reviews and new clients being onboarded. Using the system’s dashboards and notification feeds everyone can easily tell what is happening and what will be happening and when. The team doesn’t need standups to know what is happening, who is working on what, what the schedule is or what needs to be worked on next.

Managers who need to know what’s going on and who want to make sure that everyone is on track can get all of this through MBWA. And regular one-on-one meetings help reinforce priorities and make sure that everyone gets face time and a chance to ask questions without wasting the rest of the team’s time.

Standups help at the start

I can see that standups are more useful in the early stages of a project, when the team is starting to come together and everyone is working through the design and learning the domain and the technology and feeling out each other, finding out each other’s strengths and quirks.

Without standups, it is harder for new people joining the team to understand what is going on and what’s important, and figure out where they fit in. Pairing up new team members with someone more experienced helps, but if you have a couple or more people joining a team, it makes sense to hold standups for a while at least to help them get up to speed.

Declining value over time

But when people have been working together for a while, standups offer declining value. Once the team has gelled and people know each other and can depend on each other, they don’t need to meet in a room every morning to talk about what they are working on. Like some other practices that are important and useful in the beginning, the team grows up or grows out of them.

And as you move more into maintenance and support work, when people on the team are working on smaller individual changes and fixes and the work tends to be more interrupt-driven, standups are a waste. What one person is working on doesn’t have much if anything to do with what somebody else is doing. They don’t need to, or want to, listen to each other talking about what they did and what they’re going to do, because if it was important they would already know about it anyways.

Stuffed Pigs, Nerf Balls and other Silly Games

One of the things I don’t like about standups is that they are fundamentally paternalistic – you’re treating the team like children, forcing them to get together and stand up in a room every day (they have to stand up because they can’t be trusted to sit down) and making them speak in turn for only a few minutes (but only of course if they are a pig, not a chicken). And on some teams, people go so far as to hand around a stuffed pig or a Nerf ball or follow some other awkward ritual to make sure that people don’t speak out of turn. If this sounds silly and childish, it is. You have a room full of children: team members who can’t ask questions, they can’t solve problems, they have to stand there and follow silly rituals. Day after day, week after week, month after month, year after year…

Rather than treating adults like little children, why not let people meet when they need to or want to – to solve problems, to review requirements or designs, to respond to incidents (Root Cause Analysis), to plan, and to make sure that people know when requirements or priorities are changing in a significant way.

Would we have had different results if we had implemented standups? Probably. Would the results have been better for the team and for the company? I’m not sure. We found other good ways for the team, and the rest of the company, to track what was going on and what needed to be done next. We’ve had some misunderstandings, and some people did get off track – standups might have helped prevent this. And standups would have helped new people joining the team, to come up to speed. But the rest of the team gelled quickly during our crazy startup days, and built up a high level of trust and openness and shared commitment without standups. Can you build high-performing, collaborative teams without standups? Of course you can.

Standups have a place, especially early on with people who don’t know each other, where everyone is learning and uncertain about what to do and about each other. But don’t do something, or keep doing something if you don’t think that you need to, just because somebody read it in a book or learned about it in a 3 day SCM class. It’s your team, and you can find your own way to succeed.

Bugs and Numbers: How many bugs do you have in your code?

If you follow Zero Bug Tolerance of course you’re not supposed to have any bugs to fix after the code is done. But let’s get real. Is there any way to know how many bugs you're missing and will have to fix later, and how many bugs you might already have in your code? Are there any industry measures of code quality that you can use as a starting point?

For questions like these, the first place I look is one of the books by Capers Jones: arguably the leading expert in all things to do with software development metrics. There’s Applied Software Measurement, or Estimating Software Costs, or Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies. These books offer different views into a fascinating set of data that Capers Jones has collected over decades from thousands of different projects. It’s easy to spend hours getting lost in this data set, and reading through and questioning the findings that he draws from it.

So what can we learn from Capers Jones about bugs and defect potentials and defect density rates? A lot, actually.

On average 85% of bugs introduced in design and development are caught before the code is released (this is the average in the US as of 2009). His research shows that this defect removal rate has stayed roughly the same over 20 years, which is disappointing given the advances in tools and methods over that time.

We introduce 5 bugs per Function Point (Capers Jones is awfully fond of measuring everything by Function Points, which are an abstract way of measuring code size), depending on the type of system being built. Web systems are a bit lower surprisingly, at 4 bugs per Function Point; other internal business systems are 5, military systems average around 7. Using backfiring (a crude technique to convert function points back into LOC measures) you can equivalence 1 Function Point to about 50-55 lines of Java code.

For the sake of simplicity, let’s use 1 Function Point = 50 LOC, and keep in mind that all of these numbers are really rough, and that using backfiring techniques to translate Function Points to source code statements introduces a probability of error, but it’s a lot easier than trying to think in Function Points. And all I want here is a rough indicator of how much trouble a team might be in.

If 85% of bugs are hopefully found and fixed before the code is released, this leaves 0.75 bugs per Function Point unfound (and obviously unfixed) in the code when it gets to production. Which means that for a small application of 1,000 Function Points (50,000 or so lines of Java code), you could expect around 750 defects at release .

And this is only accounting for the bugs that you don’t already know about: a lot of code is released with a list of known bugs that the development team hasn’t had a chance to fix, or doesn’t think is worth fixing, or doesn’t know how to fix. And, this is just your code: it doesn’t account for bugs in the technology stack that the application depends on: the frameworks and application platform, database and messaging middleware, and any open source libraries or COTS that you take advantage of.

Of these 750+ bugs around 25% will be severity 1 show stoppers – real production problems that cause something significant to break.

Ouch – no wonder most teams will spend a lot of time on support and fixing bugs after releasing a big system. Of course, if you’re building and releasing software incrementally, you’ll find and fix more of these bugs as you go along, but you’ll still be fixing a lot of bugs in production.

Remember that these are rough averages. And remember (especially the other guys out there), we can’t all be above average, no matter how much we would like to be. For risk management purposes, it might be best to stick with averages, or even consider yourself below the bar.

Also keep in mind that defect potentials increase with the size of the system – big apps have more bugs on average. Not only is there a higher potential to write buggy code in bigger systems, but as the code base gets bigger and more complex it’s also harder to find and fix bugs. So big systems get released with even more bugs, and really big apps with a lot more bugs.

All of this gets worse in maintenance

In maintenance, the average defect potential for making changes is higher than in development, about 6 bugs per Function Point instead of 5. And the chance of finding and fixing mistakes in your changes is lower (83%). This is all because it’s harder to work with legacy code that you didn’t write and don’t understand all that well. So you should expect to release 1.08 bugs per Function Point when changing code in maintenance, instead of 0.75 bugs per Function Point.

And maintenance teams still have to deal with the latent bugs in the system, some of which may hide in the code for years, or forever. This includes heisenbugs and ghosts and weird timing issues and concurrency problems that disappear when you try to debug them. On average, 50% of residual latent defects are found each calendar year. The more people using your code, the faster that these bugs will be found.

Of course, once you find these bugs, you still have to fix them. The average maintenance programmer can be expected to fix around 10 bugs per month – and maybe implement some small enhancements too. That's not a great return on investment.

Then there’s the problem of bug re-injections, or regressions – when a programmer breaks something accidentally as a side-effect of making a fix. On average, programmers fixing a bug will introduce a new bug 7% of the time – and this can run as high as 20% for complex, poorly-structured code. Trying to fix these bad fixes is even worse – programmers trying to fix these mistakes have a 15% chance of still messing up the fix, and a 30% chance of introducing yet another bug as a side effect! It's better to roll-back the fix and start again.

Unfortunately, all of this gets worse over time. Unless you are doing a perfect job of refactoring and continuously simplifying the code, you can expect code complexity to increase an average of between 1% and 3% per year. And most systems get bigger over time, as you add more features and copy-and-paste code (of course you don't do that): the code base for a system under maintenance increases between 5-10% per year. As the code gets bigger and more complex, the chance for more bugs also increases each year.

But what if we’re not average? What if we’re best in class?

What if you are doing an almost perfect job, if you are truly best in class? Capers Jones finds that best in class teams create half as many bugs as average teams (2.5 or fewer defects per Function Point instead of 5), and they find and fix 95% or more of these bugs before the code is released. That sounds impressive - it means only 0.125 bugs per Function Point. But for a 50,000 LOC system, that’s still somewhere around 125 bugs on delivery.

And as for zero bugs? In his analysis of 13,000 projects over a period of more than 40 years, there were 2 projects with no defects reported within a year of release. So you can aspire to it. But don’t depend on it.

The C14N challenge

Failing to properly validate input data is behind at least half of all application security problems. In order to properly validate input data, you have to start by first ensuring that all data is in the same standard, simple, consistent format – a canonical form. This is because of all the wonderful flexibility in internationalization and data formatting and encoding that modern platforms and especially the Web offer. Wonderful capabilities that attackers can take advantage of to hide malicious code inside data in all sorts of sneaky ways.

Canonicalization is a conceptually simple idea: take data inputs, and convert all of it into a single, simple, consistent normalized internal format before you do anything else with it. But how exactly do you do this, and how do you know that it has been done properly? What are the steps that programmers need to take to properly canonicalize data? And how do you test for it? This is where things get fuzzy as hell.

To read my latest post on canonicalization problems (and the search for solutions), go to the SANS Application Security blog.