Developing and Testing in the Cloud

There’s a lot of hype around “the Cloud” and what it can do.

One of the things that I am interested in is Cloud solutions that can help small software companies, and especially to kickstart software startups. Good tools that development teams can take advantage of to build and test their own stuff, without all of the hassle and expense of internal IT, buying and provisioning their own gear, setting up the tools and systems and networks and finding someone who understands all of it well enough to do it right and to keep it running. I want to find Cloud-based technology that can do for software teams what Salesforce.com does for customer-service businesses, so that software teams can get started off quickly, and doing things right from the start.

Software teams need a core set of shared tools and capabilities:

• Source code management / version control
• Bug tracking
• Collaboration and shared documentationBuild and Continuous Integration
• Source code scanning and static analysis
• Unit and functional testing
• System, load and stress testing
• Code deployment.

Can the Cloud provide all of this in an effective way?

Managing and Building Code in the Cloud

I keep running into people using GitHub to host their source code repositories. Not just people working on Open Source projects, but companies using GitHub to manage commercial projects in hosted private repositories, a service used by a number of startups. In GitHub, developers have a platform for managing code with the Git distributed version control system, and they get access to a good set of tools for a development team, especially distributed teams: admin functions to control permissioning, wikis, a bug tracking system and an online code review tool.

One alternative to GitHub is BitBucket which uses the Mercurial DVCS. Like GitHub, BitBucket can be used to manage Open Source and private projects, and it looks like it offers a similar set of management capabilities and tools. GitHub is free for Open Source projects and cheap for small teams while BitBucket’s pricing is based on the number of users, small teams (up to 5 users) are free for open or closed source.

Another On Demand SCM platform built on Mercurial is Fog Creek Software’s Kiln which integrates nicely with Fog Creek’s bug tracking and planning system FogBugz.

Atlassian, which bought BitBucket last year also offers Jira Studio a comprehensive hosted development platform centered on a Subversion code repository integrated with the rest of Atlaissian’s strong development toolset: Jira bug tracking, FishEye for code searching, Confluence wiki, Greenhopper for Agile team planning, Elastic Bamboo for build and Continuous Integration on Amazon EC2 and Crucible for online code review. That’s almost everything that a team needs (all that is missing is static analysis and a functional and load testing platform). Presumably the same integrated development tool support will extend to BitBucket over time, although there are already some overlaps between BitBucket and Jira Studio.

And there’s CloudBees Dev@cloud which offers a choice of secure Git, SVN or Maven repositories, and Continuous Integration through a hosted Jenkins server.

Spring Source’s Code2Cloud might be a good option especially for Java / Spring developers when it is ready – the details are fuzzy at the moment.

Google Code is cool, but for now is only available for Open Source projects.

For enterprises, IBM recently announced Smart Business Development and Test Cloud and IBM Smart Business Development and Test on the IBM Cloud (also known as SmartCloud Enterprise - I’m not sure what the difference is between them, I got too worn out reading through the marketing speak) , with all sorts of IBM technology and partner tools. This stuff isn’t for the faint of heart, or the small of wallet.

Continuous Integration in the Cloud

Continuous Integration is a problem that development teams need to solve before they get too far along, and setting up a CI server and keeping it running isn’t trivial. It’s another good fit for the elastic On Demand model of the Cloud, paying for more infrastructure only when you need it.

Besides Atlassian’s hosted end-to-end platform, and CloudBees’ Jenkins as a Service which can be used to build code hosted on a repository accessible through the Internet, there are a few options for CI in the Cloud (courtesy of Pascal Thivent on Stack Overflow):

For Open Source projects there is CodeBetter based on the TeamCity build server. For proprietary projects there is Mike CI which is hosted on Amazon EC2, and includes interfaces to Subversion, Git and Mercurial repositories; and CI Foundry. However, I don’t know how real these services are. Mike CI was down the last couple of times I went to check on it, and CI Foundry’s home page includes this not-very-convincing testimonial:

What people are saying...
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris quis ligula vel ligula varius cursus tristique eget sem. Mauris sagittis, ante id imperdiet auctor, sem mi condimentum neque, quis feugiat quam dui dictum risus.
- Chris Read
Continuous Deployment expert

Thivent suggests that you can roll-your- own CI solution on Amazon EC2 or maybe use something like “CI in a box”. Using an On Demand platform like EC2 makes sense for CI, rather than provisioning your own gear: it’s a bit more work for you to do the setup, but you can rely on Amazon’s infrastructure to reduce costs and simplify operations.

Bug Tracking in the Cloud

There are some decent and reasonably priced Cloud-based bug tracking systems suitable for small teams, including Fog Creek Software’s FogBugz and Atlassian’s Jira Hosted option both of which are also included in their hosted code management suites.

Testing in the Cloud

The Cloud has a lot of promise for app testing, especially for large-scale performance and load testing. You’re building a cool new Web platform that needs to support 10,000 or 100,000 concurrent sessions according to your business plan. Before your investors put more money in, they want to see just how real your software is, how far away you are from delivering something that could work. Of course you don’t have the money or time or IT skills to build a large-scale test center of your own at this point, and there’s a lot of waste in doing this – you need a lot of gear, but you won’t need to use all of it very often.

You can spin up a test farm on EC2 surprisingly quickly, even a big one – and as long as you don’t need to run the tests for a long time and you don’t ask for too high a service level, it won’t cost you much at all. You pay for what you need when you need it.

To run stress tests you need more than just the app deployed in a farm. You also need a scalable stress test harness to drive load scenarios and to measure the results. There are some viable options available in the Cloud today: SOASTA with its CloudTest solution for load testing and extreme stress testing, scaling from smallish sites to really big; HP Load Runner in the Cloud running on EC2; JMeter in the Cloud; LoadStorm and BrowserMob.

There are also some other interesting test capabilities, including technology from Sauce Labs which lets you run Selenium tests on Web apps in the Cloud or manually test your app against different Cloud-based instantly-available browsers, all with video records of failed tests and other problems found. Simple, useful and cool.

A different take on Cloud-based testing that could work for startups and smaller companies is crowd-sourced testing services like uTest, where you get people “in the Cloud” to test your app (web apps, desktop apps and mobile apps), including functional coverage and regression testing (you give them tests, they run them and maybe help improve them), exploratory testing, usability testing, load testing and test planning and management. Like other Cloud-based On Demand solutions, you pay for work when you need it. uTest has a community of something like 40,000 testers available, although it’s not clear how many of them are much good.

Mob4Hire is another crowd-sourced solution specifically for mobile apps – they offer functional testing and usability testing and other services like market research.

Static Analysis in the Cloud

Some of the leading static analysis suppliers have also gone to the Cloud. HP offers Fortify on Demand to scan code for security vulnerabilities, and IBM has Rational Appscan OnDemand which includes assistance from an experienced team to help you interpret the results and plan remediation. Both of these solutions are targeted more towards the enterprise than small teams.

Veracode, which runs binary code analysis for security vulnerabilities and now also offers web vulnerability scanning, is only available through the Cloud. They offer a free trial scan of small Java apps for XSS and SQL Injection flaws (two of the most common and fatal web application vulnerabilities).

For startups that care about building reliable and secure apps (and most of them should, especially Web 2.0 startups and mobile app developers) it probably makes more sense to look at simpler developer IDE-based tools like Findbugs and Google’s CodePro Analytix (both free) or Klocwork Solo for Java which is priced per user.

The Tradeoffs

Managing your code, building it and testing it in the Cloud has a number of clear advantages:

• Faster time to market – you can get access to the infrastructure and tools quickly, with minimal hassle and setup time.
• Savings on cost and Capex – especially for load testing – you don’t have to pay upfront, you only have to pay for what you use when you use it, so you can better manage your cash flow.
• Convenience – you don’t have to provision and operate the gear yourself and add to your operational headaches.
• Access to specialist skills – your Cloud provider will know more about these technologies and how to solve a specific problem much better than you can afford to, so you don’t need to contract or hire gurus to help solve problems or make sure everything is setup properly, or waste time figuring it out yourself.
• Service levels – they’ll have more people to help keep things running, to make sure code and data are backed up, systems are cleaned up and monitored and patched. They’re less likely to lose your code and data than you are.

When it comes to security, the common argument for working with a Cloud-based provider is that they will be more responsible and serious about security than most of their customers will be or can be. Because the of economies of scale, they can invest in doing things right, and because they have to be secure to compete. GitHub for example looks like they have a good security program in place and they are hosted by Rackspace so the data center, network and servers will be setup and taken care of much better than a small company, especially a startup, can afford to or would know how to do.

The concern with using a Cloud platform like this comes down to some fundamental points:

1. It’s a shared platform, used by a lot of companies. The more successful the platform is, the more customers and interesting and valuable data / IP that they manage, the more attractive a target they are to bad guys. A small company by itself is an easy target, but not especially interesting or easy to find. But a platform that holds data for a lot of small, innovative companies? Yummy….[pronounced in a deep, growly voice with a foreign accent]

2. You take on some important risks any time that you outsource something that is core and critical to your business. Your data / code / customer base is more important to you than it is to whoever you outsource this responsibility to. If something goes badly wrong, if you can’t get an important change or an important bug fix out to a customer because the Cloud service isn’t working, or if the integrity or confidentiality of your code is compromised, they will be sad, and they will lose a customer (you)... but you may go out of business.

You can ask for, and pay for, good SLAs - but Amazon’s major EC2 outage earlier this year showed that even the best providers can’t meet their SLA commitments. While Amazon did a lot of things right in handling and recovering from that outage, it affected a lot of customers for too long.

3. There’s more to a secure online platform than using SSL and network firewalls and running in a good data center - pretending otherwise is at best naïve. As a customer, you need to be confident in how the Cloud services provider designed and built the software architecture and platform to protect the confidentiality and integrity of your IP and data, in their multi-tenant partitioning scheme and software security controls, and in their SDLC. Security vulnerabilities in application code are a serious source of risk to businesses on the Web, and most companies still do a poor job of building apps in a secure way.

DropBox, a Cloud-based document storage facility is an example of a platform that seemed to be secure and confidential, until somebody started to look deeper into it, and they continue to run into problems with basic security issues Unfortunately, none of the hosted platforms provide any kind of statement on their secure SDLC that I could find, on what steps they take to ensure that the platform is designed and implemented safely.

Testing in the Cloud, especially load testing for Web apps, looks to be a no-brainer – the business case is too compelling to ignore. Many of the other tools have a lot of promise: something like Atlassian’s end-to-end hosted studio would save a lot of time and trouble for a small company, and the upfront cost savings are real.

It comes down to how much trust you can put into your Cloud provider and their SLAs for availability and support, and whether in the end you can afford to trust your core IP to somebody else. 37 Signals, which builds and operates its own Cloud solutions for project management and document sharing, doesn’t think it is a good idea:

We host all the source code for our applications internally for obvious security reasons. That’s not to say Github’s private repository hosting isn’t a good option, especially if you want a hassle-free setup. It’s just not for us.
JH 22 Aug 08

For some companies, especially startups, IP embodied in their code is critically important – it may be all that they have. Ironically, it’s these companies that are most likely to use a Cloud platform like GitHub or Jira Studio or BitBucket. Deciding whether to trust your future to the Cloud is not an easy decision to make. But as the technology continues to get better, it’s a question that is worth asking.

Please, no more Manifestos

For some reason, people involved in software development have a thing for Manifestos (always with a Capital M). It all started with the Agile Manifesto. Then came the Software Craftsmanship Manifesto, signed by serious programmers big and small, except ironically the original author of Software Craftsmanship.

Then there have been other, less practical and less successful manifestos, like the Rugged Software Manifesto which as I have explained before was proclaimed with the naïve understanding that reciting feel-good phrases will somehow make developers write better software.

“I am rugged…”

And the even-less-practical but much-more fun-to-read manifesto for truly hard-core programmers (there really is no problem that you can’t program your way out of) and of course the cynical Half-Arsed Agile Manifesto.

What brought this all up is that some people involved in DevOps still feel incomplete without a Manifesto of their very own.

This is a shame, because I like following what’s happening in DevOps and learning from the best (and worst) of it, and I want them to keep doing what they’re doing.

I don’t see the point in manifestos. They don’t move me or change the way that I think or work. I can get through each day without having to refer to a manifesto. I want tools and concrete ideas that I can use to get things done, to do a better job. Not motherhood or bullshit. Patterns and anti-patterns and recipes and best practices (and worst practices) – these are useful. But Manifestos? Useless, or at their worst, dangerous:

Workers of the world unite….

Look how that one worked out.

Manifestos s are a way to keep people from thinking and asking questions.

Oh no, people have more than one way of thinking about a problem, or what something means!

This is not a bad thing. This is a GOOD THING! This is how we move forward, this is how we get better. What we’re doing isn’t simple, or right, or wrong, so let’s stop pretending.

Or maybe Andrew Shafer is right

A manifesto, by definition, is essentially a marketing document.
“Re: Time for a DevOps Manifesto?”

Either way, we don’t need more manifestos. What we need is people to keep thinking and asking questions, and getting things done.

Moving forward from Failure

System failures at scale are inescapable, as I have talked about before in the context of designing systems for failure in high-scale computing and how to apply these ideas to enterprise systems. Failures are wasted if you don’t learn enough from them; if the way that you design and deliver application systems, and the way that you deploy and run these systems, don’t get better as a result. You have to find lessons in each failure and constantly move forward, and make the system, and the team, more resilient.

I’ve read and heard a lot of good stuff over the last year or so from the DevOps community on handling system failures: how to minimize the risk of failures through deployment planning, how complex systems fail, how to communicate failures effectively to stakeholders so that you don’t destroy trust, and how to understand failures through postmortem analysis, including Jacob Loomis’ essay “How to Make Failure Beautiful” in the Web Operations book and John Allspaw’s keynote on Advanced Postmortem Fu and Human Error 101 at this year’s Velocity conference.

Many of these ideas fit with my own experience, and fill in some important gaps – they extend beyond the concerns of operating high-volume Web sites to broader, general problems in system operations and systems engineering, and deserve a wider audience outside of the Web operations and Web startup worlds.

Postmortems and Root Cause Analysis

The keys to successful postmortems and Root Cause Analysis are deceptively simple, and therefore difficult to get right:

• get the right people together in a room.


• create the right environment for blameless problem solving: make it safe for people to be open and honest, don’t point fingers, focus on facts and solving problems and how to get better.


• postmortems are expensive and painful, and people want to get out of them as soon as possible. Don’t stop too soon, before the team has really understood the problems, and the solutions.


• don’t be satisfied with a single root cause – complex system failures usually have multiple causes if you dig deep enough.


• human error isn’t a root cause – it’s a symptom of something else that you’ve done wrong.

You can’t stop with Root Cause Analysis

Root Cause Analysis is important, but it is just the first step. Once you’ve reviewed a failure as a team and found your way to the root causes or at least identified some real problems, now you have to do something about it, beyond the immediate fixes and workarounds that the team made recovering from the incident.

There are straightforward things that you can do now – detective and corrective actions, quick fixes, low cost and low risk – so you do them. Fix a bug, plug a hole, add defensive coding, add some tests. Better logging and diagnostics and alerting, better error handling, better metrics – you can always do this stuff better – so that ops can see the problem coming or at least recognize it when it happens again. Better troubleshooting tools and training for ops.

Don’t stop with corrective actions either

But it’s still not enough to recover from a failure and patch the holes and tighten things up, or even build a better incident management capability. You have to make sure to find the lessons in each failure, really learn about why the failure happened, how it relates to other failures or problems that you have; and about how you dealt with the failure when it happened, and what you need to do to do a better job the next time. Then you have to act to reduce the risk and cost of failures, take steps to make sure that failures like this one, and even failures that aren’t like this one, don’t happen again. And that when the next failure does happen, that the team is more prepared for it, that you can recover faster and with less stress and impact to the business.

Some of these preventative actions are straightforward: training for developers and testers and ops, and better communications in and especially between teams, checklists, health checks and dependency checks, more disciplined change controls and deployment controls, and other safeties.

But most preventative changes, the ones that make a real difference, are deeper, more fundamental: fixing architecture, culture, organization, or management weaknesses. Fixing these kinds of problems takes longer, involves more people, and costs more. Making changes to the architecture – switching out a core technology or implementing a partitioning strategy to contain failures and to scale up – can take months to work out and implement. Organizational and management changes are hard because this directly impacts people’s jobs. Changing culture is even harder and takes longer, especially to make the changes stick.

You can’t improve what you can’t see – you need data

These problems are also hard to see and understand. It’s naïve to think that you can recognize or prove the need for fundamental changes to architecture or organization or management controls or culture from a postmortem meeting – it takes time and perspective and experience with more than one failure to see this kind of problem. And because the fixes are fundamental and expensive, they can be hard to justify, hard to get management and the business and your own people to buy-in.

You need data to help you understand what you’re doing wrong, what you need to change or what you need to stop doing – and what you’re doing right, what you need to keep doing, and to build a case for change. At last year’s Velocity conference, John Allspaw explained how to use metrics to find patterns and trends in failures, to understand what’s hurting you the most, or hurting you the most often, what’s working well and what’s not. To find out what changes are safe, and what changes aren’t; what failures are cheap and easy to recover from, and what failures can take the company down.

Metrics are important to help you see problems, and to help show if you are learning and getting better. Track the types and severity of failures, frequency of failures, your response to failures – time to detect and time to recover from failures; and the frequency and type and size of changes, and the correlation between changes and failures (by type and size of changes, by type and severity of failures). And make sure to track regressions – the number of times that you take a step backward.

Deciding what to fix, and what not to

Metrics can also help you to decide what you can fix and change, and what’s not worth fixing or changing – at least not for now. You need to recognize when a problem is a true outlier, an isolated case that is unlikely to happen again and that you can accept and move forward. It’s important that you can see the difference between a problem that is a one-of, and a first-of –an indicator of something deeply wrong, a fundamental weakness in the way that the system was built or the way that you work. You don’t want to over-react to outliers but you also don’t can’t treat every problem as unique and miss seeing the patterns and connections, the underlying root causes.

There is diminishing returns in preventing (or even reducing the probability) of some problems, trying to stop the unstoppable. Some problems are too rare, or too expensive or difficult to prevent – and trying to prevent these problems can introduce new complexities and new risks into the system. I agree with John Allspaw that there are situations when it makes more sense to focus on how to react and recover faster, on improving MTTR rather than MTBF, but you need to know when to make that case.

Moving forward from Failures is real work and needs to be managed

People don’t want to think about failures for long – they want to put the mistakes and confusion and badness behind them and move on. And the business and management want priorities back on day-to-day delivery as soon as possible. When failures happen, you need to act fast, get the most that you can out of the moment, make decisions and commitments and reinforce changes while the pain is still fresh.

But fixing real problems in the system or the way that people are organized or the way that they work or the way that they think, can’t be done right away. Building reliability and resilience and operability into how you work and how you think and how you build and test software and how you solve problems takes time and commitment and continuous reinforcement. Management, and the team, have to be made accountable for longer-term preventative actions, for making bigger changes and improvements.

This work needs to be recognized by the business and management and the team and built in to the backlog, and actively managed. You need to remind people of the risks when you see them missing steps or trying to cut corners, or falling back into old patterns of thinking and acting. You’ll need to use metrics and cost data to drive behaviour and to drive change, and to decide how much to push and how often: are you changing too much too often, running too loose; or is change costing you too much, are you overcompensating?

A Resilient system is a DevOps problem

Development can’t solve all of these problems on its own, and neither can Operations. It needs the commitment of development (and by development, I mean everyone responsible for designing, developing, testing, securing and releasing the software) and operations (everyone responsible for building and securing and running the infrastructure, deploying the code, and running and monitoring the system). This is a real DevOps problem: development and operations have to be aligned and work together, communicating and collaborating, sharing responsibilities and technology. This is one of the places where I see DevOps adding real value in any organization.

Estimation is not a Black Art

Construx Software is one of the few companies that take the problems of estimation in software development seriously. Earlier this year I attended Steve McConnell’s Software Estimation in Depth course based on his book Software Estimation: Demystifying the Black Art. Two days reviewing factors in estimation and the causes of errors in estimating, the differences between estimates and targets and commitments, how to present estimates to the customer and to management, the importance of feedback and data, estimating techniques that work and why they work, and when and how to apply these techniques to different estimating problems.

You can get a lot out of reading Steve’s book, but the course setting gives you a chance to work with other people and solve real problems in estimation and planning, dig deep into questions with the man himself, and it introduces new material and research not included in the book. The key takeways from this course for me were:

The Basics

Even simple techniques work much better than “expert judgment” (aka wild ass guesses). With training and using straightforward tools and simple mathematical models and following a bit of structure, you can get to 70-75% accuracy in estimating, which is good enough for most of us.

Never estimate on the spot – it is professionally irresponsible, and you’re setting yourself, your team, and your customer up to fail. Always give the person who is doing the estimate time to think. Everyone knows this, but it’s important to be reminded of it, especially under pressure.

People’s systemic error tendency is to underestimate. Many organizations underestimate by a factor of at least 2x. This also applies to comparing back to actual costs: a common mistake is to remember how much you estimated something would take, not how long it actually took.

Estimates are always better if you can use real data to calibrate them: to compare estimates against evidence of how your organization has performed in the past. I knew this. But what I didn’t know is that you don’t need a lot of data points: even a few weeks of data can be useful, especially if it contradicts with judgment and forces you to question your assumptions.

Use different techniques at different stages as you move along a project’s Cone of Uncertainty, depending on how much information you have available at the time, and how high the stakes are – what the cost of estimation errors could be to the business. If you need higher confidence or higher quality estimates, use multiple techniques and compare the results.

I like T-Shirt sizing to help in planning and making a business case. Developers come up with a rough order of magnitude estimate on cost or effort (small, medium, large, extra-large) while the Customer/Product Owner/Product Manager does the same for the expected business value of a feature or change request. Then you match them up: Extra-Large business value and Small development cost gets a big thumbs-up. Small business value and Extra-Large cost gets a big thumbs down.

Estimating by analogy – comparing the work that you need to estimate to something similar that you’ve done before – is a simple technique that we use a lot. It works well if you are adding “another screen”, writing “another report”, or building “another interface”. It’s a good fit for maintenance, if you’ve been working with the same code for a while and know most of the gotchas.

Intact teams (like mine) tend to view problems in a similar way – ideas and experience converge. This is good if you work on similar problems, in maintenance or a long-running project, because you can make decisions quickly and confidently. But this is bad if you are confronted with new problems – you need techniques like Wideband Delphi to challenge people’s thinking patterns and let new ideas in.

Agile Estimating and Story Points and Velocity

We spent more time than I expected exploring issues in estimating Agile development work. Incremental and iterative planning in Agile development poses problems for a lot of organizations. The business/customer needs to know when the system will be ready and how much it will cost so that they can make their own commitments, but the team ideally wants to work this out iteratively, as they go. This means instead that they have to define the scope and cost as much as they can upfront, and then work out the details in each sprint – more like staged or spiral development methods. Once they have the backlog sized, they can plan out sprints based on forecast velocity or historical velocity - if they can figure this out.

I’m still not convinced that Agile story point estimating is better than (or as good as) other techniques, or that programmers sitting around a table playing Planning Poker is really an effective alternative to thoughtful estimating. Story points create some problems in planning new project development, because they are purposefully abstract – too abstract to be useful in helping to make commitments to the business. You might have an idea of how many story points give or take you have to deliver, but what’s a story point in the real world? People who use story points can’t seem to agree on how to define a story point, what a story point means and how or when to use them in estimating.

More fundamentally, you can’t know what a story point costs until the team starts to deliver, by measuring the team’s velocity (the actual story points completed in an iteration).This leaves you with a bootstrapping problem: you can’t estimate how long it is going to take to do something until you do it. You can look at data from other projects (if you’ve tracked this data), but you’re not supposed to compare story points across projects because each project and each team is unique, and their idea of story points may be different. So you have to make an educated guess as to how long it is going to take you to deliver what’s in front of you, and now we’re back to the “expert judgement” problem.

The good news is that it won’t take long to calibrate Story Point estimates against evidence from the team’s actual velocity. Mike Cohn in Succeeding with Agile says that you need at least 5 iterations to establish confidence in a team’s velocity. But Construx has found that you can have a good understanding of a team’s velocity in as little as 2 iterations. That’s not a long time to wait for some kind of answer on how long it should take to get the work that you know that you have in front of you done.

There is more to estimating

There is a lot more to estimating, and to this estimation course: details on different techniques and rules of thumb, models and software tools, how to measure estimation error, how to turn estimates into schedules, how to handle schedule compression and how to stay out of the Impossible Zone. To get a taste of the course, check out this webinar on the 10 Deadly Sins of Software Estimation.

Software estimation, like other problems in managing software development, doesn’t require a lot of advanced technical knowledge. It comes down to relatively simple ideas and practices that need to be consistently applied; to fundamentals and discipline. That doesn’t mean that it is easy to do it right, but it’s not dark magic either.

Safer Software through Secure Frameworks

We have to make it easier for developers to build secure apps, especially Web apps. We can't keep forcing everybody who builds an application to understand and plug all of the stupid holes in how the Web works on their own — and to do this perfectly right every time. What we need is implementation-level security issues taken care of at the language and framework level. So that developers can focus on their real jobs: solving design problems and writing code that works.

Go to the SANS Application Security Street Fighter for my latest post on how to write safer software using secure frameworks, and application frameworks that are secure. And to read more about the OWASP Developer Outreach.

Not doing Code Reviews? What’s your excuse?

All of us have known for a long time that code reviews find defects, and that reviews are cheaper and can be more effective than most kinds of testing. In Code Complete, Steve McConnell builds an overwhelming case for code reviews: disciplined code inspections can find between 45%-70% of all defects in code, while even fast, informal reviews can find 20%-30%. Studies at IBM, HP, Microsoft and other places show that it is several times cheaper to find bugs in code reviews than through testing. And evidence keeps coming in to support that code reviews work.

“Reviews catch more than half of a product’s defects regardless of the domain, level of maturity of the organization, or lifecycle phase during which they were applied”. What We Have Learned About Fighting Defects

Recent research into code review practices and advances in tools make reviews more effective and less expensive, and can change the way that we think of code reviews and the way that we do them.

Lightweight code reviews work

A lot of the literature on code reviews, including Code Complete, focus on formal, team-based code inspections, using a model defined by Michael Fagan at IBM in the 1970s. Heavyweight and high-ceremony, Fagan inspections typically involve a minimum of 4 people: the programmer, the reader, a moderator, and one or more reviewers. Each review meeting requires extensive preparation, lots of documentation, and can last several hours.

More people now are having success with lightweight but still disciplined code reviews; a middle ground between informal “hey pal can you take a look at this” over-the-shoulder code walkthroughs, and expensive full-on Fagan inspections. In the Modern Code Review chapter of Making Software, Jason Cohen explains that lightweight reviews can be almost as effective, and much less expensive than formal inspections. The formal review meeting in a Fagan inspection, which is expensive to run and difficult to arrange, is mostly a waste of time – 95% of defects are found before the meeting. The real purpose of the meeting is to communicate the findings and to make sure that the programmer understands them. If that’s the case, the meetings can be held in much less formal way, or not at all.

Tool-assisted reviews

Reviewers today can rely on tools to help make their work more effective and efficient. Static Analysis bug checkers like Findbugs, tools from Coverity and Klocwork, and even checkers built-in to IDEs like IntelliJ Idea make the job of code reviewers easier and more valuable. These tools catch subtle bugs and common mistakes in coding that get past compilers and that can get past reviewers too, they highlight crappy or questionable code, and can enforce style conventions and best practices in coding. This means that reviewers can focus on more fundamental issues like correctness and error handling, and safety and security issues (and these tools, and others like Fortify 360 help find security vulnerabilities too).

Formal review meetings are expensive and difficult to setup. A lot of smaller teams do reviews offline instead through email, with the occasional one-on-one meeting when the programmer or reviewer feels that it is necessary. Bigger, distributed teams can use peer review collaboration tools like Crucible or Review Board or SmartBear’s CodeCollaborator or Google Code Reviews (aka Rietveld) to share code review findings. These tools integrate with your version control system (and sometimes your bug tracker) and are especially useful if you are involving multiple reviewers in different locations and different timezones. Reviewers and the programmer can add annotations or comments directly in with the changes, cutting back on the need for review meetings.

Some teams use these tools to ensure that code reviews are done, and prove that they are being done for compliance: they can enforce workflows (that reviews are done before code is committed to the main code line) and managers (and auditors) can see the results of code reviews. New programmers can look at the review history for code that they are working on to understand more about the code and about how to do an effective review, and programmers and managers can go back over the results of reviews and find ways to improve them. Some of these tools collect code review metrics, and others, like Klocwork Inspect, integrate reviewer findings and static analysis findings into the same review space.

Another code review tool, this time to help with security reviews, is Agnitio from David Rook, the Security Ninja. Agnitio serves a different purpose: rather than helping developers collaborate on a code review, it structures and guides a reviewer through a security review, following a detailed code and design review checklist, and then records the results of each review.

Reviews find more than functional defects

Reviews can find important and fundamental design and implementation problems. This includes subtle logic problems and operational safety weaknesses that testers may not find because they are hard to test for, and reviewers have the “white box advantage” that they can see problems or omissions in the code. And code reviews are important for finding security vulnerabilities – often the only way to find vulnerabilities, except through exhaustive and expensive pen testing. This is why code reviews are a fundamental part of secure SDLC’s like Microsoft’s SDL.

But there’s even more to reviews than finding bugs and security vulnerabilities. An interesting study by Mantyla and Lassenius in 2009 “What types of defects are really discovered in code reviews?” builds on earlier research by Siy and Votta at Bell Labs in the late 1990s, to show that the majority of problems found by reviewers are not functional mistakes, but what the reviewers call evolvability defects: issues that make code harder to understand and maintain, more fragile and more difficult to modify and fix.

On average, between 60% and 75% of the defects found in code reviews fall into this class. Of these, around 1/3 are simple code clarity issues: improving element naming and comments, making sure that the code makes sense. Most of the rest of the findings are organizational problems where the code is poorly structured, or duplicate or unused code; or approach problems where the reviewer can see a much simpler and cleaner implementation, sometimes replacing hand-rolled code with built in language features or library calls. And reviewers can also find changes that don’t belong or aren’t required, copy-and-paste mistakes and inconsistencies.

These defects or recommendations feed back into refactoring and are important for future maintenance of the software, reducing complexity and making it easier to change or fix the code in the future. But it’s more than this: many of these changes also reduce the technical risk of implementation, offering simpler and safer ways to solve the problem, and isolating changes or reducing the scope of a change, which in turn will reduce the number of defects that could be found in testing or escape into the wild.

So why aren’t more people reviewing code?

A study in 2005 found that less than half of development teams are following code reviews.

WTF?

Sure, reviews are hard. One of my colleagues who spends a lot of his time reviewing code finds it exhausting, much harder than writing code himself. To do a good review, you have to spend the time to understand the code and what the change was supposed to do, and then put yourself in to the mind of the programmer and work through the solution approach that they decided on. It takes time and it takes discipline. But the return on investment is clear.

I’ll give a pass to XP-style teams that already follow disciplined pair programming – although the focus of pairing is on finding a good solution, rather than looking for problems, a lot of mistakes and structural issues will still be caught in pair programming. Other problems can be caught by a good static analysis tool. And most of these teams are also writing disciplined unit tests, probably following TDD. They’re not writing sloppy code.

Solo programmers have an obvious problem: if you’re coding on your own, who’s going to review your code? You’re limited to self-reviews: setting the code aside for a while and then reviewing it on your own. This is more effective than you would expect. Jason Cohen says that programmers reviewing their own code can find as many as half of the issues that another reviewer would find, just by taking a second look. And solo programmers can also rely on static analysis tools to find some problems.

Reviews don’t need to be a big deal, you don’t need formal review meetings. And there are tools to help make reviews cheaper, easier and more effective. So, what about the rest of you? Why aren’t you doing code reviews? What’s your excuse?

Continuous Deployment is no Holy Grail

In Prerequisites for Continuous Deployment Dan Ackerson asked “What are your major obstacles to deploying continuously to your live servers”?

This must have been a rhetorical question, since my response is “awaiting moderation”. Why ask a question if you don’t want answers?

There are a number of obstacles to deploying meaningful changes continuously to live production servers, besides having a working Continuous Integration environment and following disciplined development practices.

Making interface changes and certifying them with partners. Making database schema and data changes. Security checks. Destructive testing and stress testing for performance-sensitive online transactional systems. And so on. Most of the changes that people talk about releasing continuously are trivial: minor tweaks, cosmetic fiddling or small bug fixes. Anything bigger has to be done more carefully.

Schema changes can’t be made continuously. Bigger functional changes can’t and shouldn’t be made continuously, even with dark launching. Etsy for example (one of the companies used as a poster child for Continuous Deployment), doesn’t continuously deploy bigger public-facing features. They take their time and design them and prototype them and test them and review them and plan them out with operations and customer support and product management like any sane organization. See John Allspaw’s keynote at Surge last year.

Ackerson talks about “sufficient” automated test coverage. Automated testing isn’t enough for every (any?) system or every (any?) company, certainly not automated testing at 35% coverage which is much much lower than say Jez Humble recommends in Continuous Delivery.

You also have to account for code reviews and security checks, which could be done before check-in. This is how some companies are able to achieve Continuous Deployment – they move some of the necessary and responsible steps like code reviews up before check-in, so you know the code is already pretty good.

Too many descriptions of Continuous Deployment make it sound too simple and too easy. It’s not, and even organizations that have a lot of experience with it continue to have security and reliability problems: Facebook, Wordpress, …

Yes there’s a lot to learn from Continuous Deployment, about streamlining and simplifying release and deployment, and reducing risk by breaking work down into smaller and smaller pieces and tying all of this together with ops monitoring and metrics. But it’s not the “Holy Grail of Devops”, or at least it shouldn’t be. There's a lot more to DevOps than Continuous Deployment, which is a good thing.