Continuous Deployment is no Holy Grail

In Prerequisites for Continuous Deployment Dan Ackerson asked “What are your major obstacles to deploying continuously to your live servers”?

This must have been a rhetorical question, since my response is “awaiting moderation”. Why ask a question if you don’t want answers?

There are a number of obstacles to deploying meaningful changes continuously to live production servers, besides having a working Continuous Integration environment and following disciplined development practices.

Making interface changes and certifying them with partners. Making database schema and data changes. Security checks. Destructive testing and stress testing for performance-sensitive online transactional systems. And so on. Most of the changes that people talk about releasing continuously are trivial: minor tweaks, cosmetic fiddling or small bug fixes. Anything bigger has to be done more carefully.

Schema changes can’t be made continuously. Bigger functional changes can’t and shouldn’t be made continuously, even with dark launching. Etsy for example (one of the companies used as a poster child for Continuous Deployment), doesn’t continuously deploy bigger public-facing features. They take their time and design them and prototype them and test them and review them and plan them out with operations and customer support and product management like any sane organization. See John Allspaw’s keynote at Surge last year.

Ackerson talks about “sufficient” automated test coverage. Automated testing isn’t enough for every (any?) system or every (any?) company, certainly not automated testing at 35% coverage which is much much lower than say Jez Humble recommends in Continuous Delivery.

You also have to account for code reviews and security checks, which could be done before check-in. This is how some companies are able to achieve Continuous Deployment – they move some of the necessary and responsible steps like code reviews up before check-in, so you know the code is already pretty good.

Too many descriptions of Continuous Deployment make it sound too simple and too easy. It’s not, and even organizations that have a lot of experience with it continue to have security and reliability problems: Facebook, Wordpress, …

Yes there’s a lot to learn from Continuous Deployment, about streamlining and simplifying release and deployment, and reducing risk by breaking work down into smaller and smaller pieces and tying all of this together with ops monitoring and metrics. But it’s not the “Holy Grail of Devops”, or at least it shouldn’t be. There's a lot more to DevOps than Continuous Deployment, which is a good thing.