I’ve spent a lot of time in my career working on problems that cross the lines between development and operations. That’s why I am interested in the emerging DevOps community: a bunch of smart people who are trying to bring development and operations closer together, and applying ideas and practices from Agile development to improving system and application operations. I try to keep up as much as I can with what DevOps is about, what people are doing and thinking, and understanding what it means to me. I went to Velocity last year, I’ve been reading the DevOps blogs, following the DevOps discussion group, I read and enjoyed the Web Operations book.
So, from a software development manager and CTO perspective, this is what I like (and don’t like) about DevOps so far.
What I like
The DevOps community is real: it's about people solving real problems, hard problems that a lot of us face in managing large systems. It's about people working together to come up with new ideas on how to do a better job. It's about taking ownership of problems and the solutions to these problems.
It is not an idea that has been crafted outside and that is being sold to people because it is good for them – not like Rugged Software for example. DevOps has authentic buy-in, it is being built from the ground up, and I can see things actually happening.
And there is no right or wrong, there’s no Manifesto (please please keep it that way), there’s no fundamentalism or religious wars – so far at least. DevOps is still in that early place and time where things are fluid and searching. People can disagree, and as long as they are smart and pragmatic and fair, they are listened to. Nobody is worried about contradicting one of the Creators (unlike in the Scrum community, or the the software Kanban community). Most of the people involved in DevOps today seem to be practical and open minded, and focused on finding answers. I hope it stays this way, forever and ever and ever.
DevOps has strong support from vendors. There’s configuration management and deployment automation technology like Puppet and Chef of course, and companies selling Continuous Delivery services and technology, the stuff needed for Infrastructure as Code, for effectively integrating application and infrastructure changes. And there is an important need in DevOps for highly scalable but lightweight data management technology, network management and monitoring platforms, tools to collect and analyze system and application metrics and logs. And then there's the Cloud.
Yes, there has to be a role for vendors: somebody has to act as a sponsor, to help sustain momentum, and people need technology to solve DevOps problems, to make deployment and management and operations of large-scale systems possible. But DevOps is not vendor-led, or consultant-led. Big Blue and HP haven’t figured out what DevOps means to them, or how to turn it to their advantage. Most of the technology is Open Source, making it easier for the community to get involved in solving their own problems.
The people in DevOps who are worth listening to are techies or hands-on managers who have real day-to-day responsibilities and solve real problems and who are sharing what has worked for them, and who are actively trying to find ways to do their jobs better and are interested in answers. Most of the thinking and writing in DevOps is being done by the doers, not by people trying to sell something, trying to convince you that you have a problem and that you need their help to solve it.
DevOps is about solving some interesting and challenging problems, in some cases extreme problems. I don’t have to manage the technology for a platform with thousands or tens of thousands of servers or millions of online customers, but I can learn from people who do. It’s fun and it’s worth paying attention to.
What I Don’t like
There’s a lot of enthusiasm in the DevOps community, people excited because they are onto something that is working, creating new tools, finding new solutions to problems. There are times when those of us who have been around for a few years recognize old ideas, even when they’ve been given cool new names. Listening to someone excitedly describing “dark launching” and “feature flippers” and pretending that this is a new idea is tiresome… But you can get over it, you can still enjoy the enthusiasm and wait to see if there are any new lessons to be learned.
Not surprisingly, there is much talk by DevOpsers about Values and Culture and Collaboration and Trust and other Capitalized Important People Stuff. This is all necessary when you are trying to get people to work together in new ways, but it has the potential to slide into pop psychology and New Age philosophizing like too much of what passes for thinking in the Agile development community today by consultants and "coaches". Hopefully, DevOpsers will stick to solving problems and getting things done in an open way, and not wandering down this path.
DevOps hasn’t come to terms with ITIL, and it needs to. ITIL is used as a straw man by some of the DevOps thinkers, in the way that Waterfall development is used by some Agile development enthusiasts: as an outmoded way of thinking and working, a collection of antipatterns. This naive approach is not fair or practical, it alienates a large community of people working in enterprise organizations and government IT organizations who have adopted more structured frameworks to get their work under control. There’s a lot to like in DevOps, but there’s a lot that DevOps can learn and has to learn from ITIL as well.
More fundamentally concerning to me is that DevOps has failed to take responsibility for security. They don’t seem to understand or care about what’s involved in deploying and operating secure systems, or if they do, they don’t explain how it fits. This will limit DevOps adoption, because most operations and development teams who want to take advantage of the ideas and practices and technologies coming out of DevOps, to become more efficient and more agile, also need to do this in a secure way. And this is reflected in the ongoing security problems that some of the organizations that are pioneering DevOps ideas continue to face. Would you trust your email to Facebook, would you really?
There’s a lot of interesting ideas coming out of large-scale online social media companies like Facebook and Twitter, and Flickr and Etsy, and some of the online multi-player Internet game platforms. But there’s a gap in trying to apply their lessons to the problems and requirements that the rest of us deal with: smaller-scale, but just as hard, just as real. Secure design and deployment and operations, all kinds of compliance and auditing, real data privacy, real transaction reliability, B2B integration, protecting compatibility with customers and legacy systems. Ideas taken to extremes like Continuous Deployment are exciting and challenging, they force you to think about how you can get better. But they aren’t ready for the enterprise yet.
DevOps doesn’t represent or meet the needs and priorities of most of us working on enterprise systems. But that doesn’t mean that they shouldn’t keep going. And that doesn’t mean that the rest of us shouldn’t keep following them – and trying to keep up.
I'm not sure exactly what you mean by the capitalized terms and pop psych references - but as someone working within an environment with serious issues surrounding trust and people desperate to protect their turf -- turf they ultimately resent even having in many cases, but can't give up..."because" -- I don't think the issues you seem to be referencing are an inconsequential part of this movement. If you can't get dev and ops to trust each other to want the same thing, you're not going to get very far with the rest of it.
ReplyDeleteI read the Enterprise post you linked, but as someone working in an environment not too far from the one described, I think he overstates the case. It's true that auditors are generally ignorant, but they can be educated, and it's well within SOX compliance to do continuous integration -- you just have to define your process accurately and follow it.
The problem is getting the people responsible for these pieces all on the same page and getting the process defined for the auditor. Again, this takes us back to people issues.
Locked down production environments aren't much of an issue either. It's not too hard to get to a place where integration happens to a sort of "Bastion" host, which is then pulled from by the production environment to do deployments. Depending on your "people" success, you may have an extra step or two in there, but it's still better than a fully manual process.
Anonymous
Because I also cannot speak about certain things - at least not in a way that can easily identify my company.
On the subject of ITIL, Jez Humble wrote a great article about Continuous Delivery within an ITIL environment. Unfortunately, the site is currently down - but the cached copy is here: http://webcache.googleusercontent.com/search?q=cache:http://continuousdelivery.com/2010/11/continuous-delivery-and-itil-change-management/ .
ReplyDeleteThe comments on that post make the argument that the obstructions to agile operations aren't inherent to ITIL itself, but come from the mindset which tends to be associated with it. My experience agrees with that, and that's why I see the people issues as the most important (and most difficult) place to effect change.
At the same time, I agree that ITIL shouldn't be framed as the "Waterfall of Operations", and that greater attention needs to be given to security, and the compliance requirements imposed upon many larger organisations. This will become easier as the community expands to include individuals placing greater professional emphasis on those activities.
I've got a lot of enthusiasm for the tools which are talked about in the DevOps community, but few (if any) are revolutionary - high-functioning operations groups have home-grown solutions to these problems for years. The novelty is making those solutions more accessible.
If improved tooling - and broad uptake of those tools - was the only thing to come out of the DevOps movement, it would still be a positive step forward for the industry. But it would be a missed opportunity.
Thankyou for writing such a thought-provoking post.
I'd pretty fundamentally disagree about several issues with your take on DevOps but particularly ITIL and security. I personally think you're far too focussed on continuous delivery and "Agile" as a part of DevOps. Areas I think are probably secondary features rather than primary drivers.
ReplyDelete* ITIL
There is no conflict between ITIL and DevOps. I've been doing ITIL stuff for more than 10 years now. Certified ITIL master, etc, etc. ITIL is hard to do right. DevOps makes it easier. All of the ITIL arenas but especially Change & Problem Management benefit from the cooperation that DevOps can foster between groups. The shared knowledge between groups, the heavy use of automation and the inherent need for concepts like CMDB all play well in the ITIL space.
DevOps to me is not a "take it or leave it" concept or one that has to surplant existing frameworks. It's one that can be used to supplement existing frameworks. You can take the pieces that work for your environment and not the ones that don't.
* Security
DevOps is PERFECT for security. Especially in an enterprise. Why perfect? Because in many enterprises security are the pariahs of the IT world (it's not that bad in some shops ... but in others...). It's often:
"Don't ask Security - they always say no"
"Oh shite - new Unix security standards ... don't they know the application won't run if x,y, or z."
"Security want us to patch the blah..."
Security people are IT people too. They are often Operations people. DevOps makes life easier for Security people. All of the things DevOps is about: cooperation, life cycle management, metrics, etc. All things Security people love.
Automation is another big ticket item. It's not possible to manage a modern enterprise environment without automation these days and running an environment includes securing it. SSOEs (Secure Standard Operating Environment), patching, compliance, auditing are all much easier to do in an automated, life cycle managed DevOps environment. Both in terms of enhanced cooperation between the operational teams to get security changes made and in enhanced operational efficiency.
If I ever go back to a Security Management role I'm going to be turning my SecOps team onto DevOps principals.
As the author of one of your links, I should probably qualify a few things that weren't originally clear. I don't think that DevOps and ITIL are mutually exclusive and I don't think that anything about DevOps inherently subverts any existing policy. The point of my original post was that the enthusiasm that so many of us have can cause a negative reaction. I've often told people that you can get to the point where you can do things like continuous deployment without actually "flipping the switch". I clarified some of this in a presentation I made to the local Atlanta devops user group:
ReplyDeletehttp://devops-culture-hurdles.heroku.com/
One thing that's not clear in the slides regarding "boogeymen" is that very little of the regulation from things like HIPPA and SOX impose specific technical requirements. Much of the policy is around auditability and accountability. The problem is that companies use a checklist approach to addressing those regulations because it's most cost-effective. If,for instance, the requirement is that all user access and actions are logged why is it not acceptable to simply eliminate that user access altogether and use an automated tool instead?
Auditor: "Show me who logged on to the server and what they did"
Me: "I can do you one better. No one logs onto the servers. Here's an exact list of every single configuration change applied to the server and when."
In fact, Tools like puppet, chef, mcollective, run-deck and the like actually encourage MORE security, auditability and accountability. By approaching your infrastructure as code, using configuration management tools and automation you can eliminate most if not all of the cases where, for instance, a person needs to physically log in to a server. You get disaster recovery built in because you've now codified in "code" how to define your infrastructure and you can "compile" that infrastructure into a finished product. You attack the root cause and not just bandaid it.
I think companies like WealthFront (originally Kaching) are a good example of what's possible in a regulated industry. It will be interesting to see how Facebook deals with the additional regulation should they ever go public.
Something about my comments is causing Blogger to freak out so I've posted my response here.
ReplyDeleteJim, sorry about all the emails I know you got.
Thank you everyone for your feedback.
ReplyDelete@Anonymous, I appreciate that the people stuff is important, and that anything that involves change and collaboration means getting people to work together and it isn't easy. But it shouldn't take the front seat. Agile development is an example where technical people are all hung up on organizational change, telling us how we all have to fundamentally change the way that we think and work and that we should force the customer and the rest of the organization to change because it is right for Scrum or whatever. Engineers like simple black and white answers and it doesn't work this way, and it doesn't have to. I am sad when I see people get caught up in this. I think that if people stay practical and open and focused on solving problems together, problems will get solved.
@Kartar, you raise some excellent points about how DevOps can actually make security work better. I hadn't thought of this, I appreciate your insight. My concern was that nothing that I have seen, read or listened to about DevOps addresses security issues headon, and some ideas coming out of DevOps, like Continuous Deployment, aren't compatible with secure operations. That doesn't mean that it can't work. But some companies, especially on the extreme edge, are putting agility ahead of security and other factors, and that doesn't work for the rest of us.
@Zac Stevens, thank your for the link to Jez Humble's article, I will spend some time reading through it. I would like to see these two worlds bridged, I think they have to be bridged.
@Lusis, thank you I read your posted comments (no idea why Blogger wouldn't accept it). You are right, like others I get caught up in the cool extreme face of DevOps like other people. There is a big difference between the kind of control that you can get from automated configuration management and deployment of infrastructure changes, which I agree obviously makes a lot of sense, and going from there to continuously deploying application changes 50 times a day or whatever at IMVU. Like I said I think we can learn from people on the extreme edge, but that doesn't mean that what they are doing makes sense for the rest of us.
Jim – we agree with many of the points you make. DevOps is an exciting new community that is making great strides in bringing development and operations teams together. Although Agile has many benefits, one pitfall is that it tends to increase the amount of deployments as a result of shorter iterations, pushing development and operations teams farther apart. The backlog that is created by the development team puts pressure on operations, and unfortunately this creates a large gap between the two teams. Luckily, deployment automation software is working to close that gap between dev and ops, resulting in a higher functioning organization. Have you heard much about deployment automation in your DevOps research?
ReplyDelete