Appsec: The gaps between Builders and Defenders

This year's SANS Institute State of Application Security Survey, which I worked on with Eric Johnson and Frank Kim, looks at the gaps between Builders (the people who design and develop software) and Defenders (application security and information security professionals and operations).

We found that more developers - and managers - are coming to understand the risks and costs of insecure software, and are taking security more seriously. And defenders are doing a better job of understanding software development and how to work with developers. But there's still a long way to go.

Developers still need better skills in secure software development and a better understanding of application security risks. And time to learn and apply these skills. Defenders are trying to catch up with developers and Lean/Agile development, injecting security earlier into requirements and design, leveraging automated tools and services to accelerate security testing. But they are coming up against organizational and communications silos, and managers who put marketing priorities (features and time-to-market) ahead of everything else.

More than 1/3 of the organizations surveyed are looking at secure DevOps as a way to help bridge these gaps, break down the silos and bring development and security together. This is going to require some serious changes to how application security and development are done, but it offers a new hope for secure software.

You can read the detailed report of the survey results here.

DevOps is Killing Maintenance. Let’s Celebrate.

DevOps probably isn't killing developers.

But it is changing how people think about development - from running projects to a focus on building and running services. And more importantly, DevOps is killing maintenance, or sustaining engineering, or whatever managers want to call it. And that’s something that we should all celebrate.

High-bandwidth collaboration and rapid response to change in Agile put a bullet in the head of offshore development done by outsourced CMMI Level 5 certified development factories. DevOps, by extending collaboration between development teams and operations teams and by increasing the velocity of delivery to production (up to hundreds or even thousands of times per day), and by using real feedback from production to drive development priorities and design decisions, has pulled the plug on the sick idea that maintenance should be done by sustaining engineering teams, offshored together with the help desk to somewhere far away in order to get access to cheap talent.

Agile started the job. Devops can finish it

While large companies were busy finding offshore development and testing partners, Agile changed the rules of the game on them.

Off shoring coding and testing work made sense in large-scale waterfall projects with lots of upfront planning and detailed specs that could be handed off from analysts to farms of programmers and testers.

But the success of Agile adoption in so many organizations, including large enterprises, made outsourcing or offshoring development work less practical and less effective. Instead of detailed analysis and documented hand-offs, Agile teams rely on high-bandwidth face-to-face collaboration with each other and especially with the Customer, and rapid iteration and feedback. Everything happens faster. Customers change priorities and requirements. Developers respond and build and deliver features faster.

Time-intensive and people-intensive work like manual testing and reviews are replaced with automated testing and static analysis in Continuous Integration, pair programming, and continuous review and improvement.

In this dynamic world, it doesn’t make sense to try to shovel work offshore. You have to give up too much in return for saving on staff costs. Teleconferencing across time zones and cultures, “virtual team rooms” using webcams, remote pair programming over Skype… these are all poor compromises that lead to misunderstandings, inefficiencies and mistakes. Sure you can do offshore Agile development, but just because you can do something doesn’t mean that it is a good idea.

Devops is going to finish the job

In DevOps, with Continuous Delivery and Continuous Deployment, changes happen even faster. Cycle times and response times get shorter, from months or weeks to days or hours. And feedback cycles are extended from development into production, letting the entire IT organization experiment and learn and improve from real customer use.

Developers collaborate even more, not just with each other and with customers, but with operations too, in order to make sure that the system is setup correctly and running optimally. This can’t be done effectively by development and operations teams working in different time zones. And it doesn’t need to be.

We all know how outsourcing has played out. In the name of efficiency we sliced out non-strategic parts of core IT and farmed them out to other companies, whether offshore or domestic. CIOs loved it because of the budgetary benefits. Meanwhile, it sparked a thousand conversations about what outsourcing meant for IT, the US economy, individual careers, and the relationship between people and businesses.

But it turned out that we took outsourcing too far. It makes sense for some functions, but it can also mean losing control over management, quality, and security, among other things. Now we're seeing a lot of those big contracts being pulled back, and the word of the day is insourcing.

InformationWeek, DevOps: The New Outsourcing

Blurred Lines

DevOps intentionally blurs the lines between developers and operations, between coding and support. Engineering is engineering. Project work gets broken down into piece work: individual features or fixes or upgrades that can be completed quickly and pushed into production as soon as possible. Development work is prioritized together with operations and support tasks. What matters is whatever is important to the business, whatever is needed for the system to run. If the business needs something fixed now, your best people are fixing it, instead of giving it to some kids or shipping it overseas.

In DevOps, developers are accountable for making sure that their code works in production:

“You build it, you run it”

Which means making sure that the code gets into production, monitoring to make sure that it is working correctly, diagnosing and fixing any problems if something breaks.

New features, changes, fixes, upgrades, support work, deployment… everything is done by the same people, working together. Which means that maintenance and support gets the same management focus as new development. Which means that nobody is stuck in dead end job sustaining a dead end system. Which means that customers get better results, when they need them.

Except for enterprise legacy systems on life support, maintenance as most of us think of it today should die soon, thanks to DevOps. That alone makes DevOps worth adopting.