Can DevOps(Sec) make Software more Secure?

There was a lot of talk at RSA this year about DevOps and security: DevOpsSec or DevSecOps or Rugged DevOps or whatever people want to call it. This included a full-day seminar on DevOps before the conference opened and several talks and workshops throughout the conference which tried to make the case that DevOps isn’t just about delivering software faster, but making software better and more secure; and that DevOps isn't just for the Cloud, but that it can work in the enterprise.

Rugged DevOps

The Rugged DevOps story is based on a few core ideas:

Delivering smaller changes, more often, reduces complexity. Smaller, less complex changes are easier to code and test and review, and easier to troubleshoot when something goes wrong. And this should result in safer and more secure code: less complex code has fewer bugs, and code that has fewer bugs also has fewer vulnerabilities.

If you’re going to deliver code more often, you need to automate and streamline the work of testing and deployment. A standardized, repeatable and automated build and deployment pipeline, with built-in testing and checks, enables you to push changes out much faster and with much more confidence, which is important when you are trying to patch a critical vulnerability.

And using an automated deployment pipeline for all changes – changes to application code and configuration and changes to infrastructure – provides better change control. You know what was changed, by who and when, on every system, and you can track all changes back to your version control system.

But this means that you need to re-tool and re-think how you do deployment and configuration management, which is why so many vendors – not just Opscode and Puppet Labs, but classic enterprise vendors like IBM – are so excited about DevOps.

The DevOps Security Testing Problem

And you also need to re-tool and re-think how you do testing, especially system testing and security testing.

In DevOps, with Continuous Delivery or especially Continuous Deployment to production, you don’t have a “hardening sprint” where you can schedule a pen test or in-depth scans or an audit or operational reviews before the code gets deployed. Instead, you have to do your security testing and checks in-phase, as changes are checked-in. Static analysis engines that support incremental checking can work here, but most other security scanning and testing tools that we rely on today won’t keep up.

Which means that you’ll need to write your own security tests. But this raises a serious question. Who’s going to write these tests?

Infosec? There’s already a global shortage of people who understand application security today. And most of these people – the ones who aren’t working at consultancies or for tool vendors – are busy doing risk assessments and running scans and shepherding the results through development to get vulnerabilities fixed, or maybe doing secure code reviews or helping with threat modeling in a small number of more advanced shops. They don’t have the time or often the skills to write automated security tests in Ruby or whatever automated testing framework that you select.

QA? In more and more shops today, especially where Agile or DevOps methods are followed, there isn't anybody in QA, because manual testers who walk through testing checklists can’t keep up, so developers are responsible for doing their own testing.

When it comes to security testing, this is a problem. Most developers still don’t have the application security knowledge to understand how to write secure code, which means that they also don’t understand enough about security to know what security tests need to be written. And writing an automated attack in Gauntlt (and from what I can tell, more people are talking about Gauntlt than writing tests with it) is a lot different than writing happy path automated unit tests in JUnit or UI-driven functional tests in Selenium or Watir.

So we shouldn’t expect too much from automated security testing in DevOps. There’s not enough time in a Continuous Delivery pipeline to do deep scanning or comprehensive fuzzing especially if you want to deploy each day or multiple times per day, and we won’t get real coverage from some automated security tests written in Gauntlt or Mittn.

But maybe that’s ok, because DevOps could force us to change the way that we think about and the way that we do application security, just as Agile development changed the way that most of us design and build applications.

DevOpsSec – a Forcing Factor for Change

Agile development pushed developers to work more closely with each other and with the Customer, to understand real requirements and priorities and to respond to changes in requirements and priorities. And it also pushed developers to take more responsibility for code quality and for making sure that their code actually did what it was supposed to, through practices like TDD and relentless automated testing.

DevOps is pushing developers again, this time to work more closely with operations and infosec, to understand what’s required to make their code safe and resilient and performant. And it is pushing developers to take responsibility for making their code run properly in production:

“You build it, you run it”
Werner Vogels, CTO Amazon

When it comes to security, DevOps can force a fundamental change in how application security is done today, from "check-then-fix" to something that will actually work: building security in from the beginning, where it makes the most difference. But a lot of things have to change for this to succeed:

Developers need better appsec skills, and they need to work more closely with ops and with infosec, so that they can understand security and operational risks and understand how to deal with them proactively. Thinking more about security and reliability in requirements and design, understanding the security capabilities of their languages and frameworks and using them properly, writing more careful code and reviewing code more carefully.

Managers and Product Owners need to give developers the time to learn and build these skills, and the time to think through design and to do proper code reviews.

Infosec needs to become more iterative and more agile, to move out front, so that they can understand changing risks and threats as developers adopt new platforms and new technologies (the Cloud, Mobile, IoT, …). So that they can help developers design and write tools and tests and templates instead of preparing checklists – to do what Intuit calls “Security as Code”.

DevOps isn’t making software more secure – not yet. But it could, if it changes the way that developers design and build software and the way that most of us think about security.