Application Security at Scale

This week’s SANS AppSec conference in Las Vegas took on Application Security at Scale: how can we scale application security programs and technologies to big organizations, to small organizations and across organizations to millions of programmers world wide. You can find the presentation slides here. Lots of hilights for me: The conference was kicked off by Jeremiah Grossman from WhiteHat Security who made it clear that the problem of web application security alone is much bigger than we can take care of with the people and technology that we have today. We need to try different things like:

1. Game-ification: get developers interested and involved in Appsec using games and challenges like capture-the-flag, or the Elevation of Privilege card game (a game I have to try out)
2. Use peer pressure and score cards between teams, products, business units – drive better application security through competition (as we learned later, Cisco is one of the organizations score carding business units and products to drive improvement in software security programs)
3. Good, simple (and I will add inexpensive) online training to get as many developers as possible up to speed on secure design and coding
4. Write good, usable security frameworks and libraries and build security in by default into the major application frameworks – unfortunately we don’t know what frameworks will get widely adopted until they are widely adopted, so we will always be playing catch-up
5. Build security into the developer’s workflow – this is what SD Elements is doing
6. Use WAFs and virtual patching where it makes sense – to raise the bar on attacks by plugging simple issues found by scanners (WAFs used properly could block more than 2/3 of web application vulnerabilities, the kinds that scanners find); and to secure legacy code that nobody wants to try to figure out and fix by hand or in shops where it is too expensive and slow to get fixes out (in many Agile / Devops shops, it’s faster to fix and deploy the code than it is to put in a patch to the WAF).
7. Bug Bounty programs – if this works for Google and Facebook, it could work for you.

Chris Eng at Veracode presented some metrics collected from the scans that they have done for customers over the past 18 months. The interesting thing for me was the correlation between attack data (from the Verizon 2011 Data Breach Report)and vulnerability data, the intersections highlighting what we need to focus on. Just like last year (and the year before) SQL injection is the leading problem: 32% of apps scanned had SQL injection vulnerabilities, 20% of attacks are SQL injection.

The XSS Problem (and some Solutions)

According to Veracode’s data, 68% of web apps have XSS vulnerabilities. This is no surprise after you listen to Jim Manico explain in detail what programmers have to do to prevent XSS. Even getting every developer building web apps to understand all of the different rules for context-correct output encoding and escaping isn’t going to solve the problem: there are too many details for developers to take care of without missing something or making mistakes. “It’s more complex to stop XSS in large-scale apps than it is to do applied crypto and key management properly…. We have never seen a web app that can’t be attacked through XSS”. But there is hope – in a later presentation he explained how Context-Aware Auto-Escaping (aka Auto-Encoding) technology like JXT (a close-to-drop-in replacement for JSP, if you are writing well-formed JSP) and Ivan Ristic’s work on Apache Velocity Auto-Escaping can help protect at least some web apps from XSS. The most promising new technology for me was HTML5 iFrame Sandboxing which looks like it could actually be dropped in today to protect apps, at least if your customers are using modern browsers. I also learned about JavaScript object freezing and sealing to help protect rich client apps.

I was on a panel that looked at application security in small companies, together with Nick Galbreath at Etsy and Cameron Morris at Partnet – a small company that builds online web shopping portals for the US DoD. At Partnet everyone owns security: many of the developers have been trained on application security, all of them understand the OWASP Top 10, all code that is checked in is reviewed. I presented a case study on our AppSec program, what worked and what didn’t for us, from startup to now. Nick built on an earlier presentation by Zane Lackey at Etsy which explained some of the security controls in Etsy’s frameworks and Continuous Deployment pipeline, and the extensive monitoring and instrumentation feedback loops that they have from production back into development. This includes cool automated checks on changes to high-risk code (they have automated tests that hash specific pieces of code, if the hash value changes the build system automatically alerts the AppSec team and provides them the change set for code review).

Being able to deploy several times a day means that they can deploy new code (including security fixes) extremely quickly with high confidence. Nick also presented on rate limiting to monitor and control event activity in production. I am critical of Continuous Deployment – too many people who try to follow this model put speed ahead of reliability and security, and unnecessarily put their customers at risk. They don’t know when they have crossed the line from a web startup to running a real business. But if you are going to try this and want to do it right, learn everything that you can from Etsy.

Mobile AppSec - Android is a Train-Wreck

From the panel on mobile Appsec: Google has a good page on writing secure apps for Android (I am guessing it is this page) but it’s clear that most developers don’t know about it. The consensus was that Apple’s IOS is the most secure smart phone platform, and Android is a train wreck – according to one of the researchers (Georgia Weidman at Bulb Security), 1/2 of the default Android apps have serious security vulnerabilities.

Secure Frameworks and APIs

Chenxi Wang’s Day 2 keynote emphasised the importance of isolating security code and sharing and reusing security code through APIs. This was reinforced in a later panel by Jason Chan at Netflix and Adam Migus at E*Trade – like us, both these organizations rely on simple, extensible secure frameworks or APIs that developers can use to take care of problems like identity management, permissioning, crypto, secure transport, validation. At E*Trade, they find that most security vulnerabilities are because developers didn’t use this code (or didn’t use it properly). It doesn’t cost that much to write and support a secure framework – a few smart people can take care of this for the rest of the organization. And there are Open Source examples today like Apache Shiro that we can use to solve a lot of common security problems.

Pen Testing

An excellent panel on “inside the mind of a pen tester”, how expert pen testers think and work and approach problems, the tools that they use (I learned about chaining multiple attack proxies like Burp Suite and Zap together to take advantage of the different strengths of each tool). The most important thing in the pen test is for developers and management to get a clear understanding of the risks in the application: what kind of problems the testers found, how serious were they, what you have to do to fix them and what you have to do to prove that you fixed them. The real value of pen testing, like any other kind of testing, is the information that you get out of the test. If you’re not learning from pen tests, if the next time the tester comes back and tests the same system and finds the same problems, what are you paying for?

These pen testing experts had mixed opinions of WAFs – if a customer has a WAF it is usually installed out-of-the-box without tuning, and doesn’t present more than a speed bump to a determined attacker. But like anti-virus protection, it will stop most drive-by attacks – some big sites are seeing that as much as 10% or even 20% of their traffic is potentially dangerous, and a good WAF should be able to catch at least some of this.

Most memorable quotes from the conference

There is no such thing as an internal application. Jeremiah Grossman, WhiteHat Security

You can checkbox compliance but you can’t checkbox security. Monica Bush, University of Wisconsin-Madison

The closing message was sobering. We need more people who understand AppSec and who can write secure code – a lot more people. Big companies may only have a few AppSec generalists supporting thousands of developers, most companies don’t have anyone at all. This isn’t enough. Point-in-time assessments like pen tests aren’t enough either, because the attack space is always changing and the code is always changing – in a few weeks or months at most the results of a pen test may be invalidated. What we are doing today isn’t enough and it’s not going to scale. We need more security burned in and we need continuous security testing, which means more people who understand AppSec and better and more effective tools.