Appsec: The gaps between Builders and Defenders

This year's SANS Institute State of Application Security Survey, which I worked on with Eric Johnson and Frank Kim, looks at the gaps between Builders (the people who design and develop software) and Defenders (application security and information security professionals and operations).

We found that more developers - and managers - are coming to understand the risks and costs of insecure software, and are taking security more seriously. And defenders are doing a better job of understanding software development and how to work with developers. But there's still a long way to go.

Developers still need better skills in secure software development and a better understanding of application security risks. And time to learn and apply these skills. Defenders are trying to catch up with developers and Lean/Agile development, injecting security earlier into requirements and design, leveraging automated tools and services to accelerate security testing. But they are coming up against organizational and communications silos, and managers who put marketing priorities (features and time-to-market) ahead of everything else.

More than 1/3 of the organizations surveyed are looking at secure DevOps as a way to help bridge these gaps, break down the silos and bring development and security together. This is going to require some serious changes to how application security and development are done, but it offers a new hope for secure software.

You can read the detailed report of the survey results here.

Security Testing: Less, but More Often, can make a Big Difference

Some interesting findings from the 2012 SANS Appsec Security Survey: almost 1/4 of companies are testing their software on an ongoing, near-continuous basis. How are they doing this, and what does this mean to how applications can be developed and should be tested? Check out my latest post on the SANS Application Street Fighter Blog - "Security Testing: Less, but More Often can make a Big Difference".

SANS Application Security Survey

Frank Kim and I helped out with an industry-wide survey on application security practices. The results of the survey and our analysis can be found in the SANS Analyst Program Reading Room here.

SANS Ask the Expert - the Cost of Remediation

An interesting interview with Dan Cornell of Denim Group on the work that they are doing to understand the cost of remediating security vulnerabilities, here on the SANS Application Street Fighter blog.

Ask the Expert - John Steven on Threat Modeling

John Steven closes out a series of interviews with appsec experts on the challenges in security threat modeling and how teams can succeed with threat modeling. You can read the interview with John here.

Ask the Expert - Rohit Sethi on Threat Modeling

Rohit Sethi talks about how to take a pragmatic and lightweight approach to application security threat modeling, in the most recent interview in the SANS AppSec "Ask the Expert" series. You can read the interview with Rohit here.

What Appsec can learn from Devops

Appsec and Devops are trying to solve many of the same kinds of problems, trying to get developers and operations working together to build safer and more secure and more reliable and more resilient systems. But the way that devops is doing this is very different.

Read my latest post on the SANS Appsec Street Fighter blog on What Appsec can learn from Devops.

Different ways of looking at security bugs

When a development team first starts to take application security seriously, they'll end up with a list (probably a long list) of security bugs. It's useful to look at security bugs in different ways. In my latest post at the SANS Appsec Street Fighter blog, I explore 4 different ways to look at security bugs.

Continuous Deployment and Security: Ask the Expert Nick Galbreath

Another excellent interview in the "Ask the Expert" series on AppSec, with Nick Galbreath at Etsy, who looks at how teams can follow Continuous Deployment to deliver software quickly, while still being resilient and secure. You can read the interview here on the SANS AppSec blog.

Application Security at Scale

This week’s SANS AppSec conference in Las Vegas took on Application Security at Scale: how can we scale application security programs and technologies to big organizations, to small organizations and across organizations to millions of programmers world wide. You can find the presentation slides here. Lots of hilights for me: The conference was kicked off by Jeremiah Grossman from WhiteHat Security who made it clear that the problem of web application security alone is much bigger than we can take care of with the people and technology that we have today. We need to try different things like:

1. Game-ification: get developers interested and involved in Appsec using games and challenges like capture-the-flag, or the Elevation of Privilege card game (a game I have to try out)
2. Use peer pressure and score cards between teams, products, business units – drive better application security through competition (as we learned later, Cisco is one of the organizations score carding business units and products to drive improvement in software security programs)
3. Good, simple (and I will add inexpensive) online training to get as many developers as possible up to speed on secure design and coding
4. Write good, usable security frameworks and libraries and build security in by default into the major application frameworks – unfortunately we don’t know what frameworks will get widely adopted until they are widely adopted, so we will always be playing catch-up
5. Build security into the developer’s workflow – this is what SD Elements is doing
6. Use WAFs and virtual patching where it makes sense – to raise the bar on attacks by plugging simple issues found by scanners (WAFs used properly could block more than 2/3 of web application vulnerabilities, the kinds that scanners find); and to secure legacy code that nobody wants to try to figure out and fix by hand or in shops where it is too expensive and slow to get fixes out (in many Agile / Devops shops, it’s faster to fix and deploy the code than it is to put in a patch to the WAF).
7. Bug Bounty programs – if this works for Google and Facebook, it could work for you.

Chris Eng at Veracode presented some metrics collected from the scans that they have done for customers over the past 18 months. The interesting thing for me was the correlation between attack data (from the Verizon 2011 Data Breach Report)and vulnerability data, the intersections highlighting what we need to focus on. Just like last year (and the year before) SQL injection is the leading problem: 32% of apps scanned had SQL injection vulnerabilities, 20% of attacks are SQL injection.

The XSS Problem (and some Solutions)

According to Veracode’s data, 68% of web apps have XSS vulnerabilities. This is no surprise after you listen to Jim Manico explain in detail what programmers have to do to prevent XSS. Even getting every developer building web apps to understand all of the different rules for context-correct output encoding and escaping isn’t going to solve the problem: there are too many details for developers to take care of without missing something or making mistakes. “It’s more complex to stop XSS in large-scale apps than it is to do applied crypto and key management properly…. We have never seen a web app that can’t be attacked through XSS”. But there is hope – in a later presentation he explained how Context-Aware Auto-Escaping (aka Auto-Encoding) technology like JXT (a close-to-drop-in replacement for JSP, if you are writing well-formed JSP) and Ivan Ristic’s work on Apache Velocity Auto-Escaping can help protect at least some web apps from XSS. The most promising new technology for me was HTML5 iFrame Sandboxing which looks like it could actually be dropped in today to protect apps, at least if your customers are using modern browsers. I also learned about JavaScript object freezing and sealing to help protect rich client apps.

I was on a panel that looked at application security in small companies, together with Nick Galbreath at Etsy and Cameron Morris at Partnet – a small company that builds online web shopping portals for the US DoD. At Partnet everyone owns security: many of the developers have been trained on application security, all of them understand the OWASP Top 10, all code that is checked in is reviewed. I presented a case study on our AppSec program, what worked and what didn’t for us, from startup to now. Nick built on an earlier presentation by Zane Lackey at Etsy which explained some of the security controls in Etsy’s frameworks and Continuous Deployment pipeline, and the extensive monitoring and instrumentation feedback loops that they have from production back into development. This includes cool automated checks on changes to high-risk code (they have automated tests that hash specific pieces of code, if the hash value changes the build system automatically alerts the AppSec team and provides them the change set for code review).

Being able to deploy several times a day means that they can deploy new code (including security fixes) extremely quickly with high confidence. Nick also presented on rate limiting to monitor and control event activity in production. I am critical of Continuous Deployment – too many people who try to follow this model put speed ahead of reliability and security, and unnecessarily put their customers at risk. They don’t know when they have crossed the line from a web startup to running a real business. But if you are going to try this and want to do it right, learn everything that you can from Etsy.

Mobile AppSec - Android is a Train-Wreck

From the panel on mobile Appsec: Google has a good page on writing secure apps for Android (I am guessing it is this page) but it’s clear that most developers don’t know about it. The consensus was that Apple’s IOS is the most secure smart phone platform, and Android is a train wreck – according to one of the researchers (Georgia Weidman at Bulb Security), 1/2 of the default Android apps have serious security vulnerabilities.

Secure Frameworks and APIs

Chenxi Wang’s Day 2 keynote emphasised the importance of isolating security code and sharing and reusing security code through APIs. This was reinforced in a later panel by Jason Chan at Netflix and Adam Migus at E*Trade – like us, both these organizations rely on simple, extensible secure frameworks or APIs that developers can use to take care of problems like identity management, permissioning, crypto, secure transport, validation. At E*Trade, they find that most security vulnerabilities are because developers didn’t use this code (or didn’t use it properly). It doesn’t cost that much to write and support a secure framework – a few smart people can take care of this for the rest of the organization. And there are Open Source examples today like Apache Shiro that we can use to solve a lot of common security problems.

Pen Testing

An excellent panel on “inside the mind of a pen tester”, how expert pen testers think and work and approach problems, the tools that they use (I learned about chaining multiple attack proxies like Burp Suite and Zap together to take advantage of the different strengths of each tool). The most important thing in the pen test is for developers and management to get a clear understanding of the risks in the application: what kind of problems the testers found, how serious were they, what you have to do to fix them and what you have to do to prove that you fixed them. The real value of pen testing, like any other kind of testing, is the information that you get out of the test. If you’re not learning from pen tests, if the next time the tester comes back and tests the same system and finds the same problems, what are you paying for?

These pen testing experts had mixed opinions of WAFs – if a customer has a WAF it is usually installed out-of-the-box without tuning, and doesn’t present more than a speed bump to a determined attacker. But like anti-virus protection, it will stop most drive-by attacks – some big sites are seeing that as much as 10% or even 20% of their traffic is potentially dangerous, and a good WAF should be able to catch at least some of this.

Most memorable quotes from the conference

There is no such thing as an internal application. Jeremiah Grossman, WhiteHat Security

You can checkbox compliance but you can’t checkbox security. Monica Bush, University of Wisconsin-Madison

The closing message was sobering. We need more people who understand AppSec and who can write secure code – a lot more people. Big companies may only have a few AppSec generalists supporting thousands of developers, most companies don’t have anyone at all. This isn’t enough. Point-in-time assessments like pen tests aren’t enough either, because the attack space is always changing and the code is always changing – in a few weeks or months at most the results of a pen test may be invalidated. What we are doing today isn’t enough and it’s not going to scale. We need more security burned in and we need continuous security testing, which means more people who understand AppSec and better and more effective tools.

Chenxi Wang on the AppSec problem and what we need to do to solve it

Our second interview in the "Ask the Expert" series on AppSec is with Dr. Chenxi Wang at Forrester Research, who looks at the same hard problems in secure software:

How big is the AppSec problem that we are all facing today? Why haven't we been able to solve the problem of writing secure software? Is the problem solvable? Is it really possible for developers to write secure software? If so, where should developers and businesses start? What are the first changes that they need to make?

You can - and should - read the answers to these questions here.

How bad is the problem of insecure software, and can it be fixed? Ask the Expert - Jeremiah Grossman

Frank Kim and I are working on a series of posts where we ask experts on security and software development hard questions about the essential problems of building secure software. The first of these posts is an interview with Jeremiah Grossman, CTO of WhiteHat Security.

Jeremiah takes on some of the biggest and hardest questions: How big is the AppSec problem? The software community is made up of a lot of smart people. Why haven't we been able to solve the problem of writing secure software? And Is the problem solvable?

You can read the answers here.

What's the point of application pen testing?

Penetration testing is one of the bulwarks of an application security program: get an expert tester to simulate an attack on your system, and see if they can hack their way in. But how effective is application penetration testing, and what should you expect from it?

Read my latest post at the SANS AppSec Street Fighter blog on What's the point of application pen testing?

AppSec at RSA 2012 Conference

I spent last week at the RSA conference in San Francisco - my first time at a global IT security event like this. A lot to learn and think about, and a lot of problems to be solved.

Read my latest post at the SANS AppSec Street Fighter blog on my experience at this year's RSA conference.

Agile development teams CAN build secure software

We know that many development teams, especially small teams following Agile development practices, do a poor job of developing secure software. But is it Agile development specifically that is the problem? Many application security experts, especially those working in or for enterprises, think it is. I think they are wrong.

Read my latest post at the SANS AppSec Street Fighter blog on how Agile development teams CAN build secure software.

Software Security Starts with Software Quality

In Software Security: Building Security In, Cigital's Gray McGraw breaks software security problems down into roughly equal halves. One half of security problems are security design flaws: missing authorization or doing encryption wrong — or not using encryption at all when you are supposed to, not handling passwords properly, not auditing the right data, relying on client-side instead of server-side data validation, not managing sessions safely, not taking care of SQL injection properly, and so on. These are problems that require training and experience to understand and solve properly.

The other half are security coding defects — basic mistakes in coding that attackers find ways to exploit. Focusing on preventing, finding and fixing these mistakes is a good place to start a software security program. It's something that developers and testers understand and something that they can take ownership of right away.

Read my latest post at the SANS Appsec Street Fighter blog on how basic software security practices can take you a long way towards building secure software.

Dealing with security vulnerabilities ... er... bugs

A serious problem in many organizations is that development teams (and their business sponsors) don't take ownership for understanding and managing software security risks, and often try to ignore vulnerabilities or hide them. Without real pressure from the top, it's hard to convince developers and management that dealing with security vulnerabilities is a priority because vulnerabilities aren't requirements or real problems — they are potential problems and risks that can be put off until later.

This is wrong. Vulnerabilities found in pen testing and reviews and scans are either bugs — real problems in the code that should be fixed — or they are noise — false positives or motherhood that can be ignored. Treating them as something different and distinct and managing them in a different way is a mistake.

Read my latest post at the SANS AppSec Street Fighter blog on Dealing with Security Vulnerabilities.

The C14N challenge

Failing to properly validate input data is behind at least half of all application security problems. In order to properly validate input data, you have to start by first ensuring that all data is in the same standard, simple, consistent format – a canonical form. This is because of all the wonderful flexibility in internationalization and data formatting and encoding that modern platforms and especially the Web offer. Wonderful capabilities that attackers can take advantage of to hide malicious code inside data in all sorts of sneaky ways.

Canonicalization is a conceptually simple idea: take data inputs, and convert all of it into a single, simple, consistent normalized internal format before you do anything else with it. But how exactly do you do this, and how do you know that it has been done properly? What are the steps that programmers need to take to properly canonicalize data? And how do you test for it? This is where things get fuzzy as hell.

To read my latest post on canonicalization problems (and the search for solutions), go to the SANS Application Security blog.

Checklists, software and software security

There are practical applications of checklists in many different fields. Aviation, project engineering, now even surgery. But what about software? Sure, checklists are sometimes used in code reviews, to good effect. But can we do more, can we get the same thing out of checklists that pilots do, or that surgeons do?

Frank Kim invited me to post on software security at the SANS Application Security Street Fighter's blog. My post on using checklists in software development and software security can be found here.

SANS Appsec 2011: No easy answers, at least not yet

Earlier this week I attended the SANS Appsec Summit in San Francisco. I’ve taken SANS training before, and so have other people at my firm, but this was my first time at a SANS conference. It was well-organized and the quality of the speakers was high, especially for a smaller event like this.

There was a good mix of perspectives on software security problems: real-life examples of secure SDLCs and managing app sec programs; vulnerabilities and tools and methods; the problems and challenges that companies are facing putting in web app security; the very scary state of mobile app security. And there were a couple of cool and fun (if not immediately practical) presentations from smart guys at Google: one on their ballsy bug bounty programs, and another by Billy Rios on how botnets work, and what kind of application security programs botnets have. (I'm not sure what I can do with what I learned on this one, but it was definitely cool).

Around 2/3 of the crowd were app sec or other IT security people, the rest were developers or managers who wanted to learn more about how to build secure applications, based on a show-of-hands poll that I took on the first day.

I presented on How to Build Bridges between Development and App Sec

Building bridges between Software Development and App Sec

I was part of a panel on the divide between software development and security, together with exploratory software testing guru James Bach and Marisa Fagan who was representing the Rugged Software movement. James was entertaining and provocative as expected. He made the case that we will never be able to build secure software if we want to innovate. Marisa presented an update on what’s been happening with Rugged since it was announced at this same conference a year ago. While I appreciate the good intentions, I still don’t understand what difference Rugged is making, or will make soon.

Hilights and key take-aways for me:

You can’t get secure if you can’t convince software developers to take the bad guys seriously, and to take problems found in test and review seriously. “Developer looking at bug: that’s not a realistic case, nobody would actually do that”. This goes back to the Attacker’s Advantage: all it takes is one mistake, one soft spot that can be exploited for the bad guy to get in. “A lot of threat modeling is all about convincing developers that threats are real”. Even many testers don’t have an adversarial mindset. Find real problems and show that they are real and exploitable, especially in training.

Developers hate silly checklists and policies. Don’t expect a secure development program heavy on checklists and policies to succeed.

Logging, logging, logging. Good logging is an important line of defence for online systems. Logging inbound and outbound information with enough context to figure out what the hell is going on; logging all exceptions; making it easy for ops to monitor logs and to see what is important.

Big apps keep getting bigger, with lots of frameworks, external components, open source stuff. It’s not just your code you have to worry about. Big apps keep getting bigger and less secure. Small mobile apps are even scarier: there is no such thing as an expert on mobile app security today, because there is no such thing as mobile app security.

Robert Fly at Salesforce.com had a more upbeat story. Salesforce has found ways to make its appexchange program secure for its customers and partners. By limiting what partners can do in the system and how they do it through the salesforce development and run-time platform; by building a set of app sec tools and resources for its development community; by opening office hours to help developers inside and outside of the company with security issues; by enforcing regular code scans and reviews of every partner application. They’ve put a lot of work into their program, it seems to be paying off.

Last year everyone’s major vulnerabilities were XSS and SQL injection. This year the major vulnerabilities are XSS and SQL Injection. And next year, the major vulnerabilities will be… We don’t seem to be learning or getting better. Even with auto-escaping libraries and many of the world’s smartest programmers working there, Google is still getting caught on XSS vulnerabilities by bug bounty hunters.

Anonymous has shown that application-level DDOS attacks are real, and real hard to defend against.

The closing panel was on the future of software security tools. Tools are getting better, but not better enough, soon enough. IBM is working on much faster, incremental static analysis tools to provide quicker feedback to programmers so that they will use them more often; and end-to-end analysis that will look at code in the context of the bigger system to find real problems. Brian Chess at HP is working on integrating the results of dynamic run-time testing tools with static source code analysis tools to provide a more complete and accurate picture of vulnerabilities in the code. Ivan Ristic and Jim Manico talked about still-emerging secure, open source frameworks and template libraries and application sandboxes – technology that isn’t ready today, but that in 2-5 years will make it easier for developers to start building software secure by default.

So, there were no easy answers, at least not yet, but there are a lot of smart people taking these problems seriously, let's hope that better answers will come.