Application Security – Can you Rely on the Honeymoon Effect?

I learned about some interesting research from Dave Mortman at this year’s RSA conference in San Francisco which supports the Devops and Agile arguments that continuous, incremental, iterative changes can be made safely: a study by by the MIT Lincoln lab (Milk or Wine: Does Software Security Improve with Age?) and The Honeymoon Effect, by Sandy Clark at the University of Pennsylvania

These studies show that most software vulnerabilities are foundational (introduced from start of development up to first release), the result of early decisions and not the result of later incremental changes. And there is a “honeymoon period” after software is released, before bad guys understand it well enough to find and exploit vulnerabilities. Which means the more often that you release software changes, the safer your system could be.

Understanding the Honeymoon Effect

Research on the honeymoon period, the time “after the release a software product (or version) and before the discovery of the first vulnerability” seems to show that finding security vulnerabilities is “primarily a function of familiarity with the system”. Software security vulnerabilities aren't like functional or reliability bugs which are mostly found soon after release, slowing down over time:

“…we would expect attackers (and legitimate security researchers) who are looking for bugs to exploit to have the easiest time of it early in the life cycle. This, after all, is when the software is most intrinsically weak, with the highest density of ”low hanging fruit” bugs still unpatched and vulnerable to attack. As time goes on, after all, the number of undiscovered bugs will only go down, and those that remain will presumably require increasing effort to find and exploit.

But our analysis of the rate of the discovery of exploitable bugs in widely-used commercial and open-source software, tells a very different story than what the conventional software engineering wisdom leads us to expect. In fact, new software overwhelmingly enjoys a honeymoon from attack for a period after it is released. The time between release and the first 0-day vulnerability in a given software release tends to be markedly longer than the interval between the first and second vulnerability discovered, which in turn tends to be longer than the time between the second and the third…”

It may take a while for attackers to find the first vulnerability, but then it gets progressively easier – because attackers use information from previous vulnerabilities to find the next ones, and because the more vulnerabilities they find, the more confident they are in their ability to find even more (there's blood in the water for a reason).

This means that software may actually be safest when it should be the weakest:

“when the software is at its weakest, with the ‘easiest’ exploitable vulnerabilities still unpatched, there is a lower risk that this will be discovered by an actual attacker on a given day than there will be after the vulnerability is fixed!”

Code Reuse Shortens your Honeymoon

Clark’s team also found that re-use of code shortens the honeymoon, because this code may already be known to attackers:

“legacy code resulting from code-reuse [whether copy-and-paste or using frameworks or common libraries] is a major contributor to both the rate of vulnerability discovery and the numbers of vulnerabilities found…

We determined that the standard practice of reusing code offers unexpected security challenges. The very fact that this software is mature means that there has been ample opportunity to study it in sufficient detail to turn vulnerabilities into exploits.”

In fact, reuse of code can lead to “less than Zero day” vulnerabilities – software that is already known to be vulnerable before your software is released.

Leveraging Open Source or frameworks and libraries and copying-and-pasting from code that is already working obviously saves times and reduces development costs, and helps developers to minimize technical risks, including security risks – it should be safer to use a special-purpose security library or the security features of your application framework than it is to try to solve security problems on your own. But this also brings along its own set of risks, especially the dangers of using popular software components with known vulnerabilities – software that attackers know and can easily exploit on a wide scale. This means that if you’re going to use Open Source (and just about everybody does today), then you need to put in proactive controls to track what code is being used and make sure that you keep it up to date.

Make the Honeymoon Last as Long as you can

One risk of Agile development and Devops is that security can’t keep up with the rapid pace of change - at least not the way that most organizations practice security today. But if you’re moving fast enough, the bad guys might not be able to keep up either. So speed can actually become a security advantage:

“Software that was changed more frequently had a significantly longer median honeymoon before the first vulnerability was discovered.”

The idea of constant change as protection is behind Shape Shifter, an interesting new technology which constantly changes attributes of web application code so that attackers, especially bots, can’t get a handle on how the system works or execute simple automated attacks.

But speed of change isn't enough by itself to protect you, especially since a lot changes that developers make don’t materially affect the Attack Surface of the application – the points in the system that an attacker can use to get into (or get data out of) an application. Changes like introducing a new API or file upload, or a new user type, or modifying the steps in a key business workflow like an account transfer function could make the system easier or harder to attack. But most minor changes to the UI or behind the scenes changes to analytics and reporting and operations functions don't factor in.

The honeymoon can’t last forever any ways: it could be as long as 3 years, or as short as 1 day. If you are stupid or reckless or make poor technology choices or bad design decisions it won’t take the bad guys that long to find the first vulnerability, regardless of how often you fiddle with the code, and it will only get worse from there. You still have to do a responsible job in design and development and testing, and carefully manage code reuse, especially use of Open Source code – whatever you can to make the honeymoon last as long as possible.

Appsec and Devops at RSA 2014

At last week’s RSA security conference in San Francisco the talk was about how the NSA is compromising the security of the Internet as well as violating individual privacy, the NSA and RSA (did RSA take the money or not), the NSA and Snowden and how we all have to take insider threats much more seriously, Apple’s SSL flaw (honest mistake, or plausibly deniable back door?) and how big data is rapidly eroding privacy and anonymity on the Internet anyways (with today’s – or at least tomorrow’s – analytic tools and enough Cloud resources, everyone’s traces can be found and put back together). The Cloud is now considered safe – or at least as safe as anywhere else. Mobile is not safe. Bitcoin is not safe. Critical infrastructure is not safe. Point of Sale systems are definitely not safe – and neither are cars or airplanes or anything in the Internet of Things.

I spent most of my time on Appsec and Devops security issues. There were some interesting threads that ran through these sessions:

Third Party Controls

FS ISAC’s recent paper on Appropriate Software Security Control Types for Third Party Service and Product Providers got a lot of play. It outlines a set of controls that organizations (especially enterprises) should require of their third party service providers – and that anyone selling software or SaaS to large organizations should be prepared to meet:

1. vBSIMM - a subset of BSIMM to evaluate a provider’s secure development practices
2. Open Source Software controls – it’s not enough that an organization needs to be responsible for the security of the software that they write, they also need to be responsible for the security of any software that they use to build their systems, especially Open Source software. Sonatype has done an especially good job of highlighting the risk of using Open Source software, with scary data like “90% of a typical application is made up of Open Source components”, “2/3 of developers don’t know what components they use” and “more than 50% of the Global 500 use vulnerable Open Source components”.
3. Binary Static Analysis testing – it’s not enough that customers ask how a provider secures their software, they should also ask for evidence, by demanding independent static analysis testing of the software using Veracode or HP Fortify on Demand.

Threat Modeling

There was also a lot of talk about threat modeling in secure development: why we should do it, how we could do it, why we aren’t doing it.

In an analysis of results from the latest BSIMM study on software security programs, Gary Mcgraw at Cigital emphasized the importance of threat modeling / architectural risk analysis in secure development. However he also pointed out that it does not scale. While 56/67 of firms in the BSIMM study who have application security programs in place conduct threat modeling, they limit this to only security features.

Jim Routh, now CISO at Aetna (and who lead the FS ISAC working group above while he was at JPMC), has implemented successful secure software development programs in 4 different organizations, but admitted that he failed at injecting threat modeling into secure development in all of these cases, because of how hard it is to get designers to understand the necessary tradeoff decisions.

Adam Shostack at Microsoft outlined a pragmatic and understandable approach to threat modeling. Ask 4 simple questions:

1. What are you building?
2. What can go wrong?
3. What are you going to do about the things that can go wrong?
4. Did you do a good job of 1-3?

Asking developers to “think like an attacker” is like asking developers to “think like a professional chef” or “think like an astronaut”. They don’t know how. They still need tools to help them. These tools (like STRIDE, Mitre’s CAPEC, and attack trees) are described in detail in his new book Threat Modeling: Designing for Security which at over 600 pages is unfortunately so detailed that few developers will get around to actually reading it.

Devops, Unicorns and Rainbows

A panel on Devops/Security Myths Debunked looked at security and Devops, but didn’t offer anything new. The same people and same stories, the same Unicorns and rainbows: n thousand of deploys per day at Amazon, developer self-service static analysis at twitter, Netflix’s Chaos Monkey… If Devops is taking the IT operations world by storm, there should be more leaders by now, with new success stories outside of big Cloud providers or social media.

The Devops story when it comes to security is simple. More frequent, smaller changes are better than less frequent big changes because small changes are less likely to cause problems, are easier to understand and manage and test, easier to test, and easier to debug, fix or roll back if something goes wrong. To deploy more often you need standardized, automated deployment – which is not just faster but also simpler and safer and easier to audit, and which makes it faster and simpler and safer to push out patches when you need to. Developers pushing changes to production does not mean developers signing on in production and making changes to production systems (see automated deployment above).

Dave Mortman tried to make it clear that Devops isn’t about, or shouldn’t only be about, how many times a day you can deploy changes: it should really about getting people talking together and solving problems together. But most of the discussion came back to speed of development, reducing cycle time, cutting deployment times in half – speed is what is mostly driving Devops today. And as I have pointed out before, infosec (the tools, practices, ways of thinking) is not ready to keep up with the speed of Agile and Devops.

Mobile Insecurity

The state of mobile security has definitely not improved over the past couple of years. One embarrassing example: the RSA conference mobile app was hacked multiple times by security researchers in the days leading up to and during the conference.

Dan Cornell at Denim Group presented some research based on work that his firm has done on enterprise mobile application security assessments. He made it clear that mobile security testing has to cover much more than just the code on the phone itself. Only 25% of serious vulnerabilities were found in code on the mobile device. More than 70% were in server code, in the web services that the mobile app called (web services that “had never seen adversarial traffic”), and the remaining 5% were in third party services. So it’s not the boutique of kids who your marketing department hired to write the pretty mobile front end – it’s the enterprise developers writing web services that are your biggest security problem.

Denim Group found more than half of serious vulnerabilities using automated testing tools (58% vs 42% found in manual pen testing and reviews). Dynamic analysis (DAST) tools are still much less effective in testing mobile than for web apps, which means that you need to rely heavily on static analysis testing. Unfortunately, static analysis tools are extremely inefficient - for every serious vulnerability found, they also report hundreds of false positives and other unimportant noise. We need better and more accurate tools, especially for fuzzing web services.

Tech, Toys and Models

Speaking of tools... The technology expo was twice the size of previous years. Lots of enterprise network security and vulnerability management and incident containment (whatever that is) solutions, governance tools, and application firewalls and NG firewalls and advanced NNG firewalls, vulnerability scanners, DDOS protection services, endpoint solutions, a few consultancies and some training companies. A boxing ring, lots of games, coffee bars and cocktail bars, magicians, a few characters dressed up like Vikings, some slutty soccer players – the usual RSA expo experience, just more of it.

Except for log file analysis tools (Splunk and LogRhythm) there was no devops tech at the tradeshow. There were lots of booths showing off Appsec testing tools (including all of the major static analysis players) and a couple of interesting new things:

Shape Shifter from startup Shape Security, is a new kind of application firewall technology for web apps: “a polymorphic bot wall” that dynamically replaces attributes of HTML or Javascript with random strings to deter automated attacks. Definitely cool, it will interesting to see how effective it is.

Denim Group showed off a major upgrade to ThreadFix, a tool that maps software vulnerabilities found through different testing methods including automated dynamic analysis, static analysis and manual testing. This gives you the same kind of vulnerability management capabilities that you get with enterprise appsec testing suites: hybrid mapping of the attack surface with drill down to line of code, tracking metrics across applications and portfolios, integration with different bug tracking systems and the ability to create virtual patches for WAFs, but in a cross-vendor way. You can correlate findings from HP Fortify static analysis testing and dynamic testing results from IBM Appscan, or from a wide range of other tools and security testing platforms, including Open Source tools like Findbugs or Brakeman or ZAP and Arachni and Skipfish. And ThreadFix is Open Source itself (available for free or in a supported enterprise version).

The best free giveway at the conference was viaProtect an app from mobile security firm viaForensics that lets you check the security status of your iOS or Android device. Although it is just out of beta and still a bit wobbly, it provides a useful summary of the security status of your mobile devices, and if you choose to register, you can see track the status of multiple devices on a web portal.

Another year, another RSA

This was my third time at RSA. What has changed over that time? Compliance, not fear of being hacked, is still driving infosec: “attackers might show up; auditors definitely will”. But most organizations have accepted that they will be hacked soon, or have been hacked but they just haven’t found out yet, and are trying to better be prepared for when the shit hits the fan. More money is being poured into technology and training and consulting, but it’s not clear what’s making a real difference. It’s not money or management that’s holding many organizations back from improving their security programs – it’s a shortage of infosec skills.

With a few exceptions, the general quality of presentations wasn't as good as previous years – disappointing and surprising given how hard it is to get a paper accepted at RSA. There wasn't much that I had not already heard at other conferences like OWASP Appsec or couldn't catch in a free vendor-sponsored webinar. The expert talks that were worth going to were often over-subscribed. But at least the closing keynote was a good one: Stephen Colbert was funny, provocative and surprisingly well-informed about the NSA and related topics. And the the parties – if getting drunk with security wonks and auditors is your idea of crazy fun, nothing, but nothing, beats RSA.

Peer reviews for security are a waste of time?

At this year’s RSA conference, one of the panel’s questioned whether software security is a waste of time. A panellist, John Viega, said a few things that I agreed with, and a lot that I didn't. Especially that

“peer reviews for security are a waste of time.”

This statement is wrong on every level.

Everyone should know by now that code reviews find real bugs – even informal, lightweight code reviews.

“Reviews catch more than half of a product’s defects regardless of the domain, level of maturity of the organization, or lifecycle phase during which they were applied”. What We Have Learned About Fighting Defects

Software security vulnerabilities are like other software bugs – you find them through testing or through reviews. If peer code reviews are a good way to find bugs, why would they be a waste of time for finding security bugs?

There are only a few developers anywhere who write security plumbing: authentication and session management, access control, password management, crypto and secrets handling. Or other kinds of plumbing like the data access layer and data encoding and validators that also have security consequences. All of this is the kind of kind of stuff that should be handled in frameworks anyway – if you’re writing this kind of code, you better have a good reason for doing it and you better know what you are doing. It’s obviously tricky and high-risk high-wire work, so unless you’re a team of one, your code and decisions need to be carefully reviewed by somebody else who is at least as smart as you are. If you don’t have anyone on the team who can review your work, then what the hell are you doing trying to write it in the first place?

Everybody else has to be responsible for writing good, defensive application code. Their responsibilities are:

• Make sure their code works – that the logic is correct
• Use the framework properly
• Check input data and return values
• Handle errors and exceptions correctly
• Use safe routines/APIs/libraries
• Be careful with threading and locking and synchronization

A good developer can review this code for security and privacy requirements: making sure that you are masking or encrypting or – even better – not storing PII and secrets, auditing, properly following access control rules. And they can review the logic and workflow, look for race conditions, check data validation, make sure that error handling and exception handling is done right and that you are using frameworks and libraries carefully.

This is what code reviews are for. To look for and find coding problems. If you find these problems – and code reviews are one of the most effective ways of doing this – your code will be safer and more secure. So I don’t get it. Why are peer reviews for security a waste of time?

Appsec at RSA 2013

This was my second time at the RSA conference on IT security. Like last year, I focused on the appsec track, starting with a half-day mini-course on how to write secure applications for developers, presented by Jim Manico and Eoin Keary representing OWASP. It was a well-attended session. Solid, clear guidance from people who really do understand what it takes to write secure code. They explained why relying on pen testing is never going to be enough (your white hat pen tester gets 2 weeks a year to hack your app, the black hats get 52 weeks a year), and covered all of the main problems, including password management (secure storage and forgot password features), how to protect the app from click jacking, proper session management, access control design. They showed code samples (good and bad) and pointed developers to OWASP libraries and Cheat Sheets, as well as other free tools.

We have to solve XSS and SQL Injection

They spent a lot of time on XSS (the most common vulnerability in web apps) and SQL injection (the most dangerous). Keary recommended that a good first step for securing an app is to find and fix all of the SQL injection problems: SQL injection is easy to see and easy to fix (change the code to use prepared statements with bind variables), and getting this done will not only make your app more secure, it also proves your organization’s ability to find security problems and fix them successfully.

SQL injection and XSS kept coming up throughout the conference. In a later session, Nick Galbreath looked deeper into SQL injection attacks and what developers can do to detect and block them. By researching thousands of SQL injection attacks, he found that attackers use constructs in SQL that web application developers rarely use: unions, comments, string and character functions, hex number literals and so on. By looking for these constructs in SQL statements you can easily identify if the system is being attacked, and possibly block the attacks. This is the core idea behind database firewalls like Green SQL and DB Networks, both companies that exhibited their solutions at RSA.

On the last day of the conference, Romain Gaucher from Coverity Research asked “Why haven’t we stamped out SQL Injection and XSS yet?”. He found through a static analysis review of several code bases that while many developers are trying to stop SQL injection by using parameterized queries, it’s not possible to do this in all cases. About 15% of SQL code could not be parameterized properly – or at least it wasn't convenient for the developers to come up with a different approach. Gaucher also reinforced how much of a pain in the butt it is trying to protect an app from XSS:

“XSS is not a single vulnerability”. XSS is a group of vulnerabilities that mostly involve injection of tainted data into various HTML contexts”.

It’s the same problem that Jim Manico explained in the secure development class: in order to prevent XSS you have to understand the context and do context-sensitive encoding, and hope that you don’t make a mistake. To help make this problem manageable, in addition to libraries available from OWASP, Coverity has open sourced a library to protect Java apps from XSS and SQL injection.

The Good

While most of the keynotes offered a chance to catch up on email, the Crypto Panel was interesting. Chinese research into crypto is skyrocketing. Which could be a good thing. Or not. I was interested to hear Dan Boneh at Stanford talk more about the research that he has done into digital certificate handling and SSL outside of browsers. His team found that in almost all cases, people who try to do SSL certificate validation in their own apps do it wrong.

Katie Moussouris at Microsoft presented an update on ISO standards work for vulnerability handling. ISO 30111 lays out a structured process for investigating, triaging and resolving software security vulnerabilities. There were no surprises in the model - the only surprise is that the industry actually needs an ISO standard for the blindingly obvious, but it should set a good bar for people who don't know where to start.

Jeremiah Grossman explained that there are two sides to the web application security problem. One half is weaknesses in web sites like SQL injection and lousy password handling and mistakes in access control. The other half is attacks that exploit fundamental problems in browsers. Attacks that try to break out of the browser – which browser vendors put a lot of attention to containing through sandboxing and anti-phishing and anti-malware protection – and attacks that stay inside the browser but compromise data inside the browser like XSS and CSRF, which get no attention from browser vendors so it’s up to application developers to deal with.

Grossman also presented some statistics on the state of web application security, using data that White Hat Security is collecting from its customer base. Recognizing that their customers are representative of more mature organizations that already do regular security testing of their apps, the results are still encouraging. The average number of vulnerabilities per app is continuing to decline year on year. SQL injection is now the 14th most common vulnerability, found in only 7% of tested apps – although more than 50% of web apps are vulnerable to XSS, for the reasons discussed above.

Gary McGraw from Cigital agreed that as an industry, software is getting better. Defect density is going down (not as fast as it should be, but real progress is being made), but the software security problem isn't going away because we are writing a lot more code, and more code inevitably means more bugs. He reiterated that we need to stay focused on the fundamentals – we already know what to do, we just have to do it.

“The time has come to stop looking for new bugs to add to the list. Just fix the bugs”.

Another highlight was the panel on Rugged Devops, which continued a discussion that started at OWASP Appsec late last year, and covered pretty much the same ground: how important it is to get developers and operations working together to make software run in production safely, that we need more automation (testing, deployment, monitoring), and how devops provides an opportunity to improve system security in many ways and should be embraced, not resisted by the IT security community.

The ideas are based heavily on what Etsy and Netflix and Twitter have done to build security into their rapid development/deployment practices. I agreed with ½ of the panel (Nick Galbreath and David Mortman, who have real experience in software security in Devops shops) almost all of the time, and disagreed with the other ½ of the panel most of the rest of the time. There’s still too much hype over continuously deploying changes 10 or 100 or 1000 times a day, and over the Chaos Monkey. Etsy moved to Continuous Deployment multiple times per day because they couldn’t properly manage their release cycles – that doesn't mean that everyone has to do the same thing or should even try. And you probably do need something like Chaos Monkey if you’re going to trust your business to infrastructure as unreliable as AWS, but again, that’s not a choice that you have to make. There’s a lot more to devops, it’s unfortunate that these ideas get so much attention.

The Bad and the Ugly

There was only one low point for me – a panel with John Viega formerly of McAfee and Brad Arkin from Adobe called “Software Security: a Waste of Time”.

Viega started off playing the devil’s advocate, asserting that most people should do nothing for appsec, it’s better and cheaper to spend their time and money on writing software that works and deal with security issues later. Arkin disagreed, but unfortunately it wasn't clear from the panel what he felt an organization should do instead. Both panellists questioned the value of most of the tools and methods that appsec relies on. Neither believed that static analysis tools scale, or that manual security code audits are worth doing. Viega also felt that “peer reviews for security are a waste of time”. Arkin went on to say:

“I haven’t seen a Web Application Firewall that’s worth buying, and I've stopped looking”
“The best way to make somebody quit is to put them in a threat modelling exercise”
“You can never fuzz and fix every bug”

Arkin also argued against regulation, citing the failure of PCI to shore up security for the retail space – ignoring that the primary reason that many companies even attempt to secure their software is because PCI requires them to take some responsible steps. But Arkin at least does believe that secure development training is important and that every developer should receive some security training. Viega disagreed, and felt that training only matters for a small number of developers who really care.

This panel was like a Saturday Night Live skit that went off the rails. I couldn't tell when the panellists were being honest or when they were ironically playing for effect. This session lived up to its name, and really was a waste of time.

The Toys

This year’s trade show was even bigger than last year, with overflow space across the hall. There were no race cars or sumo wrestlers at the booths this year, and fewer strippers (ahem models) moonlighting (can you call it "moonlighting" if you're doing it during the day?) as booth bunnies although there was a guy dressed like Iron Man and way too many carnival games.

This year’s theme was something to do with Big Data in security so there were lots of expensive analytics tools for sale. For appsec, the most interesting thing that I saw was Cigital Secure Assist a plug-in for different IDEs that provides fast feedback on security problems in code (Java, .NET or PHP) every time you open or close a file. The Cigital staff were careful not to call this a static analysis tool (they’re not trying to compete with Fortify or Coverity or Klocwork), but what excited me was the quality of the feedback, the small client-side footprint, and that they intend to make it available for direct purchase over the web for developers at a very reasonable price point, which means that this could finally be a viable option for smaller development shops that want to take care of security issues in code.

All in all, a good conference and a rare opportunity to meet so many smart people focused on IT security. I still think for pure appsec that OWASP’s annual conference is better, but there's nothing quite like RSA.