DZone's 2015 Guide to Application Security

DZone recently published a Guide to Application Security. It provides a good overview of effective appsec tools and practices, including my article 10 Steps to Secure Software, which looks at the latest release of OWASP's Proactive Controls project.

How to Prevent Catastrophic Failures in Complex Distributed Systems

In his now famous paper How Complex Systems Fail, Dr. Richard Cook explains how and why failures happen in complex systems:

Some Rules of failure in Complex Systems

4. Complex systems contain changing mixtures of failures latent within them. The complexity of these systems makes it impossible for them to run without multiple flaws being present. Because these are individually insufficient to cause failure they are regarded as minor factors during operations.

3. Catastrophe requires multiple failures - single point failures are not enough. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure.

14. Change introduces new forms of failure. The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes maybe actually create opportunities for new, low frequency but high consequence failures. Because these new, high consequence accidents occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.

The net of this: Complex systems are essentially and unavoidably fragile. We can try, but we can’t stop them from failing – there are too many moving pieces, too many variables and too many combinations to understand and to test. And even the smallest change or mistake can trigger a catastrophic failure.

A New Hope

But new research at the University of Toronto on catastrophic failures in complex distributed systems offers some hope – a potentially simple way to reduce the risk and impact of these failures.

The researchers looked at distributed online systems that had been extensively reviewed and tested, but still failed in spectacular ways.

They found that most catastrophic failures were initially triggered by minor, non-fatal errors: mistakes in configuration, small bugs, hardware failures that should have been tolerated. Then, following rule #3 above, a specific and unusual sequence of events had to occur for the catastrophe to unravel.

The bad news is that this sequence of events can’t be predicted – or tested for – in advance.

The good news is that catastrophic failures in complex, distributed systems may actually be easier to fix than anyone previously thought. Looking closer, the researchers found that almost all (92%) catastrophic failures are the result of incorrect handling on non-fatal errors. These mistakes in error handling caused the system to behave unpredictably, causing other errors, which weren’t always handled correctly or predictably, creating a domino effect.

More than half (58%) of catastrophic failures could be prevented by careful review and testing of error handling code. In 35% of the cases, the faults in error handling code were trivial: the error handler was empty or only logged a failure, or the logic was clearly incomplete. Easy mistakes to find and fix. So easy that the researchers built a freely available static analysis checker for Java byte code, Aspirator, to catch many of these problems.

In another 23% of the cases, the error handling logic of a non-fatal error was so wrong that basic statement coverage testing or careful code reviews would have caught the mistakes.

The next challenge that the researchers encountered was convincing developers to take these mistakes seriously. They had to walk developers through understanding why small bugs in error handling, bugs that “would never realistically happen” needed to be fixed – and why careful error handling is so important.

This is a challenge that we all need to take up – if we hope to prevent catastrophic failure in complex distributed systems.

10 things you can do to make your app secure: #3 Validate Input

This is part #3 of a series of posts on the OWASP Top 10 Proactive Development Controls.

Your first line of defence against attacks should always be to check all data from untrusted sources. Input validation is fundamental to application security, and a basic part of good defensive programming.

This is simple, and obvious – and often done wrong.

Don't Rely on Client-Side Validation

One common mistake is relying on client-side validation to catch problems. Client-side validation is useful in providing immediate feedback at the UI. But it won’t protect your system, especially in web apps. If an attacker can find a way to insert themselves in between the browser and your app, which can be done using a proxy, they can play with any of the data, including header fields and other hidden data, after local client-side editing has already been done. Data from a client, especially a client outside of your network, should never be trusted.

Another common mistake is relying on negative checking or “black list” validation to try to catch known bad data. This is a weak technique: you can’t be sure that you will catch all of the bad things or combinations, and negative checks can be subverted through (double) encoding and other evasion tricks.

White List Validation

OWASP recommends that you always use positive restrictions – white list validation – where you define the acceptable type and size, and all allowable values using regular expressions (regex). An example from the Proactive Controls: the regex for a password field could be:

^(?=.*[a-z])(?=.*[A-Z]) (?=.*\d) (?=.*[@#$%]).{10,64}$

This regular expression ensures that a password is 10 to 64 characters in length and includes a uppercase letter, a lowercase letter, a number and a special character (one or more uses of @, #, $, or %).

White list validation is trivial for enumerated data: days of the week, months of the year … anything that you can fit into a defined list. For other common data types (such as dates and time zones, numeric types, currencies, email addresses, IP addresses, URLs and domain names, and credit card numbers), you could use the Apache Commons Validator library.

Validation of free-format text fields (names, comments) is more difficult, and especially text fields that can – or need to – contain special characters, especially markup:

If your application handles markup -- untrusted input that is supposed to contain HTML -- it can be very difficult to validate. Encoding is also difficult, since it would break all the tags that are supposed to be in the input. Therefore, you need a library that can parse and clean HTML formatted text such as the OWASP Java HTML Sanitizer. A regular expression is not the right tool to parse and sanitize untrusted HTML.

You can test how solid your input validation is with different tools, such as fuzzers or static analysis taint checking (using tools that run through execution paths in the code and identify when you are referencing data that has not been validated) and through pen testing and manual exploratory testing.

Next: Access Control. How to do it, and how not to do it.