Pen Testing - from the other side of the fence

Mike Rothman’s recent post on pen testing was interesting to me, since I come from the other side of the fence: the side of the software developers who wrote the code and the testers who test it and the project managers who are responsible for taking care of risk and the business managers who have to decide how much to spend on things like pen testing.

I’ve learned that pen testing costs and takes time to do properly, especially at the start. The time to find a good pen testing team (and there are good pen testers out there, I’ve worked with some), to understand what they do and what they need from you to do a good job. The time to setup the tests, and to setup the test system and test accounts and test data, and to harden the test system to match production-quality so that you aren’t paying an expert to report basic patching and configuration problems that you can find yourself with a Nessus scan. Sure, they are going to scan anyways, but at least save yourself and them the trouble of going through those kinds of findings.

I’ve learned that it is important to work through the testing process together, to walk through your architecture and how the important features work, to ask and answer questions so that both sides clearly understand what is going on. To make sure that the test team has enough information to focus in on what’s important, and that they are not wasting their time and your money.

To make sure that the pen tester reviews any findings with you as they go along. That if they find something important, they tell you immediately so that you can act immediately, and so that they can help verify your fix. And if they get stuck or off track, that you help them with whatever information they need to keep going.

That you will learn what it really means to think like an attacker. You might think that you are thinking like an attacker, until you see an attacker at work. The way that they probe the system, the errors and information that they find useful, the soft spots that they look for – and what they can do if they find one.

And you will learn which problems are real, exploitable. One of the arguments that you will get from a developer is whether a fault or weakness can actually be exploited. It's a valid argument – risk assessment needs to take into account how easily any vulnerability can be exploited. Pen testing helps to answer this question: if they found it, the bad guys can and probably will too. And other vulnerabilities like it.

And to work through the findings together. To understand what vulnerabilities need to be addressed and how – what’s important, what’s not. And why. What to patch now, or soon, and how to do it properly.

And finally that it’s more important to look at pen testing for what it tells you about your software and your team, than as a compliance check-mark or a quality gate. To take it seriously and really learn from anything that’s found, rather than to just fix a few bugs and move on. That you need to stop and think and look more closely at your design, and at how you build software, and consider what’s important from a risk perspective to you and your customers. And understand what you need to do to do a better job. And then do it.

It would be foolish to expect too much from pen testing in your software security program. Just like it would be foolish to expect too much from static analysis or any other technique or technology. But from my experience at least, you will get out of it what you put into it.