You can't hack and patch your way to a secure system.
You will never be able to find all of the security vulnerabilities and weaknesses in your code and network through scanning, or by paying outsiders to try to hack their way in.
The only way to be secure is to design and build security in from the beginning:
- threat modeling and risk assessment when designing apps and networks
- understanding and using the security features of your languages and frameworks, and filling in any gaps with secure libraries like Apache Shiro or KeyCzar…
- hardening the run-time using guidelines like the CIS benchmarks and tools like Chef and Puppet and UpGuard
- carefully reviewing every change that you make to code and configuration before putting them into production
- training everybody involved so that they know what to do, and what not to do
So what's the point of penetration testing? Why do organizations like Intuit and Microsoft have Red Teams attacking their production systems? And why are Facebook and Google and even the US Department of Defense running bug bounty programs, paying outsiders to hack into their system and report bugs?
Because once you've done everything you know how to do - or everything that you think you need to do - to secure your system, the only way to find out whether you've done a good enough job, is to attack your systems - before the bad guys do.
Attacking your system can show you where you are strong, and where you are weak: what you missed, where you made mistakes. It will uncover misunderstandings and hilight gaps in your design, in your defensive controls, and in your logging and monitoring. Watching your system under attack, watching what attackers do and how they do it, understanding what to look for and why, how to identify attacks and how to respond to them, will help change the way that you think and the way that you design and code and set up and run systems.
Let's look at different ways of attacking your system, and what you can learn from them:
Pen Testing
Pen testing - hiring an ethical hacker to scan and explore your application or network to find vulnerabilities and see what they can do with them - is usually done as part of due diligence, before a new system or a major change is rolled out, or once a year to satisfy some kind of regulatory obligation.
Pen testers will scan and test for common vulnerabilities and common mistakes in network and system configuration, missing patches, unsafe default settings. They'll find mistakes in authentication and user set up logic, session management, and access control schemes. They'll look at logs and error messages to find information leaks and bugs in error handling, and they will test for mistakes in some business logic (at least for well-understood workflows like online shopping or online banking), trying to work around approval steps or limit checks.
Pen tests should act as a reality check. If they found problems, a bad guy could too - or already has.
Pen testers won't usually have enough time, or understand your system well enough, to find subtle mistakes, even if they have access to documentation and source code. But anything that they do find in a few days or a few weeks of testing should be taken seriously. These are real, actionable insights into weaknesses in your system – and weaknesses in how you built it. Why didn't you find these problems yourself? How did they get there in the first place? What do you need to change in order to prevent problems like this from happening again?
Some organizations will try to narrow the scope of the pen tests as much as possible, in order to increase their chance of getting a "passing grade" and move on. But this defeats the real point of pen testing. You've gone to the trouble and expense of hiring somebody smart to check your system security. You should take advantage of what they know to find as many problems as possible – and learn as much as you can from them. A good pen tester will explain what they found, how they found it, why it is serious, and what you need to do to fix it.
But pen testing is expensive and doesn't scale. It takes time to find a good pen tester, time to set up and run the test, and time to review and understand and triage the results before you can work on addressing them. In an Agile or DevOps world. where changes are being rolled out every few days or maybe several times a day, a pen test once or twice a year won't cut it.
Red Teaming
If you can afford to have your own pen testing skills in house, you can take another step closer to what it’s like dealing with real world attacks, by running Red Team exercises. Organizations like Microsoft, Intuit and Salesforce have standing Red Teams who continuously attack their systems – live, in production.
Red Teaming is based on military Capture the Flag exercises. The Red Team - a small group of attackers - try to break into the system (without breaking the system), while a Blue Team (developers and operations) tries to catch them and stop them.
The Blue Team may know that an attack is scheduled and what systems will be targeted, but they won't know the details of the attack scenarios. While the Red Team’s success is measured by how many serious problems they find, and how fast they can exploit them, the Blue Team will be measured by MTTD and MTTR: how fast they detected and identified the attack, and how quickly they stopped it or contained and recovered from it.
Like pen testers, the Red Team's job is to find important vulnerabilities, prove that they can be exploited, and help the Blue Team to understand how they found the vulnerabilities, why they are important, and how to fix them properly.
The point of Red Teaming isn't just to find bugs - although you will find good bugs this way, bugs that definitely need to be fixed. The real value of Red Teaming is that you can observe how your system and your Ops team behaves and responds under attack. To learn what an attack looks like, to train your team how to recognize and respond to attacks, and, by exercising regularly, to get better at this.
Over time, as the Blue Team gains experience and improves, as they learn to respond to - and prevent - attacks, the Red Team will be forced to work harder, to look deeper for problems, to be more subtle and creative. As this competition escalates, as both teams push each other, your system - and your security capability - will benefit.
Intuit, for example, runs Red Team exercises the first day of every week (they call this “Red Team Mondays”). The Red Team identifies target systems and builds up their attack plans throughout the week, and publishes their targets internally each Friday. The Blue Teams for those systems will often work over the weekend to prepare, and to find and fix vulnerabilities on their own, to make the Red Team’s job harder. After the Red Team Monday exercises are over, the teams get together to debrief, review the results, and build action plans. And then it starts again.
Bug Bounties
Bug Bounty programs take one more step closer to real world attacks, by enlisting outsiders to hack into your system.
Outside researchers and white hat hackers might not have the insight and familiarity with the system that your own Red Team will. But Bug Bounties will give you access to a large community of people with unique skills, creativity, and time and energy that you can't afford on your own. This is why even organizations like Facebook and Google, who already hire the best engineers available and run strong internal security programs, have had so much success with their Bug Bounty programs.
Like Red Teaming, the rewards and recognition given to researchers drives competition. And like Red Teaming, you need to carefully establish - and enforce - ground rules of conduct. What systems and functions can be attacked, and what can't be. How far testers are allowed to go, where they need to stop, and what evidence they need to provide in order to win their bounties.
You can try to set up and run your own program, following guidelines like the ones that Google has published or you can use a platform like BugCrowd (https://bugcrowd.com/) or HackerOne (https://hackerone.com/) to manage outside testers.
Automated Attacks
But you don't have to wait until outsiders - or even your own Red Team - attack your system to find security problems. Why not attack the system yourself, every day, or every time that you make a change?
Tools like Gauntlt and BDD-Security can be used to run automated security tests and checks on online applications in Continuous Integration or Continuous Delivery, every time that code is checked in and every time that the system configuration is changed.
Gauntlt (http://gauntlt.org/) is an open source testing framework that makes it easy to write security tests in a high-level, English-like language. Because it uses Cucumber under the covers, you can express tests in Gherkin's familiar Given {precondition} When {execute test steps} Then {results should/not be} syntax.
Gauntlt comes with attack adaptors that wrap the details of using security pen testing tools, and sample attack files for checking your SSL configuration using sslyze, testing for SQL injection vulnerabilities using sqlmap or checking the network configuration using nmap, running simple web app attacks using curl, scanning for common vulnerabilities using arachni and dirb and garmr, and checking for serious vulnerabilities like Heartbleed.
BDD-Security (https://github.com/continuumsecurity/bdd-security) is another open source security testing framework, also based on Cucumber. It includes SSL checking (again using sslyze), scanning for run-time vulnerabilities using Nessus, and it integrates nicely with Selenium, so that you can add automated tests for authentication and access control, and run web app scans using OWASP ZAP as part of your automated functional testing.
All of these tests can be plugged in to your CI/CD pipelines so that they run automatically, every time that you make a change, as a security smoke test.
You can take a similar approach to attack your network.
Startups such as
- AttackIQ (https://www.attackiq.com/),
- SafeBreach (https://www.safebreach.com/),
- Verodin (https://verodin.com/) and
- vThreat (https://www.vthreat.com/)
You can automatically schedule and run pre-defined attacks and validation scenarios (or execute your own custom attacks) as often as you want, against all or parts of your network. These platforms scale easily, and provide you with an attacker's view into your systems and their weaknesses. You can see what attacks were tried, what worked, and why. You can use these tools for regular scanning and testing, to see if changes have left your systems vulnerable, to evaluate the effectiveness of a security defense tool, or, like Red Teaming, to exercise your incident response capabilities.
Running automated tests or attack simulations isn't the same as hiring a pen tester or running a Bug Bounty program or having a real Red Team. These tests have to be structured and limited in scope, so that they can be run often and provide consistent results.
But these tools can catch common and serious mistakes quickly - before anybody else does. They will give you confidence as you make changes. And they can be run continuously, so that you can maintain a secure baseline.
Why you need to Attack Yourself
There is a lot to be gained by attacking your systems. You'll find real and important bugs and mistakes - bugs that you know have to be fixed.
You can use the results to measure the effectiveness of your security programs, to see where you need to improve, and whether you are getting better.
And you will learn. You'll learn how to think like an attacker, and how your systems look from an attacker's perspective. You'll learn what to watch for, how to identify an attack, how to respond to attacks and how to contain them. You'll learn how long it takes to do this, and how to do it faster and easier.
You'll end up with a more secure system - and a stronger team.
