Maximizing value from penetration testing

Photo by Kevin Ku on Unsplash

​Regular penetration testing, or pen testing, is an essential part of understanding an organization’s security posture by mimicking a cyberattack using the same tools, techniques and procedures as an attacker.

Unfortunately, many pen tests fall short of their intended outcomes due to poor scoping, ill-defined goals and unrealistic testing — meaning testing that is too specific and technical or not technical enough.

Adding to the pain of poor outcomes is the cost. A typical pen test engagement can cost anywhere from $10,000 for a one-week engagement to test a basic web application to over $100,000 for a significant project, such as an extensive perimeter pen test for an enterprise. So how do you get the best value for the money you’re investing in pen testing?

Selecting the Right Vendor
Tip 1: Select your pen test vendor wisely.
Selecting a reasonably priced, skilled pen test vendor with a proven track record and high-quality, usable deliverables will help ensure a quality outcome and leave some budget for more in-depth testing or further tests.

There are three primary considerations when selecting a pen test vendor: price, skills/experience and the quality of their reports.

Price
This is the least important consideration. Most pen testing engagement rates are pretty similar, with a few outliers in the cheap but questionable category and the high-end, over-the-top, platinum version.

Also, look for vendors that bundle retesting of discovered vulnerabilities into the price. If that’s not included, it can be 10% to 20% of the cost of the initial test, depending on the vendor and the type of test.

Skills
Skill and experience are the next most important criteria. Look for firms with teams of testers that are CREST or OSCP certified. Also, ask for bios on the testers performing the test, and look for mentions of vulnerabilities discovered with CVE numbers or participation in bug bounty programs attributable to the pen test vendor or pen tester.

Reports
The quality of the report is the most important criterion for me when choosing a pen test vendor — provided they have adequately skilled testers. It’s the report that your organization will be left with when the testers have moved on to their next engagement.

Penetration testing is expensive, and the pre-canned “advice” delivered in a pen test report is often worthless and alarmist. I know; I’ve written my fair share of pen test reports in the past. Terms like “implement best practice” do nothing to drive the change needed to uplift an organization’s security posture.

Look for reports that deliver pragmatic remediation advice, including configuration and code snippets. Most importantly, review sample reports for alarmist findings such as cookie flags marked as “High Risk” — a pet hate of mine. Reports with alarmist findings do nothing to help you drive remediation in your organization.

Also, look for vendors that take reporting further by integrating with your ticketing system to raise tickets for issues they find or that provide videos of their hacks, which can show how simply an attacker can exploit technical security issues.

Types of Testing
Tip 2: Perform White Box Testing to save time and money.
In White Box Testing, you give the pen testers detailed information about the target environments, including domains/subdomains, hostnames, IP addresses, network diagrams, accounts of different privilege levels, Swagger/OpenAPI definitions and even access to source code. Significantly reducing or eliminating the Information Gathering, Reconnaissance and Discovery phases through a White Box Testing approach can provide significant time and cost reductions.
White Box Testing follows an “assume breach” mindset by giving pen testers access that allows them to execute test cases such as privilege escalation, lateral movement and the identification of sensitive systems or data.

Tip 3: Perform Black Box Testing to discover your actual perimeter.
In Black Box Testing, you provide minimal information to the pen testers about the target — for example, only a domain, an IP or hostname, or IP subnet, or as little as a company name. This type of testing is suitable for simulating a targeted attack such as an advanced persistent threat, or APT.
Black Box Testing is also a great way to discover shadow IT. Do you know all of your registered domains? How about your IP address space? Do you know which cloud platforms your organization is using? Are you aware of where all the systems your organization uses are?
You might feel confident in saying “Yes,” but you could be surprised at the results of a Black Box Test: turning up domains, cloud usage, SaaS and shadow IT you weren’t aware of. Remember, you can’t protect what you don’t know about.

Scoping for the Most Bang for Your Buck
Correctly scoping a pen test is the key to extracting maximum value from your pen testing investment.

Tip 4: Don’t use pen testers as expensive vulnerability scanners.
If your organization isn’t up to date with patches, why would you use penetration testers as expensive human vulnerability scanners? Why use penetration testers to tell you what your vulnerability management program can already tell you?

For example, if you give a decent pen tester access to most internal networks that aren’t updated with critical patches, they should have Windows Domain Admin in a day.

Infrastructure pen testing of internal networks that aren’t up to date with patches will yield thick pen test reports with just about guaranteed exploitation. You should ensure your penetration testing is targeted, as you’ll see in the following tips.

Tip 5: Select user-defined test cases to identify company-specific vulnerabilities.
Most pen test organizations will define standard test cases in their statement of work, typically the OWASP Top 10. While these are important test cases, as the security professional in your business, you will have concerns, misgivings or known risks for the tested systems. You can convert these to test cases and provide them to the pen test vendor when scoping the test.

In typical company-specific test cases, I’ll ask to include horizontal and vertical privilege escalation in applications. For financial applications, I’ll request a range of repudiation or fraud-based test cases, such as negative amounts in payments.

Also, don’t forget to include test cases derived from previously discovered vulnerabilities found through breaches, threat intelligence or pen tests.

Tip 6: Select objective-based testing to target a specific test case.
Objective-based testing is setting a clear objective — no surprise there — for the testers. The object is to execute against a particular test case or threat. Can the pen testers gain access to the CEO’s laptop? Can they gain access to SAP Payroll?

Objective-based testing helps validate or invalidate internal assumptions around control efficacy or risk likelihood for the chosen objective.

How do you choose the objective? Every security leader has a “what keeps me up at night” issue that gnaws at them, and objective-based testing can be used in the penetration test to run those scenarios.

Tip 7: Leverage your risk register to create test cases.
Penetration testing is ideal for driving change by exposing risks that you don’t know about and sometimes risks you do know about that the business hasn’t taken as seriously as it should. Pen testing can help validate the likelihood of those risks.
You can use the Risk Register to define test cases that:

• Bring a risk to realization;
• Prove or disprove assumptions around security controls;
• Prove or disprove exploitability — or even impact.

Trawl the risk register — there will be risks for which the impact is high but the likelihood is low. These make good candidates for company-defined test cases. Or better still, you can choose a risk in the risk register that has high impact that has been accepted.

Bringing these risks to realization can help drive change and get executive or board attention by having a third party reinforce your message.

Tip 8: Rules of engagement to prevent business disruption.
Set rules of engagement with your pen test vendor, especially around the exploitation of vulnerabilities in production environments, which should be reflected in the pen test scope. We’ve all heard stories of overreach by zealous or inexperienced pen testers. Remember, you will be accountable for their actions.

I like very few rules except for no DDoS and nondestructive testing. Permitting active exploitation means working hand in hand with the testers, daily meetings and a go/no-go decision on exploitation from you and your internal stakeholders before the testers pull the trigger on any exploit.

Exploitation should be detected and blocked well and truly by your security controls. If it isn’t, it’s time to review the configurations of your security controls and even your investment in your current technology.
The alternative to exploitation is to assume exploitation would be successful and either provide the pen testers with the access they would have gained on that system or stop the testing of the vulnerable system and move onto other systems.

Tip 9: Time-boxing when time or money is tight.
Sometimes project deadlines and, more importantly, project budgets won’t extend to the amount of effort a pen test truly deserves. By leveraging the scoping tips above, you can target the highest risks with company-defined test cases and limit the pen test’s duration to an agreed length of time, e.g., two weeks. This approach is known as time-boxing.

It is beneficial for systems that have been pen tested many times before and where there has been little change in the environment. Time-boxing is also a good approach to use with objective-based testing.
Time-boxing does come with the risk of a false sense of security, especially if there is an assumption that the system has had a comprehensive pen test and all the vulnerabilities have been discovered.

I only recommend time-boxing in cases in which those scoping the test understand the implications of it fully, and the pen testers can test the given objectives within the time frame.

Validating Your Security Controls
Tip 10: Test that your security controls are working as expected.
Do the often expensive security controls that are in your environment work as intended? Every budget cycle, we lobby for funding to maintain or purchase security controls to address the ever-changing threats, vulnerabilities and resultant risks that face our businesses.

Do the security controls work as the vendor presentation said they would in the pre-sales meeting? Have the controls been implemented and configured correctly to prevent or detect attacks? Do you have blind spots in your monitoring and detection capabilities?

Most pen testers will start “low and slow” and become increasingly more intrusive. What is the threshold where they are detected by each set of controls, if they are detected at all? Can the tester bypass the controls? I once gained access inside a financial institution by getting remote code execution on an unpatched IDS device.

A noisy pen test should light up the dashboards for your security controls like a Christmas tree.

Use the pen test as an opportunity to tune the controls or tighten up configurations — e.g. firewall rules — to ensure the testers or, more importantly, the bad guys are detected earlier next time.

Tip 11: Test your “watchers” to ensure timely detection.
Not informing the Security Operations Center (SOC) or your Managed Security Service Provider (MSSP) of the pen test is an excellent way of testing if those teams are watching the consoles or are drowning in alerts. I’ve seen pen tests complete or others go almost to completion — including breaching the perimeter — without being detected.
Reviewing how the testers got in and the tools, techniques and processes they used are a great way to help your SOC hone its threat-hunting skills and tune signatures.

The lessons learned from gaps in detection can also help test and uplift your digital forensics and incident response processes.

Summing It Up
Pen testing can uncover infrastructure or vulnerabilities you weren’t aware of and help the organization better understand its cybersecurity risks.

By adopting a targeted approach to penetration testing with well-defined scope and test cases that are company- or objective-specific, you can derive maximum value from your penetration testing spend.