4 Reasons Why Black Box Testing is Misguided

Many organizations use third parties to conduct a variety of security tests of their Internet-facing and internal systems, including vulnerability scans, web application tests and penetration tests to identify areas of opportunity for attackers. When specifying these engagements, the scope and level of testing must be determined. The information provided to the third party may range from none, such as customer name only, to all, such as lists of live IPs, hostnames, or even authentication credentials.


What is Black Box Testing?

“Black box” testing is a common catchphrase that is used to describe testing that starts with minimal information. Often the organization name and maybe headers from an email are the only items available. The tester must perform research and a variety of tests and probes to determine targets for potential exploit. The goal is to place the tester in the role of an actual external attacker and avoid the discovery of items that might remain undiscovered in a real-world attack.


Does it work?

Does this approach truly mimic an external attacker and achieve the goal of a real-world test? Kinda. It only serves toward in the context of an individual attacker with limited time and resources targeting the organization which does not mirror what attackers are doing these days. Attackers today perform research on their targets, both people and technologies before starting their attacks. Today, third-party security testers will usually start with available public information such as websites, DNS records and message board looking for clues to identify hostnames and IP addresses. But, if the website and mail server are hosted externally, the Internet-facing IP addresses of the organization’s headquarters and other offices may not be discovered. The organization may feel that this is the desired state and an accurate security test. But let’s consider how some successful attacks are conducted to compare.

Many attackers look for targets of opportunity. For example, an attacker may scan for systems that have the Cisco Smart Install buffer overflow vulnerability (CVE-2018-0171) simply by choosing Internet network ranges and scanning for port 4786. An organization may hide information about the IP addresses it owns, but that will not provide protection from an attacker that scans IP addresses sequentially.

This technique was successfully used in 2018 to target Cisco devices in Russia and Iran. Another example involves attackers that wish to store and share data such as malware, software with licensing protections removed, and videos and games with copyright protections removed. They might use the same technique to find available file servers. Such an attacker is not concerned with who the target is, only with the opportunity for exploit.

Also, a black box test performed during a typical three-week engagement will not uncover the number of vulnerable targets than a determined attacker will uncover. A real-world attacker may spend many months performing research, reconnaissance, social engineering (email and telephone “phishing” attacks), and discovery tests. While a third-party tester is capable of this level of effort, it is hardly efficient and may still leave some systems undiscovered.

Reasons to avoid black box testing:

  • Some available (and vulnerable) systems are likely to be undiscovered
  • Requires more time (and expense) for a proper test
  • Does not determine the true risk to the organization
  • Does not meet most compliance standards


What should I do?

For a truly accurate security test as well as an efficient one, organizations must at a minimum specify network addresses ranges, and for proper pricing, the expected number of live hosts. Then the security tester can perform discovery scans to find live systems and determine which are vulnerable and which are exploitable. This approach provides a true assessment of the risk posed by an attacker looking for targets of opportunity as well as by a highly determined individual or group of attackers that have unlimited time.

Another reason to provide information to the third-party tester is that black box testing does not meet most compliance standards. When test results are presented to an auditor or to management, a typical request might be to view the results for specific sensitive systems (VPN interface, external firewall IP address, etc.). The organization must show that these systems were tested, the type of tests, and the results.

Contact us for help with Security Testing