The primary goal of a web application security scanner is to eliminate the repetitive drudgery of penetration testing, leaving testers free to use their skills in areas where they’ll make a real difference.
In a typical application security audit, automated tools perform the initial triage and then experienced testers sift the results, positively confirming the reported vulnerabilities and working with the developers to eliminate them.
Unfortunately, as any experienced penetration tester will tell you, the confirmation stage often focuses more on eliminating false positives than dealing with real vulnerabilities. This is because automated scanners are notorious for misreading the signs and reporting vulnerabilities that don’t actually exist. In fact, the problem is so commonplace that most pen testers consider it to be an unavoidable consequence of automated testing.
False positives occur because of the weak static checks that scanners typically use to detect vulnerabilities. When a scanner tries to detect a known vulnerability, it uses a matching algorithm to look for one or more pre-defined signature patterns within an HTTP response. If a match is found (and subject to some additional extra checks that may increase the level of certainty) the scanner will deduce that the vulnerability exists and report it accordingly.
However, there are many situations where a positive signature match may be erroneously triggered. For example, the text in a web page may match a given vulnerability signature, causing a scanner to report a harmless static page as a security vulnerability. Although the proportion of erroneous matches may be relatively small, the false positive count may grow quickly when scanning a large web application, resulting in a significant additional workload for the penetration tester.
Aside from the obvious economic consequences of having to manually check and eliminate non-existent vulnerabilities, the problem also has some side effects that can be a direct threat to security. Since web applications evolve, security testing is an ongoing process that must be repeated every time the codebase is modified. It is therefore natural that a tester will encounter the same false positives time after time. On the first occasion, she will diligently check every reported vulnerability and eliminate the false positives. However, as time passes and the same vulnerabilities are repeatedly reported by the scanning process, there is a dangerous tendency for testers to disregard those that they “know” are false positives. This is when a new and very real vulnerability can go unnoticed.
Another equally worrying scenario is when security auditing is performed by staff with only rudimentary knowledge of penetration testing. For example, a developer may use a scanner to reassure himself that his application is secure. Assuming the scanning process reports no vulnerabilities, he may be reasonably assured that his work is done. But if vulnerabilities are detected, additional confirmation steps are required and this is a minefield for the uninitiated.
Given the list of vulnerabilities reported by his scanner, the developer will make initial attempts to confirm them by exploitation but, with only limited experience, this challenge will likely lead him into unknown territory. Eventually, after failing to exploit a vulnerability using all the common textbook techniques, the developer may fall back on the comforting assumption that it is a false positive. This is a justifiable rationalisation, given the pitiful reputation of most scanners, but it is nonetheless a risky strategy and one that will ultimately lead to real vulnerabilities slipping through the net.
So, what’s the solution?
Most scanner vendors devote considerable effort to continually improving the quality of their matching algorithms and adding new layers of cross-checks that seek to improve the level of certainty of any vulnerability report. Whilst this approach provides incremental gains in reporting accuracy, it doesn’t address the underlying problem – that a security vulnerability can only be absolutely confirmed if it is successfully exploited. Any amount of tinkering with signatures and matching algorithms is effectively just window dressing, as the only 100% reliable mechanism for confirming a vulnerability is exploitation.
When we designed Netsparker, this thinking was at the core of our design process. We knew it was going to be a tough challenge to achieve our goal of positive confirmation, because exploiting a vulnerability programmatically entails dynamically creating attacks that exactly match the context of the page they are attacking.
But the results speak for themselves. When Netsparker reports a confirmed vulnerability, it does so because it has successfully exploited it (in a completely safe and non-destructive manner). This diagnosis leaves no room for doubt – the application contains a real security vulnerability that requires action.
There are certain types of vulnerabilities that cannot currently be confirmed by automated exploitation and, for these, Netsparker classifies them as either Possible or Probable, depending on a certainty rating that is calculated using heuristic techniques. In any typical scan, the number of reported vulnerabilities in this category will always be in the minority and usually represent non-critical vulnerabilities or vulnerabilities without direct impact such as information disclosure issues.
Netsparker is currently the only web application security scanner that uses built-in exploitation technology to positively confirm vulnerabilities. This is, perhaps, why our competitors seem content to play down the issue of false positives, preferring to keep the industry’s dirty secret under wraps.