URL rewriting in web applications is one big headache for both automated web security scanners and their users. If you scan a website that uses URL rewrite technology and do not configure URL rewrite rules in your scanner, the parameters in URLs won’t be scanned. And if such parameters are vulnerable the scanner won’t report such vulnerabilities, leaving them open to potential exploitation.
Failing to configure URL rewrite rules in scanners also mean that the web security scan might take forever to finish. Some scanners will even report false positives and go haywire if you don’t configure URL rewrite rules, yet many users still do not configure them. So we automated the whole process of configuring URL rewrite rules in web security scanners.
This blog post gives an insight of how this new automated technology, which will be introduced in the upcoming versions of both Netsparker Desktop and Netsparker Cloud will work. It also explains why we automated the process of configuring URL rewrite rules in our web application security scanners.
Addressing the Problem of Configuring URL Rewrite Rules in Automated Web Security Scanners
Many users fail to configure URL rewrite rules in their automated security scanners because in most software, they are very difficult to configure. Most of the time you need to have access to the web server’s configuration files, and need to know how to write regular expressions. As always, we wanted to automate as much of this process as possible, so users can scan all of the websites’ parameters without getting bogged down into configuring their web security scanner.
Manually Configuring URL Rewrite Rules in Netsparker Is Easy
To configure URL rewrite rules in both Netsparker Desktop and Netsparker Cloud you do not need to have any information on how the web server is configured, and you do not need to know how to write regular expressions. All you need to do is use the URL rewrite rules wizard to specify the parameter name and type, the rest is all automated.
Even with such wizard in place many users still do not configure URL rewrite rules. Most users just want to launch a web security scan and get a report with all the vulnerabilities to fix. Only the very few really go into the detail of configuring several different aspects of the web security scan. So as per our usual, we thought of automating the process of configuring URL rewrite rules while still allowing the old school geeks to configure their own rules.
Limitations of the Existing URL Rewrite Rules Configuration
To understand how this new technology works, first you have to understand the limitations of the existing solution. At the moment Netsparker users can select one of the below options when scanning a web application which has URL rewrite enabled:
Use Heuristic URL Rewrite Support
When this option is selected the scanner tries to automatically determine if URL rewrite technology is being used on the target website, so if it is the scan is limited. Therefore if during a scan the scanner detects the below directory structure it will stop after a number of tries to avoid going into a loop:
http://www.example.com/users/1
http://www.example.com/users/2
http://www.example.com/users/3
…
This solution has a number of limitations. First of all, the scanners will not scan the parameters in URLs and it only recognizes numeric parameters, such as the ones in the example above.
Use Custom URL Rewrite Rules
When this option is enabled you have to configure URL rewrite rules in Netsparker scanners so the scanner can scan all the parameters in the URLs. The problem with this solution is that even though when compared to the configuration of other scanners it is a very easy solution, users still need to have some basic knowledge of the link structure of the target website, hence manual configuration is involved. And as you might have noticed by now, manual configuration is in Netsparker’s bad books.
Automatic Detection and Configuration of URL Rewrite Rule of Target Websites and Web Applications
To automate the configuration of URL rewrite rules we improved the Heuristic URL Rewrite engine. Therefore if you want to scan a website or web application which has URL rewrite enabled you can either configure the URL rewrite rules manually or select the option Use Heuristic URL Rewrite Support.
![Heuristic URL Rewrite Rules configuration in Netsparker Desktop]()
Screenshot: Heuristic URL Rewrite Rules configuration in Netsparker Desktop
![Heuristic URL Rewrite Rules configuration in Netsparker Cloud]()
Screenshot: Heuristic URL Rewrite Rules configuration in Netsparker Cloud
By default, when Netsparker Desktop or Netsparker Cloud crawl more than 60 URLs that have the same pattern, they will automatically create a custom URL Rewrite rule and scan the parameters in the URLs. Below is a list of different types of parameters which Netsparker can automatically identify;
String Parameters
Example URL: http://www.example.com/user/robert
Pattern: /user/{param1}
Numeric Parameters
Example URL: http://www.example.com/userid/1
Pattern: /userid/{param1}
Parameters with Prefixes
Example URL: http://www.example.com/users/user_robert/
Pattern: /users/user_{param1}
Parameters with Suffixes
Example URL: http://www.example.com/users/robert_user/
Pattern: /users/{param1}_user
Multiple Parameters in Same URL
Example URL: http://www.example.com/users/123/robert/
Pattern: /users/{param1}/{param2}
Slug Based Parameters
Example URL: http://www.example.com/blog/pci-dss-good-bad-insecure/
Pattern: /blog/{param1}
Fine Tuning the Automatic Detection and Configuration of URL Rewrite Rules
As seen in the above screenshots the Netsparker scanners have four new settings in the Heuristic URL Rewrite Support configuration. These settings can be used to fine tune the automated detection and configuration of URL rewrite rules and below is an explanation of what each of the setting is used for.
Note: The default settings work well in most of the cases and should only be changed should you notice any performance issues.
Maximum Dynamic Signatures
Use this option to specify how many URLs Netsparker should crawl before it determines that URL rewrite rules are needed to scan the target website. The scanners determine if URL rewrite rules are needed or not by analyzing the patterns of the crawled URLs.
If there are more than 60 URLs (default value) that have a similar pattern, for example http://www.example.com/category/[x] then the scanners will automatically create the neccessary URL rewrite rules to crawl the website and scan the parameters in the URLs.
Block Analyze Threshold
Use this option to specify the number of URLs Netsparker should crawl before checking if any of the crawled URLs have a matching pattern. Therefore if the configured value is 20 (default), each time Netsparker crawls 20 URLs it will check if there are 60 (configured in the option Maximum Dynamic Signatures) of the crawled URLs that have a matching pattern.
The process of checking if URL rewrite rules are needed is very expensive in terms of resources, hence why it is only done every so often.
Block Merge Threshold
Use this setting to specify the number of URLs that have a matching pattern but different parameters in the URL that Netsparker should crawl before merging them together. For example Netsparker crawls the following URLs:
/category1/page1 -> /category1/page30
/cateogry2/page31 -> /category2/page60
In such case the Netsparker URL rewrite analysis engine splits each of the the crawled URLs in the following 4 blocks:
Since there are 60 pages which have a matching URL pattern (page1 -> page60), but they have a different parameter name in the URL (category1 and category2) Netsparker merges the two blocks of URLs and creates a URL rewrite rule for these pages, such as /category1/{param1}.
This also applies to the category parameter. Therefore if Netsparker crawls more than 60 different category URLs, for example /category1/... -> /category60/... it will also merge those blocks of URLs and create a URL rewrite rule for them, such as /param1/param2/.
Block Separators
Use this option to specify the characters used on the target website to separate the blocks in a URL. The default list has the following characters: / _ $ . , ; | :
Therefore when using the default list, the scanner will split a crawled URL such as http://www.example.com/user/robert_abela in 6 blocks, as highlighted in the below screenshot.
![A URL split into blocks by Netsparker heuristic URL rewrite rules engine]()
Reporting the Detected URL Rewrites on Target Website
Once Netsparker Desktop or Netsparker Cloud automatically detect and configure URL rewrite rules for a target website they will report it in a new node URL Rewrite in the Knowledge Base node, as seen in the below screenshots.
![Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Desktop]()
Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Desktop
![Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Cloud]()
Automatically configured URL rewrite rules are reported in the Knowledge Base node in Netsparker Cloud