Manually exploring your site to add more URLs to the scan

A Manual Explore means you will be indicating the exact URLs for the scan to test in the configuration (the scan will not automatically crawl to discover new URLs). Use this method for web applications that require a lot of user interaction to navigate the application or if you would like to just test a specific area of your application.

Before you begin

Get ready:
  1. Ensure that the Internet Advanced Options of Internet Explorer are set to use HTTP 1.1 before doing a manual explore.
  2. If you are manually exploring applications on your local machine, you must ensure that the host name used in the manual explore browser is different than the host name used to access the Enterprise Console. Otherwise, the scan might not be able to access the URLs. For example, if you access the Enterprise Console using https://server1/ase, use https://server1.domain.com/ase when manually exploring.

About this task

Manually explore your site if you:

  • do not know the exact URL to add pages to the list of Starting URLs.
  • want to add pages not discovered automatically by a scan because the scan misses them (for example, nonstandard js postbacks as links, embedded js, or flash links).
  • want to add pages that are not discovered automatically for other reasons (for example, orphan pages).
You can also use Manual Explore in combination with an automatic crawl of your website. In this case, the scan tests all of the pages that you manually visit and those pages that AppScan Enterprise discovered automatically. By default, AppScan Enterprise includes automatic explore, but you can turn it off by using one of these methods:
  • For content scan jobs in the Scans view, go to the Explore Options page. In the Scan Limits section, select Specified URLs limit (URLs specified in Starting URLs, Manual Explore and Recorded Login properties. No spidering).
  • For *.scant template-based scans, go to the Job Properties page in the AppScan Dynamic Analysis Client. In the Scan section, select Test Only.
Sometimes you might want to test only a few pages; for example, if they are currently being developed, or they contained issues that are now fixed. Use Manual Explore with one of the options that are mentioned above to run a small, isolated scan. On other occasions, you might want to scan the entire site. You can combine the Manual Explore and the Automatic Explore options to ensure that all pages are visited for complete coverage. In those instances, use the default options instead.
CAUTION: Do not use any private information in your scan configuration because this data might be viewed by a third party. To proceed with the browser recording, ensure that you have logged out from any existing sessions. Use a test user account during the manual explore to prevent usernames and passwords from appearing in clear text in the Enterprise Console interface.

Procedure

  1. In the Manual Explore section of the What to Scan page of the job, click the Add icon (Add).
  2. On the Manual Explore page, select Use manual explorer tool or AppScan Standard explore data file.
  3. Download and install the tool.
    Note: The machine that hosts the Manual Explorer tool must also be FIPS enabled so that the tool works properly.
  4. To launch Manual Explorer, go to Start Menu > HCL AppScan Manual Explorer or the desktop icon.
  5. Click File > Preferences and configure the settings for the recording tool:
    • browser
      Note:
      1. Internet Explorer/Google Chrome: If any instance of the browser is running before you record, close them, including the instance where AppScan® Enterprise is running. When you are finished the recording, you can reopen your browser.
      2. Mozilla Firefox:
        • If AppScan® Enterprise is using the system proxy and you try to record with Internet Explorer or Google Chrome, close the Firefox browser, perform the recording, and then reopen the Firefox browser.
        • If you are using Firefox for the first time to record traffic data using Manual Explorer, make sure all open Firefox browser instance are closed.
    • invalid certificate connections
    • preferred proxy port. During recording, if this port is in use, another port will be used instead and will be indicated here.
    • trace log level
  6. Click Record on the AppScan® Manual Explorer tool and navigate your application.
    Note: If you explore an https:// site, you might get an error regarding an invalid certificate. This is an invalid certificate on the Manual Explorer tool, not the website; accept the certificate.
  7. When you have finished exploring the site, save the file and close the Manual Explorer tool.
  8. In the Manual Explore page of the content scan job, import the *.htd file, close the window, and click Save to add the URLs to the scan.
  9. On the Manually Explored URLs page, review the list of URLs that were discovered.
  10. Select the URLs you want to remove from the Manually Explored URLs list and click Remove.
  11. Select the domains you want to remove from the Manually Explored Additional Domains list, click Remove; then click Save.
    Note: If you click Save accidentally before you are finished editing, you can still make your edits in the What to Scan page.
  12. On the Manually Explored Auto Form Fill Fields page, review the Auto Form Fill Fields that were discovered during the manual explore, remove any field you do not want included in the scan, and click Save.
  13. (Optional) If you want the scan to test the URLs as an ordered sequence, select the check box in the Manual Explore section of the What to Scan page. Select this option when parts of your web application can only be reached by sending requests in a specific order (multi-step operation). The scan will play back the URLs in the order you recorded them before it sends tests.
    Note:

    Some parts of a web application, such as a shopping cart or applying for a bank account, can only be reached by sending requests in a specific order. You can configure the scan to play back these URLs in sequence. In this example, a user shops online and visits three pages in an online shopping cart application:

    • Page A: Adds one or more items to the shopping cart.
    • Page B: Fills in payment and shipping details.
    • Page C: Receives confirmation that the order is completed

    Page B can only be reached from Page A. Page C can only be reached from Page A, followed by Page B. During the manual explore, you record a single sequence: Page A > Page B > Page C. To test Page C, the scan must send the correct sequence of HTTP requests before each test. When testing Page B, the scan will send a Page A request first; when testing Page C, it will send a Page A request, followed by a Page B request.

    1. Scan sends A, performs test 1 on B
    2. Scan sends A, performs test 2 on B
    3. Scan sends A, B, performs test 1 on C
    4. Scan sends A, B, performs test 2 on C
    Due to the nature of multi-step operations, scan performance might be slow because the multi-step requests are sent in single-threaded mode.

Results

The URLs you add from a manual explore are added to the Additional URLs list and treated the same way as the list of Starting URLs. The domains you add from a manual explore are added to the Additional Domains list.

What to do next

Adding additional servers and domains to the scan