Test Automation in AppScan® Enterprise using AppScan Traffic Recorder

The HCL AppScan Traffic Recorder enables you to record traffic to use as Explore data in your AppScan Enterprise (ASE) job. The HCL AppScan Traffic Recorder is a system for managing traffic recorder instances. Traffic recorder instances can be created on demand to record traffic that will later be used for a DAST scan. This topic describes how to set up and use the traffic recorder.

Overview

In the world of DevOps it's increasingly important to be able to incorporate security scans in your CI/CD processes. If you use an automation framework (such as Selenium), you can take advantage of the scripts that are already written, to create tailor-made scans.
  • Using the HCL AppScan Traffic Recorder, you can automatically start a traffic recorder instance. The requests from the automation framework to the web app are then recorded as they are sent through the Traffic Recorder. The traffic is saved in HAR format in a .dast.config file, that you can later upload to be used by AppScan Enterprise (ASE) as Explore data for a scan.
  • You can also record traffic manually, through the Traffic Recorder, to create a .dast.config file.
  • You can also upload your own HAR file directly to ASE, without using the Traffic Recorder.

This diagram depicts the automated scan flow with AppScan® Enterprise.


Automated scan flow with AppScan Enterprise

Typical Workflow

  1. Install Traffic Recorder
  2. Configure Traffic Recorder
  3. Start Traffic Recorder
  4. Using the Traffic Recorder
Note: The demo script for this workflow ProxyServerDemoScript.py can be found in the AppScan installation folder.

Install the HCL AppScan Traffic Recorder

Prerequisite: You must have a current version of Node.js (v16 or later LTS version) installed on your machine. If your service is FIPS compliant, you must use a FIPS-compliant Node.js server.

  1. Windows: Unzip TrafficRecorder.Windows.zip to a folder on your machine.

    Linux: Unzip TrafficRecorder.Linux.zip

  2. To overwrite a default port for the Traffic Recorder to run on: Configure the port setting in Settings.json.
  3. Ensure app.js and internal java file: /java/bin/java have “run as executable” permissions.

Configure the HCL AppScan Traffic Recorder

After making any configuration changes, you must restart the server.
Note: When upgrading, if you already have a PKCS12 certificate set in Settings.json, you will need to set it again.

FIPS compliant mode

  1. Locate the file Settings.json in the root folder of the Traffic Recorder and open it in a text editor.
  2. Locate the setting requireFips and change its value from false to true.
  3. Save the file.

Traffic Recorder connection

  1. Configure a default port in Settings.json for the Traffic Recorder to run on.
  2. Configure a secure (SSL) connection to the traffic recorder. You can do this using your own certificate (Method A) or a self-signed certificate (Method B).

    Method A: Configure your own PEM or PKCS12 certificate in Settings.json.

    PEM:

    A PEM certificate requires two files paths (to private.key and certificate.pem).
    • Insert the file paths to the PEM section of Settings.json.
      Note: An escaping is required for the ‘\’ char, example: "C:\\Users\\admin\\private.key"

    PKCS12:

    You must escape characters as needed both in file paths and password, for instance, a password such as abc!”123 would become abc!\”123 (the ” symbol is escaped).
    1. In the command line, run:

      .\Java\bin\java.exe -jar .\DastProxy.jar -sc "C:\Path\to\certificate.pfx"

    2. When prompted, enter the certificate’s password and press Enter.

      Example using openssl to create a PEM certificate:

      openssl req -newkey rsa:2048 -new -nodes -keyout key.pem -out csr.pem

      openssl x509 -req -days 365 -in csr.pem -signkey key.pem -out server.crt

      Example using openssl to convert the PEM certificate to PKCS12 certificate:

      openssl pkcs12 -export -out certificate.pfx -inkey key.pem -in server.crt

      Example using Java’s keygen tool to convert a JKS certificate to PKCS12 certificate:

      keytool -importkeystore -srckeystore certificate.jks -srcstoretype JKS -destkeystore certificate.p12 -deststoretype PKCS12

    Method B: If you don’t have a certificate, you must create and use a self-signed certificate. Note that this method is less secure.

    Option A:Using OpenSSL

    OpenSSL is not included and is not required except for the purpose of creating the self-signed certificate.

    1. Create the key and pem files using the OpenSSL command: openssl req -new -newkey rsa:2048 -nodes -keyout private.key -x509 -days 365 -out certificate.pem
    2. Insert the file paths to the PEM section of Settings.json.
      Note: An escaping is required for the ‘\’ char, example: "C:\\Users\\admin\\private.key".

    Option B:Using Java keytool

    For NodeJS 17 and above, you must use “--openssl-legacy-provider” node flag due to keytool limitation, see https://nodejs.org/api/cli.html#--openssl-legacy-provider

    1. In the command line, open the Traffic Recorder root folder.
    2. Create a PKCS12 certificate file using the command: .\Java\bin\keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore-new.p12 -storetype PKCS12 -validity 365 -keysize 2048
    3. Complete the certificate details and protect it with a password, which will be required for the next step.
    4. Use the generated certificate in the HCL AppScan Traffic Recorder using the command: .\Java\bin\java.exe -jar .\DastProxy.jar -sc "keystore-new.p12"

Root Certificate

If the application you are testing uses SSL (HTTPS), the HCL AppScan Traffic Recorder must act as a man-in-the-middle to record traffic. To do this it must have a root certificate that it can use to sign its communication with the app.

By default, the HCL AppScan Traffic Recorder generates a unique root certificate, and no user intervention is needed. However, when browsing the app you will get SSL warnings. You can either ignore them, or do one of the following:
  1. Install the certificate generated by the HCL AppScan Traffic Recorder on your machine(s):
    1. Use the REST API to download the self-signed Root Certificate Authority, used by the AppScan Traffic Recorder, as a PEM file.
    2. Install it on the browser used for the Explore, or wherever needed (depending on where the traffic is sent from).
  2. Import your own root certificate to the HCL AppScan Traffic Recorder. Supported certificate formats are PKCS12 (.P12, .PFX), JKS:
    1. Open a command line window and navigate to the installation folder on the Traffic Recorder machine.
    2. Run the following command:

      .\Java\bin\java -jar DastProxy.jar -irc [path to certificate file] -ircp [password]

      Note: To see the complete command usage, run the .\Java\bin\java -jar DastProxy.jar command.

      Important: Since the certificate will be saved on the Traffic Recorder, it is recommended that you use a dedicated test certificate.

  3. Provide your own fixed server certificate (not a root certificate) in the StartProxy command.

Set Inactivity Timeout

If a traffic recorder instance is not closed with a close command after use, it remains open and listening on the port. Traffic recorder instances are closed automatically if they are idle for a predefined time.

The default inactivity timeout for traffic recorder instances is 60 minutes. You can change this value by changing the value of inactivityTimeoutInMinutes in the Settings.json file saved in the installation folder.

Encrypt Traffic

By default, the traffic (.dast.config) file is not encrypted. To configure the server to encrypt all traffic, change the "encryptDastConfig" value to “true” in the Settings.json file, found in the installation folder.

Chained Proxy

If you need to define more than one chained proxy, or exceptions to the proxy, use the chained proxy rules file (proxy.chain) found in the installation folder. The file includes instructions for use.

Start/Stop the Traffic Recorder

You can simply start the Traffic Recorder, or run it as a service (below). Note that you can’t do both in parallel.

To start the Traffic Recorder:
  • Run the command: node app.js [port]

    where, *port = the port on which the Traffic Recorder listens for REST API requests.

Note:
  • If you do not define the port when starting the Traffic Recorder, the port set in the Settings.json file is used, if none is defined there, port 8383 is used.
  • There is no need to stop the Traffic Recorder, you can leave it running permanently.

Start as a Windows Service

You can run the Traffic Recorder as a Windows Service, using the service.js executable found in the Service folder. Service uses Local System privileges.
Note: All service commands must be run with administrator privileges (Run as administrator).

To start the service, run the commands:

node service.js --install

node service.js --start

Additional Service Commands

You can print this list by running the command node service.js --help

Service commands Description
--start Start the service
--stop Stop the service
--install Install the service
--uninstall Stop and uninstall the service
--h | --help Output usage information

Stop the Windows Service

To stop the Traffic Recorder Windows Service by using the service.js executable found in the ‘Service’ folder, run the command:

node service.js --uninstall

Start as a Linux Service using systemd

  1. Create a new file under systemd directory (usually: /etc/systemd/system). For example, sudo vim /etc/systemd/system/TrafficRecorder.service.

    File content:

    [Unit]

    Description=Appscan TrafficRecorder

    After=multi-user.target

    [Service]

    ExecStart=[path to node here] [path to app.js here]

    Restart=always

    RestartSec=10StandardOutput=syslog

    StandardError=syslog

    SyslogIdentifier=TrafficRecorder

    [Install]

    WantedBy=multi-user.target

    Example for ExecStart: ExecStart= node /home/admin/Documents/TrafficRecorder.Linux.1.2.0/app.js

  2. Reload systemd files by using the command: sudo systemctl daemon-reload
  3. Start the service by using the command: sudo systemctl start TrafficRecorder.service
  4. You can check status by using the command: sudo systemctl status TrafficRecorder.service
    Note: Status should show as “Active: active (running)”
  5. Enable the service by using the command: sudo systemctl enable TrafficRecorder.service

Stop the Linux Service

  1. Stop the service by using the command: sudo systemctl stop TrafficRecorder.service
  2. Disable the service by using the command: sudo systemctl disable TrafficRecorder.service

Using the Traffic Recorder

Once the Traffic Recorder has started, you can start new traffic recorder instances, to record the traffic of your application.

  1. To start a traffic recorder instance, use the REST API request: StartProxy
  2. Send your traffic to the application you are testing, via the defined traffic recorder port.
  3. When done, send the REST API request: StopProxy.
  4. Download the .dast.config file, which contains the recorded traffic, by sending REST API request: Traffic. The file is a ZIP file containing one or more .har (HTTP Archive) files. When you download this file, the data is deleted from the HCL AppScan Traffic Recorder.
  5. You can use the .dast.config file to update the Explore data of an existing job, using the ASE REST API. See the documentation link: https://<domain>:9443/ase/api/pages/apidocs.html. For more information, see Traffic Recorder API commands section.

Traffic Recorder API commands

Once the Traffic Recorder has started, you can start a traffic recorder instance, and send your traffic to your application.

Documentation

To view the complete REST API documentation in Swagger, type in a browser: https://<server>:<port>

All commands point to an endpoint that looks like this:

https://[server]:[port]/automation/

server = IP address of the machine on which the Traffic Recorder is installed (default is localhost)

port = port on which the Traffic Recorder listens

"StartProxy"

Start a traffic recorder instance that listens on the specified port.
  • URL: https://[server]:[port]/automation/StartProxy/<recordingPort>
  • Request type: POST or GET (if using chainedProxy, proxyCertificate, and clientCertificate the request is POST; otherwise it is GET)

"StopProxy"

Stop the traffic recorder instance that is listening on the specified port.
  • URL: https://[server]:[port]/automation/StopProxy/<recordingPort>
  • Request type: GET
Note: Setting the port to 0 does not stop all open traffic recorder instances. Use StopAllProxies to do this.

"StopAllProxies"

Stop all running traffic recorder instances. This command stops all traffic recorder instances on all ports, including those started by other users.

  • URL: https://[server]:[port]/automation/StopAllProxies
  • Request type: POST

"EncryptDastConfig"

Upload DAST.CONFIG file for encryption.
  • URL: https://[server]:[port]/automation/EncryptDastConfig
  • Request type: POST

"DownloadEncryptedDast"

Download encrypted .dast.config file (that was uploaded using the EncryptDastConfig API).
Note: When the file is downloaded, both encrypted and unencrypted .dast.config files are deleted from the Traffic Recorder.
  • URL: https://[server]:[port]/automation/DownloadEncryptedDastConfig/<uuid>
  • Request type: GET

"Traffic"

Download recorded data from the Traffic Recorder identified by its port, as a .dast.config file.
  • URL: https://[server]:[port]/automation/Traffic/<recordingPort>
  • Request type: GET

"Certificate"

Download the self-signed Root Certificate Authority, used by the Traffic Recorder, as a PEM file.
  • URL: https://[server]:[port]/automation/Certificate
  • Request type: GET