Best practices for running a security scan in a production environment

Performing a security scan in a production environment is risky; however, it might be necessary to scan a production environment, perhaps to comply with audit requirements, to detect whether your site has been hacked, or to validate that the SDLC process for integrating security scans is being employed.

Regardless of your reasons, it is always best to begin scanning a preproduction environment and then move the scan to your production environment. Doing so will help to ensure that the security tests pose less risk to your servers.

Learn more about known security risks when scanning a production site:
  • Most severe: The scan might bring the server or application down.
  • Less severe: The scan re-sends the same request to the server in many variants - approximately 40 per parameter. Some of those requests will be rejected by the server, but many of them will go through and be processed by the application, which might lead to the following actions:
    • Add or change records in the database.
    • Complete transactions, such as a purchase, that the application was designed to perform.
    • Email flooding. If a form sends email to a mailing list or to an individual, the recipients might receive hundreds of them.
    • Defacement of the site. If the site has a forum page, the scan might post many entries to it, including some with cross-site scripting pop-up windows.
    • Complete modification of the site. If the site allows for changes, such as with create, edit, and delete buttons, normal requests as well as many variant requests will be submitted.
    • Initiate® requests to vendor services. These requests might spam the services, conduct transactions, and potentially incur a cost.
    • Alert your IDS/IPS (Intrusion Detection System/Intrusion Prevention System) team. The tests will trigger IDS/IPS with thousands of signatures. The tests are not true exploits and are not designed to harm your applications. However, the IDS/IPS team would not know this and they would react appropriately.
    • The IDS and IPS systems will report the testing as an attack.

The following best practices are for implementing a security scan into a production environment, with the least amount of risk and most amount of value.

Use a test user account

Use a test account that can be tracked to ensure that services are not really ordered and so that it can be reset in case it becomes corrupted. A test account also makes it easier for the Administrators to clean the site after the test. Consider the following factors for the test accounts:

  • It should only have access to test records in the database, so that modified records can be restored.
  • Delete new records created by the test account.
  • Ignore purchase orders or other transactions from the test account.
  • If the site has forums, the test account should only access test forums, so real customers will not be able to see the tests during the test phase. For example, seeing a cross-site scripting pop-up window can be frightening.
  • Use more than one test account if the site uses different privileges for different accounts. Using multiple test accounts will ensure a more comprehensive test of the application.

Conduct user acceptance and performance tests

Do not run security tests in production until after the application has been configured in production to handle normal user acceptance and performance testing. If the application is unable to handle these types of tests, it will not be able to handle the mutated requests sent by the scan job.

Get approval

A unilateral scan of any application without consulting with the business owner and development team, and obtaining their approval, is almost always a disaster. Remember to notify all parties in advance when the tests are to be performed and when they are expected to complete.

Establish an appropriate time to run the scan

Run the scan at a preset time that will not interfere with the regular operating window for the site, such as overnight.

Do not overlook running invasive tests

Invasive tests are not run by default, but they are very important for a complete test; for example, buffer overflows carry Internet worms. All tests, both invasive and noninvasive, have the potential to bring sites down.

Prevent email flooding

Identify pages that use email notification and configure these to be excluded from the scans. Email notifications will generate a lot of requests and can overload an email server. Identifying these pages can be difficult, however here are several techniques you can use:

  • Test the pages in QA and exclude them in production. In QA, change the email address to a dummy mailbox.
  • In production, test only one web server and prevent it from connecting to the SMTP server during the test, or have a script that replaces the email address before the test and restores it after the test.
  • Configure the scan to place a unique value in one of the email fields so the recipient can run a search and clear all of them. Use the scan Automatic Form Fill to perform this function.

Develop and test cleanup scripts in advance

After the production scans are executed, use cleanup scripts or programs in order to remove test data created by the scans.

Identify session parameters and tokens

Identify session parameters and tokens that would invalidate the session. You can typically get this information from the application owners. Add session parameters on the content scan job's Parameters and Cookies page.

Identify account invalidation routines

Identify any account invalidation routines. The scan should avoid these routines by adding them as exclusions.

  • If possible, turn off account invalidation for failed password attempts.
  • If it is not possible to turn off account invalidation, then scan each login page separately using a test account and password.

Consider scans without authentication

Scan pages that can be accessed outside of the application without authentication, using a separate scan job. Performing a scan from the outside can find more issues and it will help you understand who has access to the security issues found.

Use interactive scans to identify sensitive areas

Create a separate job to deal with areas requiring interaction, such as a change password page, an administrative settings page, or a user management page.

Disable IDS/IPS systems

Because IDS/IPS systems might block tests and hide issues that would be otherwise exposed, it is best to disable these systems or scan around them. At the very least, you should alert your IDS/IPS team.

Do not scan through a proxy

While scanning through a proxy is supported, it has the potential to hide issues. For example, HTTP proxies might tamper with the server error responses and thus hide security issues.