CRWAE1062I

Skipping link (Scanning not allowed by robots.txt)

The robots.txt file tells visiting robots what directories they are not permitted to search when traversing the website.

To turn off this option, go to the General Scan Options page of the content scan job and clear the Use exclusions specified in the robots.txt file check box