Site content crawler configuration

The site content crawler uses configuration files and manifest files to determine the site content crawler behavior.

droidConfig.xml

The site content crawler configuration file contains variables and parameters that determine the site content crawler behavior.

• hostname
• storePathDirecttory
• langId
• storeId
• catalogId
• language
• localename
• filterDir
• port

The variables that are specified in the site content crawler configuration file are then used to populate values further in the configuration file.

initialLocations

The starting URL for the site content crawler.

Important: You must update the starting URL for the site content crawler to operate correctly.

For example:

http://${hostname}/webapp/wcs/stores/servlet/StaticContentSitemap?storeId=${storeId}&langId=${langId}&catalogId=${catalogId}

relativePath

If specified, the relative path is omitted from the URLs added into the manifest file. For example:

4,StaticContent/Recipe.html,8fa661c4-f812-4b3c-aa5c-361894120d23.html,text/html,UTF-8,A,3 

If not specified, then an absolute path is set in the URLs. For example:

4,http://wcsolr05/webapp/wcs/stores/servlet/StaticContent/Recipe.html,5b770798-cd9a-478d-9fb3-b75c1e1c3b91.html,text/html,UTF-8,A,6 

It is important to set the relative path so that production servers do not point to the staging server, but rather point to itself.

depth

The maximum depth the crawler crawls. A value of -1 denotes no depth restrictions.

max

The maximum number of pages to crawl. A value of -1 denotes no maximum.

delay

The delay time in milliseconds between each HTTP request.

filters

The host filter configuration file location.

threadmode

The site content crawler thread mode.

0

Single thread mode

1

Multiple thread mode

maxthread

The number of threads to create when in the multiple thread mode.

destLocation

The destination directory to save the site content and manifest files.

updateCommerceDB

Used to set the languageID and indexSubType condition to select from the SRCHCONFEXT table.

autoIndex

Indicates whether to enable automatic indexing of site content after the content is crawled.

skipDownload

Indicates which URLs to not add into the manifest.txt file, therefore not indexing them. For example, StaticContentSitemap.jsp:

http://${hostname}/webapp/wcs/stores/servlet/StaticContentSitemap?storeId=${storeId}&langId=${langId}&catalogId=${catalogId}

jndiName

The jndi name of the JDBC data source for example, <jndiName>jdbc/jndiName</jndiName>. It is only used when you run the crawler through URL. When this parameter is specified, the crawler can use that data source to update the database after the crawling finishes.

filters.txt

The filters configuration file determines whether URLs are included or ignored by the site content crawler.

You can update the filters configuration file by using regular expressions to include or ignore values.

Important: You must update the filters configuration file to include your WebSphere Commerce host name.

The default sample values contain ignores such as excluding URLs containing email or FTP links, or pages that require logging on to the site.

SiteMap.jsp

The site map, which is used by web browsers and external search engines, contains pointers to the different starter store pages

StaticContentSitemap.jsp

The static site map contains pointers to the static content files that are in the WebSphere Commerce database.

The URL that is passed from the configuration file to the site content crawler is:

http://host_name/webapp/wcs/stores/servlet/StaticContentSitemap?storeId=storeId&langId=-1&catalogId=catalogId

You must update the static site map file to include your additional static content files that are in the WebSphere Commerce database.

This file is only by the site content crawler.

Site content crawler manifest files

The site content crawler manifest.txt output files are comma-separated values (CSV) formatted documents that contain generated information.

1. The manifest file that indicates which folder contains the downloaded site content files. It contains the following columns:

   Timestamp

   The time stamp for the column.

   Directory path

   The counter directory path.

   Initial location URLs

   The initial URLs separated by a comma.

2. The manifest file that contains the mappings of downloaded files to URLs. It contains the following columns:

   ID

   The ID that distinguishes each file in the document. For example, a simple sequence.

   URL

   The relative URL to the current store, or full URL pointing to external resources.

   Local file path

   The file path, either in full format or relative format, of the stored site content.

   Content-type

   The content type of the file. For example, text/html.

   Encoding

   The encoding of the file, if it is a text-based file.