Sharding input properties file

The sharding input properties file is a file that you create that is passed in to the di-parallel-process utility when indexing shards.

The sharding input properties file contains the following sections and properties:

System properties

System properties are shared by other utilities. For example, passwords that are common among utilities.

PasswordFileDir: The location of a password.properties file that you create that contains passwords encrypted by the wcs_encrypt utility.

Database properties

Database properties are used to establish database connections with the database server. All database properties start with the Database keyword, followed by the property name, which is separated by a period.

The following list shows the database properties:

Database.dbType: The database type. The accepted values are db2 or oracle.
Database.dbName: The database name.
Database.dbHost: The remote database host name.
Database.dbServerPort: The remote database port number.
Database.dbUser: The database user name.
Database.dbSchemaName: The database schema name. This value is typically the uppercase version of the database user name.
Database.jdbcURL: The database JDBC URL.
Database.jdbcDriverLocation: The database JDBC driver location.
Database.jdbcDriverType: The database JDBC driver type.

Global preprocessing and indexing properties

Global properties are used for preprocessing and indexing by all shards, if not otherwise overwritten by the shard itself. All global properties start with the Global keyword, followed by the property name, which is separated by a period.

Global properties can be grouped into the following categories: server properties, preprocessing properties, and indexing properties.

The following list shows the global server properties:

Global.index-server-name: The search server host name.
Global.index-server-port: The search server port number.
Global.index-server-user: The search server user name.

The following list shows the global preprocessing properties:

Global.preprocessing-locale

A single preprocessing locale value to be preprocessed. The default value is All, where all locales are preprocessed.

A comma-separated list of locales is allowed.

Global.auto-preprocess

Automatically preprocess all shards.

Global.preprocessing-drop-temp-table

Indicates whether to drop tables when preprocessing the search index.

Passing in a value of false uses a TRUNCATE statement on the tables.

The default value is true, which uses a DROP statement on the tables.

Note: This parameter supports only DB2 9.7 or later, or Oracle databases.

Global.preprocessing-deepSequence

Optional: Enables preprocessing for the deep search sequencing feature.

The default value is false.

Global.preprocessing-truncate-table-sql

Overwrites the default TRUNCATE table statement.

Global.preprocessing-dir

The path to the preprocessing XML files in the HCL Commerce Utility Docker container.

Global.preprocessing-initial-global-table-dir

The preprocessing XML under this path is processed just once before sharding. This path is the preferred location for preprocessing configurations for initial temporary tables that do not need to be sharded or created multiple times.

To reduce the customer's configuration efforts, auto-sharding is supported in HCL Commerce Version 9.0.0.5+ and enabled by default in Version 9.0.0.7+. When this feature is enabled, the parallel process utility can automatically divide the entry table CATENTRY into equal pieces by row number, and create temporary tables based on those pieces. The shard names are generated automatically according to the number-of-shards value, starting from A. For example, if number-of-shards equals 3, the shards would be named A, B and C.

CAUTION: You cannot use auto-sharding and manually specified shards at the same time. If you do, Global.locale will be unable to overwrite each shard's index-core-name and index-core-dir for the autosharding process, resulting in unpredictable behavior.

The following list shows auto-sharding properties.

Note: Auto-sharding is compatible with vertical sharding and the combination is recommended.

Global.auto-shard

Flag to automatically do shard. In HCL Commerce Version 9.0.07+ this property is enabled by default.

Global.number-of-shards

Number of shards into which to divide the index. The permitted value range is from 1 to 26. In Version 9.0.0.7+ the default is 3.

Global.preprocessing-shard-entry-table

Shard entry table or view. If this value is not set, the CATENTRY table is processed as the default entry table.

Global.locale.unstructured-index-core-name

Name to be used to override the unstructured-index-core-name.

Global.locale.unstructured-index-core-dir

Directory to be used to override the unstructured-index-core-dir.

Global.locale.structured-index-core-name

Name to be used to override the structured-index-core-name.

Global.locale.structured-index-core-dir

If Global.auto-index is set to true, the Global.locale... configurations overwrite each shard's index-core-name and index-core-dir configurations in Horizontal Shard properties. locale is a supported locale, for example en_US or fr_FR. The above index-core-name and index-core-dir can automatically be appended with the shard name.

For example, if customer set number-of-shards to 3 with the following configurations:

 Global.en_US.unstructured-index-core-name=MC_10001_CatalogEntry_Unstructured_en_US
Global.en_US.unstructured-index-core-dir=/search/index/solr/MC_10001/en_US/Unstructured
Global.en_US.structured-index-core-name=MC_10001_CatalogEntry_en_US
Global.en_US.structured-index-core-dir=/search/index/solr/MC_10001/en_US/CatalogEntry

Then shards A, B, and C are created automatically. For the auto-index, the values are as follows.

Shard A's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_A. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_A/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_A, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_A/.
Shard B's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_B. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_B/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_B, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_B/.
Shard C's unstructured-index-core-name is MC_10001_CatalogEntry_Unstructured_en_US_C. Its unstructured-index-core-dir is /search/index/solr/MC_10001/en_US/Unstructured_C/. Its structured-index-core-name is MC_10001_CatalogEntry_en_US_C, and its structured-index-core-dir is /search/index/solr/MC_10001/en_US/CatalogEntry_C/.

The following list shows the global indexing properties:

Global.auto-index: Automatically start indexing shards after they are preprocessed.
Global.create-core: Create the shard core in the solr.xml file, if it does not exist.
Global.unload-core: Unload the shard core from the solr.xml file, if it does not exist.
Global.index-statusInterval: The time interval to monitor the indexing process. The default value is 10000.
Global.auto-merge-index: Automatically merge the index into the master server after all shard indexing is complete.

Master search server properties

There is typically only one set of master properties. They start with the keyword Master, followed by either the common keyword or the locale name (for example, en_US), followed by the property name, which is separated by a period.

The master properties must include at least the master-specific properties. Only the global server properties can be overwritten in the master properties. Otherwise, if not redefined, the global properties are used.

Note: The following properties use the en_US locale with shards A and C as a sample. Configure your sharding input properties file according to the locale and shard types that you are using.

Master.en_US.structured-index-core-name
Master.en_US.unstructured-index-core-name: The master index cores where all shard data is merged.

Horizontal Shard properties

There is typically a set of horizontal shard properties for each horizontal shard. The horizontal shard properties start with the Shard keyword, followed by the shard name, followed by either the common keyword or the locale name, followed by the property name, all separated by periods.

The horizontal shard properties must include at least the horizontal shard's specific properties. All global properties can be overwritten in the horizontal shard, except for auto-merge-index. If no properties are redefined, the global properties are used.

The following list shows the horizontal shard-specific properties, using shard A as a sample:

Shard.A.common.preprocessing-dir: The path to the preprocessing XML files in the HCL Commerce Utility Docker container. This property is common and cannot be overwritten.
Shard.A.common.preprocessing-start-range-value
Shard.A.common.preprocessing-end-range-value: The horizontal shard range start and end values.
Shard.A.common.configuration-type: The shard type to configure. For example, horizontal or vertical. When there is a combination of horizontal and vertical shard types, this property must be set so. In this case, horizontal.
Shard.A.en_US.unstructured-index-core-name
Shard.A.en_US.unstructured-index-core-dir
Shard.A.en_US.structured-index-core-name
Shard.A.en_US.structured-index-core-dir: The shard unstructured and structured core names and directories.

Vertical Shard properties

There is typically a set of vertical shard properties for each vertical shard. The vertical shard properties start with the Shard keyword, followed the shard name, either the common keyword or the locale name, followed by the property name, all separated by periods.

The vertical shard properties must include at least the vertical shard-specific properties. Only the global preprocessing properties can be overwritten in the vertical shard. Otherwise, if not redefined, the global properties are used.

The following list shows the vertical shard-specific properties, using shard V as a sample:

Shard.V.common.preprocessing-dir: The path to the preprocessing XML files in the HCL Commerce Utility Docker container. This property is common and cannot be overwritten.
Shard.V.common.configuration-type: The shard type to configure. For example, horizontal or vertical. When there is a combination of horizontal and vertical shard types, this property must be set so. In this case, vertical.

Overriding properties

To override a shard or master global property, use the following convention to overwrite it at the common level:


Master.common.index-server-name=localhost
Shard.A.common.index-server-name=localhost

To override a shard or master global property, use the following convention to overwrite it at the locale level:


Master.en_US.index-server-name=localhost
Shard.A.en_US.index-server-name=localhost

Sample properties files

The Utility Docker image contains two sample properties files. The example di-parallel-process.properties file uses the en_US locale with shards A, B, and C. These shard definitions are commented-out in favor of auto-sharding, but have been kept as examples because manual sharding is still supported.

You can find the sample files in the directory /opt/WebSphere/CommerceServer90/properties/parallelprocess/.

di-parallel-process.properties: The sample sharding input properties file for a Linux operating system that uses DB2. Refer to the latest di-parallel-process.properties configuration file in the HCL Commerce Utility docker container if you are using the Oracle database.
password.properties: The sample password properties file, referenced by the sample sharding input properties file. It contains passwords encrypted by the wcs_encrypt utility.