Creating and indexing shards
You can set up and index a specified number of sharding cores by defining them in an input properties file and running the parallel-process utility in the Utility server Docker container.
Before you begin
- Decide on the sharding configuration you want to use. For example, you can select a language-specific core that needs to be sharded.
- Decide on the number of shards to create.
About this task
During this task, you perform the following high-level steps:
- Create and populate the input properties file that is used to process indexing shards.
- Run the di-parallel-process utility to index the shards.
Procedure
Prepare the input properties file
-
Create or update the input properties file that will be used for indexing, based on your
environment and requirements.
The sharding input properties file is used by the di-parallel-process utility to process indexing shards. It contains the following sections of properties:
- System properties such as the password property file path.
- Database properties, which are used to establish database connections with the database server.
- Global preprocessing and indexing properties, which are used for preprocessing and indexing by all shards.
- Master search server properties, which specify the master index cores where all shard data is merged.
- Horizontal Shard properties, which specify the horizontal shard properties.
The Utility Docker image contains two sample properties files. The example di-parallel-process.properties file uses the en_US locale and contains commented-out definitions for shards A, B, C and V (V is for vertical sharding). You can find the following files in the directory /opt/WebSphere/CommerceServer90/properties/parallelprocess/.- di-parallel-process.properties
- The sample sharding input properties file for a Linux operating system that uses DB2. Refer to the latest di-parallel-process.properties configuration file in the HCL Commerce Utility docker container if you are using the Oracle database.
- password.properties
- The sample password properties file, referenced by the sample sharding input properties file. It contains passwords encrypted by the wcs_encrypt utility.
For more information about the properties file and expected values, see Sharding input properties file.
Run the shard indexing utility
-
Go to the following directory:
- WCDE_installdir\bin
- /opt/WebSphere/CommerceServer90/bin in the Utility server Docker container.
-
Run the utility:
- di-parallel-process.bat input_properties_file
- di-parallel-process.sh input_properties_file [-dbuser dbuser] [-dbuserpwd dbuserpwd] [-searchuser searchuser] [-searchuserpwd searchuserpwd]
Where:
- input_properties_file
- The path for the utility's input properties file can be either relative or absolute. For example, ../../di-parallel-process-linux-oracle.properties. The default path in the Utility server Docker container is /opt/WebSphere/CommerceServer90/properties/parallelprocess/di-parallel-process.properties.
- dbuser
- Optional: The name of the user who is connecting to the database.
- dbuserpwd
- Optional: The password for the user who is connecting to the database.
- searchuser
- Optional: The search application administrative user name.
- searchuserpwd
- Optional: The search application administrative user password.
-
Ensure that the utility runs successfully. Either check the exit code, or the
wc-dataimport-parallel-processor.log file for more information. The path for
wc-dataimport-parallel-processor.log is:
- WCDE_installdir\logs.
- /opt/WebSphere/CommerceServer90/logs/.
If you want the log to provide greater detail, set the log level to FINEST in the /opt/WebSphere/AppServer/profiles/default/installedApps/localhost/ts.ear/xml/config/dataimport/parallel-processor-logging.properties properties file.Tip: Permitted logging levels differ between development and runtime environments. The logging levels are:- For development environments, off, fatal, severe, warning, audit, info, config, detail, fine, finer, finest, all. For more information, see Configuring logging for the Search Server (Developer environment).
- Runtime environments use a subset of these levels: info, fine, finer, finest, all.
-
If you are using indexed contract prices, complete the following steps:
- Open the following file for all shards: WC_installdir/instances/instance_name/search/solr/home/MC_masterCatalogId/Shards/locale/CatalogEntry_#Shard-Tag#/conf/wc-data-config.xml.
- Find all instances of the following table name TI_CNTRPRICE _0_#Shard-Tag#.
- Replace them with the following table name: TI_CNTRPRICE _0. This update enables the utilities to look for contract prices in the TI_CNTRPRICE_0 table, instead of the shard-specific tables.
- Save your changes.
- For specific usage, see Indexing contract prices in HCL Commerce Search
What to do next
Replicate the merged index into the repeater, then to the other nodes.