Setting up the HCL Commerce Developer Search environment

HCL Commerce Version 9.1 provides support for Elasticsearch by default. Use the HCL Commerce Developer Search environment to make customizations to the search solution.

Note: If you are using the Solr-based search solution, use the HCL Commerce Developer Legacy environment to develop your Search customizations. For more information, see:

Before you begin

The HCL Commerce Developer Search environment requires the following before you can use it to develop your Elasticsearch-based search customizations:

Important: The Elasticsearch-based Search solution is resource intensive. It is recommended to dedicate standalone hardware or hosted virtual machine resources and not run it on a local development machine.
Before installing HCL Commerce Developer software, ensure that your HCL Commerce development environment meets the hardware and software requirements.
- For software requirements, see Supported software levels, fixes, and settings for HCL Commerce.
- For hardware requirements, see Hardware requirements for HCL Commerce.
The requires a database.
The supports and . For more information about installing and configuring a development environment database, see Install and configure your development environment database.
HCL Cache caches classes that can be modified in newer versions of HCL Commerce. To avoid errors in de-serializing an old version of the class, it is strongly recommended to clear Redis keys after upgrading HCL Commerce. Redis keys can be cleared with the Redis flushdb or flushall commands.

Note: When using Microsoft Windows:

Ensure that ample resources are allocated to Docker.
To allocate resources to Docker, right click on your Docker executable, or shortcut to it, and select Advanced. Allocate at minimum: 8 CPU cores, 32GB of memory, and 80GB of storage to ensure adequate performance.
Ensure that the following network setting is enabled within the Docker settings > General settings:
```
Expose daemon on tcp://localhost:2375 port
```

Procedure

Obtain the HCL Commerce Docker images.
To start the deployment process, download the required software.
1. Review the list of the latest available download packages to ensure that you are obtaining the most up-to-date version of HCL Commerce software.
2. Navigate to the .
3. Search for Package ID HCL_Commerce_Devops_Version_9.1, or File ID HCL_Commerce_DevOps_9.1.x.x.bundle.
4. Download the file.
5. Download the Data Docker images.
  - HCL_Commerce_Enterprise_9.1.x.x_Data_Query_Server_x86-64.tgz
  - HCL_Commerce_Enterprise_9.1.x.x_Data_NiFi_Server_x86-64.tgz
  - HCL_Commerce_Enterprise_9.1.x.x_Data_NiFi_Registry_Server_x86-64.tgz
  - HCL_Commerce_Enterprise_9.1.x.x_Data_Ingest_Server_x86-64.tgz
  - Optional:HCL_Commerce_Enterprise_9.1.x.x_Approval_Server_x86-64.tgz
  - Optional:HCL_Commerce_Enterprise_9.1.x.x_GraphQL_Server_x86-64.tgz
  Load each Docker image. For each downloaded image file, run the following command.
```
docker load -i imagename
```
  For example:
```
docker load -i HCL_Commerce_Enterprise_9.1.0.0_Data_Query_Server_x86-64.tgz
```
6. Clone the project in Git from the downloaded git bundle, and check the code into your source control management system. Run the following command.
```
git clone bundleName projectName
```
  Where:
  
  bundleName
  
  The filename of the bundle you are cloning.
  
  projectName
  
  The name of the git project that you are creating.
  
  For example:
```
git clone HCL_Commerce_DevOps_9.1.0.0.bundle HCL_Commerce_DevOps
```
Deploy the Elasticsearch-based Search solution.
- Deployment option #1: Deploy the Elasticsearch-based Search solution via the deployment script.
  This option is limited to deployment within a Linux environment, due to the reliance on Bash scripts (.sh files).
  1. Update the env.sh bash script with the details of your environment.
    1. Open env.sh for editing.
    2. Set the following environment details depending on your version of .
      - For 9.1.12.0 and greater, the following values must be set:
        SEARCH_ENGINE=elasticsearch DEPLOY_TYPE=toolkit LICENSE_ACCEPTED=accept TOOLKIT_HOST=<toolkitHost> TOOLKIT_HOST_IP=<toolkitHostIP> SEARCH_HOST=<searchHost> SEARCH_HOST_IP=<searchHostIP> DB_TYPE=<db2|oracle> DB_HOST=<databaseHost> DB_NAME=<databaseName> DB_USER=<databaseUser> DB_PASS=<databaseUserPassword> SEARCH_QUERY_IMAGE=<searchQueryImage> SEARCH_NIFI_IMAGE=<searchNifiImage> SEARCH_REGISTRY_IMAGE=<searchRegistryImage> SEARCH_INGEST_IMAGE=<searchIngestImage> SPIUSER_PWD_ENCRYPTED=<encryptedSpiUserPassord> ADMIN_SPIUSER_PWD=<plainTextSpiuserPassword> SPIUSER_PWD_BASE64=<base46SpiuserPassword>
      - For 9.1.12.0 and greater, optionally set the following configurations:
        APPROVAL_ENABLED=<true|false> APPROVAL_APP_IMAGE=<approvalImage> SESSION_KEY_ENCRYPT=<encryptedSessionKey> GRAPHQL_ENABLED=<true|false> GRAPHQL_APP_IMAGE=<graphqlImage> ALLOW_TELEMETRY=<yes|no> CACHE_ENABLED=<true|false> CACHE_APP_IMAGE=<cacheManagerImage>
      - For 9.1.0.0 to 9.1.11.0:
        SEARCH_ENGINE=elasticsearch DEPLOY_TYPE=data DATA_FOR_TOOLKIT=true LICENSE_ACCEPTED=accept TOOLKIT_HOST=<toolkitHost> DB_TYPE=<db2|oracle> DB_HOST=<databaseHost> DB_NAME=<databaseName> DB_USER=<databaseUser> DB_PASS=<databaseUserPassword> SEARCH_QUERY_IMAGE=<searchQueryImage> SEARCH_NIFI_IMAGE=<searchNifiImage> SEARCH_REGISTRY_IMAGE=<searchRegistryImage> SEARCH_INGEST_IMAGE=<searchIngestImage> SPIUSER_PWD_ENCRYPTED=<encryptedSpiUserPassord> ADMIN_SPIUSER_PWD=<plainTextSpiuserPassword> SPIUSER_PWD_BASE64=<base46SpiuserPassword>
      Note:
      
      For more information on the env.conf configuration file, review The Docker Compose deployment env.sh configuration file.
    3. Save and close the file.
    4. Back up the env.sh script containing your environment details for future deployment.
  2. If you are using an database, ensure that there is a copy of the required JDBC driver (ojdbc8.jar) located within the volumes/jdbc/ directory.
  3. Run the deployment bash script.
```
./deploy.sh
```
    This script produces an Elasticsearch-based Search Docker Compose file that is based on the environment configuration in env.sh.
    - For 9.1.12.0 and greater, this file is docker-compose-toolkit.yml.
    - For 9.1.0.0 to 9.1.11.0, this file is docker-compose-data.yml.
    The deployment will set the project to commerce, so the created Docker services will have commerce_ prefix. For example, commerce_nifi_1.
    The deployment script will automatically start the deployment.
- Deployment option #2: Deploy the platform manually.
  This option is the only deployment that is compatible with a Windows host operating system, due to its non-reliance on Bash scripts (.sh files).
  1. Create a Docker Compose deployment file for your environment.
    1. Select a deployment Docker Compose template file based on the database that you will use for the deployment.
      The template files are follow the naming convention: docker-compose-elasticsearch-data-<DB_TYPE>-toolkit-template.yml.
      Where <DB_TYPE> can be db2 or oracle.
    2. Copy the template into your Docker Compose deployment file (docker-compose-toolkit.yml).
    3. Add optional services to your Docker Compose deployment file (docker-compose-toolkit.yml).
      - GraphQL (docker-compose-elasticsearch-svc-graphql-toolkit-template.yml)
      - Cache Manager (docker-compose-elasticsearch-data-toolkit-cache-app-svc-template.yaml)
    4. ( docker-compose-svc-approval-toolkit-template.yml)
      Note: In addition to including the content from this Docker Compose template, to enable the you must overwrite volumes/approval-app/boot-config/app-secure.properties with the contents of volumes/approval-app/boot-config/app-secure.properties.nonssl.
  2. Update the env.sh bash script with the details of your environment.
    1. Open env.sh for editing.
    2. Set the following environment details depending on your version of .
      - For 9.1.12.0 and greater, the following values must be set:
        SEARCH_ENGINE=elasticsearch DEPLOY_TYPE=toolkit LICENSE_ACCEPTED=accept TOOLKIT_HOST=<toolkitHost> TOOLKIT_HOST_IP=<toolkitHostIP> SEARCH_HOST=<searchHost> SEARCH_HOST_IP=<searchHostIP> DB_TYPE=<db2|oracle> DB_HOST=<databaseHost> DB_NAME=<databaseName> DB_USER=<databaseUser> DB_PASS=<databaseUserPassword> SEARCH_QUERY_IMAGE=<searchQueryImage> SEARCH_NIFI_IMAGE=<searchNifiImage> SEARCH_REGISTRY_IMAGE=<searchRegistryImage> SEARCH_INGEST_IMAGE=<searchIngestImage> SPIUSER_PWD_ENCRYPTED=<encryptedSpiUserPassord> ADMIN_SPIUSER_PWD=<plainTextSpiuserPassword> SPIUSER_PWD_BASE64=<base46SpiuserPassword>
      - For 9.1.12.0 and greater, optionally set the following configurations:
        APPROVAL_ENABLED=<true|false> APPROVAL_APP_IMAGE=<approvalImage> SESSION_KEY_ENCRYPT=<encryptedSessionKey> GRAPHQL_ENABLED=<true|false> GRAPHQL_APP_IMAGE=<graphqlImage> ALLOW_TELEMETRY=<yes|no> CACHE_ENABLED=<true|false> CACHE_APP_IMAGE=<cacheManagerImage>
      - For 9.1.0.0 to 9.1.11.0:
        SEARCH_ENGINE=elasticsearch DEPLOY_TYPE=data DATA_FOR_TOOLKIT=true LICENSE_ACCEPTED=accept TOOLKIT_HOST=<toolkitHost> DB_TYPE=<db2|oracle> DB_HOST=<databaseHost> DB_NAME=<databaseName> DB_USER=<databaseUser> DB_PASS=<databaseUserPassword> SEARCH_QUERY_IMAGE=<searchQueryImage> SEARCH_NIFI_IMAGE=<searchNifiImage> SEARCH_REGISTRY_IMAGE=<searchRegistryImage> SEARCH_INGEST_IMAGE=<searchIngestImage> SPIUSER_PWD_ENCRYPTED=<encryptedSpiUserPassord> ADMIN_SPIUSER_PWD=<plainTextSpiuserPassword> SPIUSER_PWD_BASE64=<base46SpiuserPassword>
      Note:
      
      For more information on the env.conf configuration file, review The Docker Compose deployment env.sh configuration file.
    3. Save and close the file.
    4. Back up the env.sh script containing your environment details for future deployment.
  3. Copy your configured env.sh values into the Docker .env file on the same level as your Docker Compose file (docker-compose-toolkit.yml). This allows for docker-compose to use these environment variables in the deployment.
  4. Set file permission for your persistent volumes.
    Some volumes are mounted to your running containers and require write permissions.
    Ensure that you recursively set write access to /volumes/, to ensure that it and all of the contained sub-directories are writable.
  5. Run the following command to start the Elasticsearch Search solution.
```
docker-compose up -d
```
- Legacy deployment option: Deploy a legacy version of the platform manually.
  In the event that you need to support an older version of 9.1, this option is the only method available for deploying its corresponding Elasticsearch-based solution.
  - This deployment option is no longer supported with 9.1.6.0 and greater.
  - This is the only deployment option for versions 9.1.0.0 through 9.1.5.0.
  1. From the docker-compose folder, copy the file docker-compose-elasticSearch-data-db2-toolkit-template.yml or docker-compose-elasticSearch-data-oracle-toolkit-template.yml to docker-compose.yml.
  2. Update the .yml image tag for all HCL Commerce Docker containers.
    1. Determine the image repository and tag by running the Docker images.
    2. Update each image tag in the .yml file with the its respective repository:tag. For example, image: commerce/search-nifi-app:9.1.0.0
  3. Accept the LICENSE parameter for all docker containers, for example: - LICENSE=accept
  4. Update the following configuration for Nifi to point to the database.
```
  AUTH_JDBC_URL: "jdbc:DBType://DBHost:50000/mall"
  AUTH_JDBC_DRIVER_LOCATION: "/opt/nifi/nifi-current/lib/DBDriver"
  AUTH_JDBC_USER_NAME: "DBUser"
  AUTH_JDBC_USER_PASSWORD: "DBUserPassword"
  NLP_ENABLE_LANGUAGE_CODE: "nlpEnableLanguageCode"
```
    For example:
```
AUTH_JDBC_URL: "jdbc:db2://tk.hclcommerce.com:50000/mall"
  AUTH_JDBC_DRIVER_LOCATION: "/opt/nifi/nifi-current/lib/db2jcc4.jar"
  AUTH_JDBC_USER_NAME: "wcs"
  AUTH_JDBC_USER_PASSWORD: "wcs1"
  NLP_ENABLE_LANGUAGE_CODE: "EN"
```
```
AUTH_JDBC_URL: "jdbc:oracle:thin:@//tk.hclcommerce.com:1521/orclpdb1"
AUTH_JDBC_DRIVER_CLASSNAME: "oracle.jdbc.driver.OracleDriver"
AUTH_JDBC_DRIVER_LOCATION: "/opt/nifi/nifi-current/lib/ojdbc8.jar"
AUTH_JDBC_USER_NAME: "wcs"
AUTH_JDBC_USER_PASSWORD: "wcs1"
NLP_ENABLE_LANGUAGE_CODE: "EN"
```
  5. Update the configuration for data-query.
```
  TX_HOST: "TXHost"
```
    For example:
```
TX_HOST: "tk.hclcommerce.com"
```
  6. Create the following sub-directories within in the same directory where docker-compose.yml is located, and ensure that their permissions are set to 777.
    Important: If you are upgrading from 9.1.0.0 or 9.1.1.0 to a newer version of , or if you have any issues related to the search index or NiFi service connector, you must delete the contents of these directories.
    - nifiLogs
    - elasticsearch/data
    - elasticsearch/config
    - zookeeper
    - nifi/data
    - redis/data
  7. Copy the hcl-cache-search folder to the location where the Docker Compose file (docker-compose.yml) is located.
  8. If you are using an or database, ensure that there is a copy of the required JDBC driver (ojdbc8.jar) in the same directory where the Docker compose file (docker-compose.yml) is located.
  9. Run the following command to start the Elasticsearch Search solution.
```
docker-compose up -d
```
Verify that all the containers are up and healthy.
Set your HCL Commerce development environment Elasticsearch environment variables.

Note: If you want to change back to the Solr search engine at a later data, make a backup of jndi.xml under Liberty_Dir\usr\servers\crsServer\configDropins\overrides\, and namebindings.xml under WCDE_InstallDir\wasprofile\config\cells\localhost\nodes\localhost\servers\server1\. Then, you can revert to Solr with your Elasticsearch settings backed up.
1. Navigate to the WCDE_InstallDir/bin directory.
2. Run the following command script:
```
configureElasticSearch.bat elasticSearchDataQueryHost elasticSearchDataQueryPort elasticSearchIngestHost elasticSearchIngestPort elasticSearchHost elasticSearchPort elasticSearchScheme redisHost redisPort toolingSDKInstallDir reactStoreSDKInstallDir
```
  Where:
  
  elasticSearchDataQueryHost
  
  The Elasticsearch query hostname or IP address.
  
  elasticSearchDataQueryPort
  
  The Elasticsearch query port number.
  
  elasticSearchIngestHost
  
  The Elasticsearch ingest hostname or IP address.
  
  elasticSearchIngestPort
  
  The Elasticsearch ingest port number.
  elasticSearchHost
  
  The Elasticsearch hostname or IP address.
  
  elasticSearchPort
  
  The Elasticsearch port number.
  
  elasticSearchScheme
  
  The Elasticsearch URL scheme. Values: http or https.
  
  RedisHost
  
  The Redis hostname or IP address.
  
  RedisPort
  
  The Redis port number.
  
  toolingSDKInstallDir
  
  The local installation directory of the HCL Commerce Developer Tooling environment Software Development Kit.
  Note: If the toolingSDKInstallDir path contains any spaces, ensure that it is fully encapsulated within double quotation marks ("").
  
  reactStoreSDKInstallDir
  Optional: The local installation directory of the HCL Commerce Developer React Store environment Software Development Kit.
  Note:
  
  The reactStoreSDKInstallDir parameter is optional.
  
  If the reactStoreSDKInstallDir path contains any spaces, ensure that it is fully encapsulated within double quotation marks ("").
  For example:
```
configureElasticSearch.bat elasticsearch.commerce.com 30921 elasticsearch.commerce.com 30801 elasticsearch.commerce.com 30200 http elasticsearch.commerce.com 6379 C:\git-src\github01\toolingsdk C:\git-src\github01\store-web
```
  Important: Review Troubleshooting: Ingest indexing fails due to certificate issue if you encounter Ingest indexing failures when upgrading to HCL Commerce Developer 9.1.17.0.
Optional: Enable the HCL Commerce Cache Manager application.

Note: For releases of HCL Commerce 9.1.12.0 and greater, the Cache Manager application can be deployed more easily by updating the env.sh file with CACHE_ENABLED=true and then specifying the CACHE_APP_IMAGE.
If you are upgrading from HCL Commerce 9.1.11.0 to 9.1.12.0 or greater, you must update the database to launch the Emerald or Sapphire marketplace.
In a database command prompt, run the following SQL.
```
insert into storeconf values(32,'hcl.marketplace.self-registration','true',1);
insert into storeconf values(31,'hcl.marketplace.self-registration','true',1);
```

Results

Your HCL Commerce development environment is now configured to develop Elasticsearch-based search customizations.

What to do next

Build your Elasticsearch search index. To build your search index, see Building the Elasticsearch index.