Migrating REST-based Search to HCL Commerce Version 9

Migrate your REST-based Search index and configurations from a previous version of HCL Commerce to HCL Commerce version 9.0.0.3+.

HCL Commerce version 9.0.0.3+ uses Solr 5.4.4, so your index data from an earlier version of Solr is unsupported by Solr 5.4.4.

Review the following information to understand what HCL Commerce Search architecture and functionality has changed in HCL Commerce version 9.0.0.3+.

The HCL Commerce Search server has its own container within your production environment. You deploy your HCL Commerce Search server as part of your CI/CD pipeline.
The programming model for HCL Commerce Search is changed to coincide with the new build and deployment process in HCL Commerce version 9.0.0.3+. The basis of the new programming model is to separate custom HCL Commerce Search assets and configuration settings from the product code, which reduces the resource cost of maintenance and operation. The following customizations must be updated for the new programming model:
- Solr runtime is upgraded to 5.5.4, so any customizations to Solr must be updated to follow new programming model.
- HCL Commerce Search utilities are replaced by the container utility service, which includes di-preprocess, di-buildindex, di-calculateprice, and indexprop. The SetupSearchIndex utility is discontinued. The index core directory is now automatically synchronized with the SRCHCONF table and the SRCHCONFEXT table when the HCL Commerce Search server is started. To create a new master core, extension core, or language you must maintain the SRCHCONF and SRCHCONFEXT tables. The workspace core is created automatically if the HCL Commerce Search server detects that the workspace schema is on an authoring environment.
- In HCL Commerce version 9.0.0.3+, the table view is used for preprocessing and index building, so any customizations to preprocessing and index building must be reconfigured according to the new programming guide.
- In HCL Commerce version 9.0.0.3+, the common foundation-based scheduler is enabled on the HCL Commerce Search server. Authoring environments use the scheduler to replicate indexes from authoring environments to the HCL Commerce Search repeater.
- HCL Commerce version 9.0.0.3+ moved to JAX-RS 2.0(JSR-339). Also, the documentation API is Swagger 2.0.
- Previous versions of HCL Commerce used direct JDBC calls, which went through DSL(data service layer) to the database. In HCL Commerce version 9.0.0.3+, JPA 2.1 (EclipseLink) native query is used. Custom queries from previous versions are ported over to the new query service. No additional configure is required.
- In HCL Commerce version 9.0.0.3+, when Price or Inventory are worked as extended core, SolrJoin preserves the document relationship between the main CatalogEntry core and the Price/Inventory subcore. MultipleQueryComponent and MultipleFacetComponent, which were used to join or filter the subcores in previous versions of HCL Commerce, are now discontinued. To handle facet and result fields from extension indexes, a new SearchCatalogEntryExtensionIndexPostprocessor makes a subquery against each of the extension indexes, then joins back with the main index. A new join parameter was also introduced in wc-search.xml, so any previous customization to an extension index needs to be implemented to use SolrJoin.

Before you begin

Drop all your temporary tables, including your custom temporary tables, but exclude the following temporary tables:
- TI_DELTA_CATENTRY
- TI_DELTA_CATGROUP
- TI_DELTA_INVENTORY
Your temporary tables use a TI_ prefix. Whereas, your custom temporary tables use a XI_ prefix.
Changes were made to the temporary tables between the previous versions of HCL Commerce and HCL Commerce Version 9. Failure to drop the temporary tables might result in preprocess errors, for example, SQLSTAE=56098. For more information about temporary HCL Commerce Search tables, see Temporary table schema definition.

Procedure

Migrate your index cores. In HCL Commerce version 9.0.0.3+, the setupSearchIndex utility is discontinued. Now, you must maintain your SRCHCONF table and SRCHCONFEXT table to set up your index distribution.
1. Migrate the main CatalogEntry / CatalogGroup core by updating your SRCHCONF table by running the following SQLs.
```
Update srchconf set config='IndexScopeTag=0,createAll=true,PreProcessConfigDirectory=/v3/db2/CatalogEntry,Version=3' where indextype='CatalogEntry'
```
```
update srchconf set config='IndexScopeTag=1,createAll=true,PreProcessConfigDirectory=/v3/db2/CatalogGroup,Version=3' where indextype='CatalogGroup’
```
  Notes:
  - IndexScopeTag is an internal tag that is used to mark the temporary table, which is generated during preprocess. If you are migrating multiple master catalogs, you must incrementally increase this value for each entry you add.
  - createAll is used to determine whether to create a workspace counterpart core for this specific index. Setting this value to true creates the workspace counterpart.
  Rerun the previous SQLs for every catalog entry and catalog group that you want to migrate to HCL Commerce version 9.0.0.3+.
2. If you need to migrate price or inventory subcores, you must register the subcores into the SRCHCONFEXT table. Use the following SQL as an example on how to register a core to your SRCHCONFEXT table.
```
insert into srchconfext(srchconfext_id,indextype,indexscope,indexsubtype,config) values(-1,'CatalogEntry','3074457345616676668','Price','')
```
  Note: After you register your subcores, you must run separate build index to build your subcores. Complete this build index after the master core is built. For more information about building subcores, see Building the HCL Commerce Search index.
3. Migrate your languages. In previous HCL Commerce versions, different languages used separate schema definitions. Even though different languages still use different index data in HCL Commerce version 9.0.0.3+, languages now share a schema definition. Registers your languages in your SRCHCONF table by using the following SQL example.
  If you used en_US, zh_CN, and fr_FR languages, you run the following SQL.
```
Update srhconf set languages=’-1,-2,-7’ where indextype=’CatalogEntry’ and indexscope=’10001’
```
4. Restart your HCL Commerce Search server. As a result, the HCL Commerce Search server creates your cores and subcores during start-up.
Migrate your index schema customizations. In HCL Commerce version 9.0.0.3+, the catalog entry fieldType definitions use two templates:
- A non-customizable template: search-config/src/main/resources/managed-solr/config/v3/common/schema-field-types.xml.
  Note: When the first index is created, this XML file is copied to the resources/search/index/managed-solr/config/v3/common directory. After the index creation, other indexes share this fieldType definition.
- A customizable template: search-config-ext/src/main/resources/index/managed-solr/config/v3/common/x-schema-field-types.xml.
  Note: When the first index is created, this XML file is copied to resources/search/index/managed-solr/config/v3-index-ext/common directory. After the index creation, other indexes share this customizable fieldType definition.
1. Migrate your custom schema fieldType by using the following example.
  If you created a new field Type x_textSpell for the fr_FR locale in a previous HCL Commerce version, you updated the schema.xml file, which was located under core configuration directory specific to the fr_FR locale. In HCL Commerce version 9.0.0.3+, you must add the field type x_textSpell_fr to the file x-schema-field-types.xml as follows.
```

<fieldType omitNorms="true" positionIncrementGap="100" class="solr.TextField" name="x_textSpell_fr">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory" preserveOriginal="0" splitOnNumerics="1" splitOnCaseChange="1" 
    catenateAll="0" catenateNumbers="0" catenateWords="0" generateNumberParts="1" generateWordParts="1"/>
<filter class="solr.ShingleFilterFactory" fillerToken="" tokenSeparator=" " maxShingleSize="3" minShingleSize="2" 
    outputUnigrams="true"/>
<filter class="solr.PatternReplaceFilterFactory" replace="all" replacement=" " pattern="\s{2,}"/>
<filter class="solr.TrimFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>

<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
</analyzer>
</fieldType>
```
2. After the new x_textSpell_fr field type is defined in the x-schema-field-types.xml file, define an entry into your x-schema.xml file by using the following example.
```
<field name="spellCheck" type="x_textSpell_fr" indexed="true" stored="false" multiValued="true" />
```
  Notes:
  - If you want to all support languages, you must create a field type for every language in the x-schema-field-types.xml, then add following field definition into x-schema.xml file.
    <field name="spellCheck" type="x_textSpell_${lang:en}" indexed="true" stored="false" multiValued="true" />
Migrate your Solr supported file assets.
1. Migrate your customized stop words.
  In previous versions of HCL Commerce, you could change the stopwords.txt file for each core configuration directory. For example, if you customized stopwords.txt for the English language, you need only to change the stopwords.txt file under the en_US core. When you migrate to HCL Commerce version 9.0.0.3+, the index schemas are shared among multiple languages. In order to customize stopwords.txt for a specific language, HCL Commerce version 9.0.0.3+ requires that you point to the file in the SRCHCONFEXT database table. Run the following SQL command to put a stopwords.txt into the config field:
```
Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt' 
where indextype=’CatalogEntry’ and indexscope=’$MASTERCATALOGID’
```
  When the HCL Commerce Search server is restarted, the following contents are added into x-core.properties under the en_US index data directory.
```
stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt
```
  If you need to change the stopwords.txt that is under workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common directory, the new stopwords.txt will be copied to the Liberty_installdir\resources\search\index\managed-solr\config\v3-index-ext\common directory as part of setting up your WCB and CI/CD pipeline.
  For example, in previous versions of HCL Commerce, you added a new English stop word: can. The following steps show you how to migrate this customization.
  1. Add can in the bottom of your workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common\stopwords.txt file on HCL Commerce version 9.0.0.3+.
  2. Add customized stopwords.txt path in CONFIG column of SRCHCONFEXT table, run sql:
    Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-index-ext/common/stopwords.txt' where indextype='CatalogEntry' and indexscope='10001';
2. Migrate your customized pre-process files.
  In previous versions of HCL Commerce, when you create a customized preprocess.xml file to index external data, the customized file is under the solrhome/pre-processConfig/MC_$MasterCatalog/Dbtype directory. In HCL Commerce version 9.0.0.3+, such pre-process XML files reside under the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory.
  Copy the wc-dataimport-preprocess-custom.xml file from your previous environment to the WCDE_installdir/xml/search/dataImport/v3/$Dbtype/ directory on HCL Commerce version 9.0.0.3+. Also copy any files that are referenced by the wc-dataimport-preprocess-custom.xml file to HCL Commerce version 9.0.0.3+, and update the wc-dataimport-preprocess-custom.xml file to ensure that it points to correct path file if you change the file location on HCL Commerce version 9.0.0.3+. For example, if you specify an input file in the wc-dataimport-preprocess-custom.xml,
```

<_config:property name="inputFile" value="WCDE_installdir\bin\Ratings.xml"/>
```
  Then you must copy your old Ratings.xml file to HCL Commerce version 9.0.0.3+, and put it under location WCDE_installdir\bin, and update wc-dataimport-preprocess-custom.xml to point to the new location.
```

<_config:property name="inputFile" value="WCDE_installdir\bin\Ratings.xml"/>
```
3. Migrate your customized data import handler files.
  If you want to index more data in a previous version of HCL Commerce, you change the wc-data-config.xml file in the specific core directory. For example, if you want to index more data from the X_RATING table in the English language, you edit wc-data-config.xml under the en_US core directory to allow query and deltaImportQuery to join the X_RATING table. In HCL Commerce version 9.0.0.3+, the Solr DataImportHandler (DIH) gets data from the view VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# and X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag#. Add this new customizable table into the view X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# in wc-dataimport-preprocss-x-finalbuild.xml. For more information, see Configuring the Data Import Handler mapping.
  
  After the customized tables are added into X_VI_CE table, during DIH, customized data can be returned and accessible to Solr. Define the field-column mapping so that Solr can transfer the new customized column into Solr fields. Update the search-config-ext/src/main/resources/index/managed-solr/config/v3/CatalogEntry/x-data-config.xml file to add the field mapping.
  
  The following is an example of the new mapping. With this mapping, the RATING column from the X_VI_CE view maps to the customerRanking field in the catalog entry index.
```
<field column="RATING" name="customerRanking"/>
```
4. Migrate the SRCHATTRPROP table to enable price range queries.
  In HCL Commerce version 7.0, the default price range facet takes the following format.
```
"price_USD:{* 100} 100;{100 200} 200;{200 300} 300;{300 400} 400;{400 500} 500;{500 *}"
```
  In Solr version 7.3.1, this format causes a syntax error in the query parser. If you are using HCL Commerce version 9.0.0.5+, change the query string into the following format.
```
 "price_USD:{* TO 100];{100 TO 200];{200 TO 300];{300 TO 400];{400 TO 500];{500 TO *}" 
```
  Note: Convert previous range query formats that take the form "({lower upper} upper)" into "({lower TO upper])". Migrate any other customization that involves the old query format to the new one.
  For example, the following JSP files that are used to display price ranges in the facet navigation area must be modified in order to consume the new format, and display lower and upper price range properly.
  1. Open following two files for editing.
    
    Stores/WebContent/Widgets_701/Common/Facet/PriceFacetDisplay.jspf
    
    Stores/WebContent/Widgets_801/Common/Facet/PriceFacetDisplay.jspf
  2. In each of the files, locate the following lines.
    <c:set var="rangeLabel" value="${fn:replace(item.label,'({','')}" /> <c:set var="rangeLabel" value="${fn:replace(rangeLabel,'})','')}" /> <c:set var="rangeLabel" value="${fn:replace(rangeLabel,'}','')}" /> <c:set var="rangeLow" value="${fn:substringBefore(rangeLabel,' ')}" /> <c:set var="rangeHigh" value="${fn:substringAfter(rangeLabel,' ')}" />
  3. Replace the lines above with the following.
    <c:set var="rangeLabel" value="${fn:replace(item.label,'({','')}" /> <c:set var="rangeLabel" value="${fn:replace(rangeLabel,'])','')}" /> <c:set var="rangeLabel" value="${fn:replace(rangeLabel,')','')}" /> <c:set var="rangeLabel" value="${fn:replace(rangeLabel,'}','')}" /> <c:set var="rangeLow" value="${fn:substringBefore(rangeLabel,' TO ')}" /> <c:set var="rangeHigh" value="${fn:substringAfter(rangeLabel,' TO ')}" />
  4. Save and close the files.
5. Migrate the default schema field types.
  Starting with Solr version 7.0.0, the Trie*Field fields are deprecated. Replace them with *PointField. The default setting retains the old data type fields (for example, int, tint, sint) and creates new fields (for example, pint, pints). Although the old fields still function, for future compatibility upgrade the old data type to the new one. Some deprecated fields are still used, for example, protected field types, for compatible considerations.
6. Migrate any customized solrconfig.xml parameters.
  For Solr version 7.3.1, move the configuration parameter solr.mergeFactor in the solrconfig.xml file into the SRCHCONFEXT.CONFIG column. It is replaced by two parameters: solr.mergePolicy.maxMergeAtOnce and solr.mergePolicy.segmentsPerTier. If you previously set the value to something like <mergeFactor>5</mergeFactor>, replace it with solr.mergePolicy.maxMergeAtOnce=5,solr.mergePolicy.segmentsPerTier=5.
Migrate your HCL Commerce Search runtime configuration.
1. Migrate the UserData property name mapping.
  In previous versions of HCL Commerce, when you want to enable some specific feature, such as result grouping or searchBasedSuggestion, you edit the Search.ear/xml/config/com.ibm.commerce.catalog/wc-component.xml file. When you add more data into the index at run time, you also add new field mappings into wc-component.xml. Here is an example of a field1 mapping in the wc-component.xml file:
```
<_config:valuemappingservice>  
    <_config:valuemapping externalName="CatalogEntryUserDataFieldNameMapping" 
       internalName="CatalogEntryUserDataFieldNameMapping">
      <_config:valuemap externalValue="x_customerRanking" internalValue="customerRanking_display"/>
    </_config:valuemapping>
  </_config:valuemappingservice>
```
  In HCL Commerce version 9.0.0.3+, you add such configuration information into the Workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-component.xml file. For more information, see Changing properties in the HCL Commerce configuration file (wc-component.xml).
  Note: Except for the result group, all other features need only to be updated in the HCL Commerce Search server component configuration file.
2. Migrate the HCL Commerce Search configuration.
  In order to create or change an HCL Commerce Search profile in previous versions of HCL Commerce, you configure the Search.ear/xml/config/com.ibm.commerce.catalog-ext/wc-search.xml file. In HCL Commerce version 9.0.0.3+, you update the Toolkit_Workspace/search-config-ext/runtime.config/com.ibm.commerce.search/wc-search.xml file. For example, previous versions of HCL Commerce have the following customized HCL Commerce search profiles.
```
<_config:profile name="IBM_findProductsBySearchTerm" indexName="CatalogEntry">
		<_config:query>
			<_config:postprocessor
				classname="com.ibm.commerce.foundation.server.services.rest.search.postprocessor.solr.SolrRESTSearchCatalogEntryViewUserDataQueryPostprocessor" />
		</_config:query>
		<_config:result>
			<_config:field name="customerRanking_display" />
		</_config:result>            
	</_config:profile>
	
	<_config:profile name="IBM_findProductsByCategory" indexName="CatalogEntry">   	
    	<_config:query> 
    	<_config:provider classname="com.mycompany.search.internal.expression.provider.CustomizedExpressionProvider"/> 
    	<_config:postprocessor classname="com.mycompany.search.internal.expression.postprocessor.CustomizedGiftPostprocessor" />  
        </_config:query>        
    </_config:profile>
    
    <_config:profile name="IBM_findProductByIds_Details" indexName="CatalogEntry"> 	
    	<_config:query> 
    	<_config:postprocessor classname="com.mycompany.search.internal.expression.postprocessor.CustomizedGiftPostprocessor" />  
        </_config:query>          
    </_config:profile>
```
  In HCL Commerce version 9.0.0.3+, you add this configuration information into the Workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-component.xml file. For instructions on configuring this file, see Changing properties in the HCL Commerce Search configuration file (wc-search.xml).
  
  After you run the WCB process and pipeline deployment, the runtime configuration files are deployed into Liberty_installdir/resources/search/runtime/config/com.ibm.commerce.search-ext directory.
3. Disable the Search server callback on the Transaction server to enable support for the Store Preview feature in your migrated environment.
  1. Open the workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-component.xml configuration file for editing.
  2. Locate the <_config:extendedconfiguration> section.
  3. Within the extendedconfiguration section, add the following configuration:
```
<_config:configgrouping name="TransactionServerSetting">
      <_config:property name="TransactionServerSetting/SessionManagement/allowPreviewContextCallback" value="false"/>
</_config:configgrouping>
```
In previous versions of HCL Commerce, you might create an SQL query in the Search_eardir/xml/config/com.ibm.commerce.catalog/wc-query-utilities.tpl file and rely on the JDBCQueryService to retrieve data from the HCL Commerce Search server. In HCL Commerce version 9.0.0.3+, you must update the workspace_dir/search-config-ext/runtime.config/com.ibm.commerce.search/wc-query.tpl file. Later, when you deploy your customizations, the .tpl file is deployed to the Liberty_installdir/resources/search/runtime/config/com.ibm.commerce.search-ext directory.
In previous versions, you might create custom expression providers, preprocessors, or postprocessors under the WebSphereCommerceServerExtensionsLogic project. In HCL Commerce version 9.0.0.3+, you must create these customizations in the workspace_dir/search-logic-ext directory.

What to do next

The next step in the migration process is to build and deploy your custom containers. After those containers are deployed, you can build your index. For more information about building your index, see Building the HCL Commerce Search index.