Migrating IBM Websphere Commerce Version 7 Feature Pack 6 BOD-based Search

Migrate your HCL Commerce Search index and configurations from IBM Websphere Commerce Version 7 Feature Pack 6 to HCL Commerce version 9.0.0.3+.

HCL Commerce Version 9 uses Solr 5.5.4, so your index data from an earlier version of Solr is unsupported by Solr 5.5.4.

Review the following information to understand what HCL Commerce Search architecture and functionality has changed in HCL Commerce Version 9.
  • BOD-base search is discontinued in HCL Commerce Version 9.
  • The HCL Commerce Search server has its own container within your production environment. You deploy your HCL Commerce Search server as part of your CI/CD pipeline.
  • The programming model for HCL Commerce Search is changed to coincide with the new build and deployment process in HCL Commerce Version 9. The basis of the new programming model is to separate custom HCL Commerce Search assets and configuration settings from the product code, which reduces the resource cost of maintenance and operation. The following IBM Websphere Commerce Version 7 customizations must be updated for the new programming model:
    • Solr runtime is upgraded to 5.5.4, so any customizations to Solr must be updated to follow new programming model.
    • HCL Commerce Search utilities are replaced by the container utility service, which includes di-preprocess, di-buildindex, di-calculateprice, and indexprop. The SetupSearchIndex utility is discontinued. The index core directory is now automatically synchronized with the SRCHCONF table and the SRCHCONFEXT table when the HCL Commerce Search server is started. To create a new master core, extension core, or language you must maintain the SRCHCONF and SRCHCONFEXT tables. The workspace core is created automatically if the HCL Commerce Search server detects that the workspace schema is on an authoring environment.
    • In HCL Commerce Version 9, the table view is used for preprocessing and index building, so any customizations to preprocessing and index building must be reconfigured according to the new programming guide.
    • In HCL Commerce Version 9, the common foundation-based scheduler is enabled on the HCL Commerce Search server. Authoring environments use the scheduler to replicate indexes from authoring environments to the HCL Commerce Search repeater.
    • HCL Commerce Version 9 moved to JAX-RS 2.0(JSR-339). Also, the documentation API is Swagger 2.0.
    • IBM Websphere Commerce Version 7 used direct JDBC calls, which went through DSL(data service layer) to the database. In HCL Commerce Version 9, JPA 2.1 (EclipseLink) native query is used. Custom queries from previous versions are ported over to the new query service. No additional configure is required.
    • In HCL Commerce Version 9, when Price or Inventory are worked as extended cores, SolrJoin preserves the document relationship between the main CatalogEntry core and the Price and Inventory subcore. MultipleQueryComponent and MultipleFacetComponent, which were used to join or filter the subcores in IBM Websphere Commerce Version 7, are now disabled by default. To handle facet and result fields from extension indexes, a new SearchCatalogEntryExtensionIndexPostprocessor makes a subquery against each of the extension indexes, then joins back with the main index. A new join parameter was also introduced in wc-search.xml. Any IBM Websphere Commerce Version 7 customization to an extension index needs to be implemented to use SolrJoin.
      Note: If needed, you can restore the previous functionality of MultipleQueryComponent and MultipleFacetComponent.

Before you begin

  • Drop all of the temporary and custom temporary tables from your database, except for the following temporary tables:
    • TI_DELTA_CATENTRY
    • TI_DELTA_CATGROUP
    • TI_DELTA_INVENTORY

    Your temporary tables use a TI_ prefix. Whereas, your custom temporary tables use a XI_ prefix.

    Changes were made to the temporary tables between the previous versions of HCL Commerce and HCL Commerce Version 9. Failure to drop the temporary tables might result in preprocess errors, for example, SQLSTAE=56098. For more information about temporary HCL Commerce Search tables, see Temporary table schema definition.

Procedure

  1. Migrate your custom index cores. The following substeps illustrate how to migrate the extended core xCatalogEntry for master catalog 10001 as an example. These steps should be repeated for all of your custom index cores that you want to migrate.
    1. Register your custom index core by running the following SQL statement. During Search server start-up, Search runtime automatically creates the registered core.
      insert into srchconfext (srchconfext_id, indextype, indexscope, language_id, indexsubtype, config) values(srchconfext_id,’CatalogEntry’,10001,’xCatalogEntry’,NULL);
      Where:
      SRCHCONFEXT_ID
      The primary key for the table.
      INDEXTYPE
      Indicates the search engine index to set up. Valid values for the column are:
      CatalogEntry
      Sets up the index for catalog entries in the master catalog.
      CatalogGroup
      Sets up the index for categories in the master catalog.
      INDEXSCOPE
      The scope of the indexed data. For example, if the scope is the master catalog, enter the master catalog ID here.
      LANGUAGE_ID
      Indicates which language to use for the corresponding subtype search index core. For a list of language IDs, see the LANGUGAE_ID column in the LANGUAGE table.
      Note: "LANGUAGE_ID" must be null for Inventory or Price.
      INDEXSUBTYPE
      Indicates which subtype is set for search index core. Valid values are:
      Structured
      Sets up the index for structured content.
      Unstructured
      Sets up the index for unstructured content.
      WebContent
      Sets up the index for site content.
      Inventory
      Sets up the index for inventory data.
      Price
      Sets up the external index core for price data.
      CONFIG
      Indicates extra configurations for a specified Search index core. For example, you can set BasePath and StoreId for the subtype WebContent index core. BasePath indicates the crawled site content path, and StoreId indicates the store into which to build the index. Separate different configurations by commas. For example:
           BasePath=W:\IBM\WebSphere\Liberty\usr\servers\searchServer\resources\search\index\crawler\cache\2017-11-01\1\,StoreId=10501 
      Note: If your customized index core is a master core, such as CatalogEntry or CatalogGroup, you must add records into SRCHCONF table. If your customized index core is an extended core, such as Inventory or Price, you must add records to SRCHCONFEXT table.
    2. Create the following directory.

      Liberty_installdir/usr/servers/default/resources/search/index/managed-solr/config/v3-core-extension

    3. Under the v3-core-extension directory, create a directory called xCatalogEntry. Then, add the following extended Solr core configuration files to the xCatalogEntry folder.
      • schema.xml
      • solrconfig.xml
      • wc-data-config.xml
    4. Restart your Search server.
      Your custom index core is created automatically during start-up and can be found under the following directory.

      Search_ServerDir/default/resources/search/index/managed-solr/config/v3-core-extension

  2. Reimplement your custom index fields, and your preprocessing and build index scripts.
    1. Register all your index cores to your SRCHCONF table.
    2. Migrate your index schema customizations. In HCL Commerce Version 9, the catalog entry fieldType definitions use two templates:
      • A non-customizable template: search-config/src/main/resources/managed-solr/config/v3/common/schema-field-types.xml.
        Note: When the first index is created, this XML file is copied to the resources/search/index/managed-solr/config/v3/common directory. After the index creation, other indexes share this fieldType definition.
      • A customizable template: search-config-ext/src/main/resources/index/managed-solr/config/v3/common/x-schema-field-types.xml.
        Note: When the first index is created, this XML file is copied to resources/search/index/managed-solr/config/v3-core-extension/common directory. After the index creation, other indexes share this customizable fieldType definition.
    3. Migrate your custom schema fieldType by using the following example.

      If you created a new field type x_textSpell for the fr_FR locale in a previous HCL Commerce version, you updated the schema.xml file, which was located under core configuration directory specific to the fr_FR locale. In HCL Commerce Version 9, you must add the field type x_textSpell_fr to the file x-schema-field-types.xml as follows.

      <!-- Spell checking text field -->
      <fieldType omitNorms="true" positionIncrementGap="100" class="solr.TextField" name="x_textSpell_fr">
      <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
      <filter class="solr.WordDelimiterFilterFactory" preserveOriginal="0" splitOnNumerics="1" splitOnCaseChange="1" 
          catenateAll="0" catenateNumbers="0" catenateWords="0" generateNumberParts="1" generateWordParts="1"/>
      <filter class="solr.ShingleFilterFactory" fillerToken="" tokenSeparator=" " maxShingleSize="3" minShingleSize="2" 
          outputUnigrams="true"/>
      <filter class="solr.PatternReplaceFilterFactory" replace="all" replacement=" " pattern="\s{2,}"/>
      <filter class="solr.TrimFilterFactory"/>
      <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      
      <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
      </analyzer>
      </fieldType>
    4. After the new x_textSpell_fr field type is defined in the x-schema-field-types.xml file, define an entry into your x-schema.xml file by using the following example.
      <field name="spellCheck" type="x_textSpell_fr" indexed="true" stored="false" multiValued="true" />
      Notes:
      • If you want to support all languages, you must create a field type for every language in the x-schema-field-types.xml, then add the following field definition into x-schema.xml file.
        <field name="spellCheck" type="x_textSpell_${lang:en}" indexed="true" stored="false" multiValued="true" />
    5. Migrate your customized stop words.
      In IBM Websphere Commerce Version 7, you could change the stopwords.txt file for each core configuration directory. For example, if you customized stopwords.txt for the English language, you need only to change the stopwords.txt file under the en_US core. When you migrate to HCL Commerce Version 9, the index schemas are shared among multiple languages. In order to customize stopwords.txt for a specific language, HCL Commerce Version 9 requires you point to the file in the SRCHCONFEXT database table. For more information about customizing the stopwords.txt, see Customizing the stopwords.txt file. Run the following SQL command to put a stopwords.txt into the config field:
      Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt' 
      where indextype=’CatalogEntry’ and indexscope=’$MASTERCATALOGID’
      When the HCL Commerce Search server is restarted, the following contents are added into x-core.properties under the en_US index data directory.
      stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt

      If you need to change the stopwords.txt file that is under workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common directory, the new stopwords.txt is copied to the Liberty_installdir\resources\search\index\managed-solr\config\v3-core-extension\common directory as part of setting up your WCB and CI/CD pipeline.

      For example, in IBM Websphere Commerce Version 7 you added a new English stop word: can. The following steps show you how to migrate this customization.
      1. Add can to the bottom of your HCL Commerce Version 9 workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common\stopwords.txt file.
      2. Add the customized stopwords.txt path in the CONFIG column of SRCHCONFEXT table by running the following SQL statement:
        Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt' 
        where indextype='CatalogEntry' and indexscope='10001';
    6. Migrate your customized preprocess files.

      In IBM Websphere Commerce Version 7, when you created a customized preprocess.xml file to index external data, the customized file is under the solrhome/pre-processConfig/MC_MasterCatalog/Dbtype directory. In HCL Commerce Version 9, such preprocess XML files reside under the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory.

      Copy the wc-dataimport-preprocess-custom.xml file from your IBM Websphere Commerce Version 7 environment to the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory on HCL Commerce Version 9. Also, copy any files that are referenced by the wc-dataimport-preprocess-custom.xml file to HCL Commerce Version 9, and update the wc-dataimport-preprocess-custom.xml file to ensure that it points to correct path file if you change the file location on HCL Commerce Version 9. For example, you might have specified an input file in the wc-dataimport-preprocess-custom.xml as follows.

      <!-- this property is added new to locate the input file path instead of hard coding it to be in WC\bin -->
      <_config:property name="inputFile" value="W:\WCDE_INT70\bin\Ratings.xml"/>
      
      If so, you must copy the Ratings.xml file from IBM Websphere Commerce Version 7 to HCL Commerce Version 9.0.0.1+, and put it under location WCDE_installdir\bin, and update wc-dataimport-preprocess-custom.xml to point to the new location.
      <!-- this property is added new to locate the input file path instead of hard coding it to be in WC\bin -->
      <_config:property name="inputFile" value="W:\WCDE_v9\bin\Ratings.xml"/>
      
    7. Migrate your customized data import handler files.

      If you wanted to index more data in IBM Websphere Commerce Version 7, you changed the wc-data-config.xml file in the specific core directory. For example, if you want to index more data from the X_RATING table in the English language, you edited the wc-data-config.xml file under the en_US core directory to allow query and deltaImportQuery to join the X_RATING table. In HCL Commerce Version 9, the Solr DataImportHandler (DIH) gets data from the view VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# and X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag#. Add this new customizable table into the view X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# in wc-dataimport-preprocess-x-finalbuild.xml. For more information, see Configuring the Data Import Handler mapping.

      After the customized tables are added into X_VI_CE table, customized data can be returned and made accessible to Solr by the DIH. Define the field-column mapping so that Solr can transfer the new customized column into Solr fields. Update the search-config-ext/src/main/resources/index/managed-solr/config/v3/CatalogEntry/x-data-config.xml file to add the field mapping.

      The following is an example of the new mapping. With this mapping, the RATING column from the X_VI_CE view maps to the customerRanking field in the catalog entry index.
      <field column="RATING" name="customerRanking"/>
    8. Migrate the SRCHATTRPROP table to enable price range queries.
      In HCL Commerce version 7.0, the default price range facet takes the following format.
      "price_USD:{* 100} 100;{100 200} 200;{200 300} 300;{300 400} 400;{400 500} 500;{500 *}"
      In Solr version 7.3.1, this format causes a syntax error in the query parser. If you are using HCL Commerce version 9.0.0.5+, change the query string into the following format.
       "price_USD:{* TO 100];{100 TO 200];{200 TO 300];{300 TO 400];{400 TO 500];{500 TO *}" 
      Note: Convert previous range query formats that take the form "({lower upper} upper)" into "({lower TO upper])". Migrate any other customization that involves the old query format to the new one.
    9. Migrate the default schema field types.
      Starting with Solr version 7.0.0, the Trie*Field fields are deprecated. Replace them with *PointField. The default setting retains the old data type fields (for example, int, tint, sint) and creates new fields (for example, pint, pints). Although the old fields still function, for future compatibility upgrade the old data type to the new one. Some deprecated fields are still used, for example, protected field types, for compatible considerations.
    10. Migrate any customized solrconfig.xml parameters.
      For Solr version 7.3.1, move the configuration parameter solr.mergeFactor in the solrconfig.xml file into the SRCHCONFEXT.CONFIG column. It is replaced by two parameters: solr.mergePolicy.maxMergeAtOnce and solr.mergePolicy.segmentsPerTier. If you previously set the value to something like <mergeFactor>5</mergeFactor>, replace it with solr.mergePolicy.maxMergeAtOnce=5,solr.mergePolicy.segmentsPerTier=5.
    11. Restart your Search server.
    12. Build your Search index.
  3. Migrate your Search configuration files. Any configuration updates that you made under the IBM Websphere Commerce Version 7 WC_eardir directories must be copied to the corresponding extension directories under the Search_eardir directory in HCL Commerce Version 9. The extention directories arecom.ibm.commerce.foundation-ext and com.ibm.commerce.search-ext.
  4. Register your custom search profiles by associating them to a REST service.
    1. Go to the following directory.

      workspace_dir\search-config-ext\src\runtime\config\com.ibm.commerce.rest

    2. Create a file and name it wc-rest-resourceconfig.xml, then add the following XML boilerplate.
      <?xml version="1.0" ?>
      <!--
       =================================================================
        Licensed Materials - Property of IBM
        HCL Commerce
        (C) Copyright IBM Corp. 2013, 2017 All Rights Reserved.
        US Government Users Restricted Rights - Use, duplication or
        disclosure restricted by GSA ADP Schedule Contract with
        IBM Corp.
       =================================================================
      -->
      <!--
      	This XML defines services related configuration data for rest services.
      	Currently the only configurable attributes are searchProfile for GET methods.
      -->
      <ResourceConfig>
          
      </ResourceConfig>
    3. Identify which REST service the custom search profile should be listed under.
    4. Append your custom search profile to the end of the defined list of search profiles.
    5. Save and close the file.
  5. Reapply all of your custom configurations to your wc-search.xml file.
    Notes:
    • Delete all connection configurations that pertain to a remote Solr server. In the HCL Commerce Version 9 Search server, the connection to Solr is embedded.
    • Because your cores are now read from the SRCHCONF and SRCHCONFEXT tables, remove all registered cores from your wc-search.xml file.
    • Any custom search profiles that are defined in the HCL Commerce server must be redefined in the HCL Commerce Search server extension directory.
    • Any custom search profiles that extend a default search profile in the HCL Commerce server must be updated. New default search profiles are introduced in the Search server that contain different names or naming conventions.
    • Any custom search profiles that use any of the default search query providers, processors, or search result filters must be updated. New alternatives are introduced in the Search server that contain different names or use different naming conventions.
  6. Reapply any custom configurations that you made in the wc-component.xml files under the com.ibm.commerce.foundation-ext and com.ibm.commerce.search-ext directories.

    Most of the search-related custom properties that you defined in the wc-component.xml file can be reused in the Search server, except for the global price mode property. The price mode configuration is now stored in the STORECONF table.
 For more information about the search configuration properties in the STORECONF table, see, Search configuration properties in the STORECONF table.

  7. Migrate any custom object mediators that you made in the IBM Websphere Commerce Version 7 wc-business-objectmediator.xml file to HCL Commerce Version 9 wc-component.xml.

    The Search server no longer supports business object mediators. Therefore, any customizations that you applied to the wc-business-object-mediator.xml file must be moved to the wc-component.xml file.

    For example, mappings between a userData custom field to an internal index field or database field must now use the correct mappings under the wc-component.xml file. The following example shows how an existing userData mapping under the HCL Commerce server wc-business-object-mediator.xml file can be moved into the Search server custom wc-component.xml file.

    1. Open your wc-business-object-mediator.xml, and locate the following line of code:
      <_config:mediator-property naem="CatalogEntryView/UserData [ (Name='SKU') ]" value partNumber_ntk"
    2. Open your Search wc-component.xml, and add the corresponding mapping.

      In the Search server, the mapping between internal and external names is performed by using the valuemappingservice section in the wc-component.xml file. There are different maps for CatalogEntry - UserData and CatalogGroup - UserData.

      For the CatalogEntry - UserData mapping, add the following code to your wc-component.xml file.
      <_config:valuemapping externalName="CatalogEntryUserDataFieldNameMapping" internalName="CatalogEntryUserDataFieldNameMapping"> 
              <_config:valuemap externalValue="SKU" internalValue="partnumber_ntk" /> 
      </_config:valuemapping> 
      For the CatalogGroup - UserData mapping, add the following code to your wc-component.xml file.
      <!-- 
          Custom index field name => CategoryView REST response field name 
      
          This CatalogGroupUserDataFieldNameMapping mapping is for defining the mapping from a custom index field 
          name used in the CatalogGroup search index to the field name used in the UserData area in the REST response. 
          
          For example, 
          
                  <_config:valuemap externalValue="response_field_label" internalValue="my_index_field_name" /> 
          
          The name of the index field name can be a dynamic field and this name pattern must end with "*". 
          There is a restriction when using for dynamic fields - only one dynamic field that matches the 
          given name pattern will be mapped. 
          
          For example, 
          
                  <_config:valuemap externalValue="response_field_label" internalValue="my_index_*" /> 
      
          Note: SearchCatalogGroupViewUserDataQueryPostprocessor must be added to the end of 
                the query section of the search profile in order to activate this configuration. 
                In addition, make sure the custom index field is also defined in the result section of 
                the search profile so that this custom index field can be returned from Solr. 
      -->                 
      <_config:valuemapping externalName="CatalogGroupUserDataFieldNameMapping" internalName="CatalogGroupUserDataFieldNameMapping"> 
      </_config:valuemapping> 
      
      
      Where the mappings are being populated by the following postprocessors: 
                      
          <_config:postprocessor classname="com.ibm.commerce.search.internal.expression.postprocessor.SearchCatalogEntryViewUserDataQueryPostprocessor" />   
                                          
              <_config:postprocessor classname="com.ibm.commerce.search.internal.expression.postprocessor.SearchCatalogGroupViewUserDataQueryPostprocessor" /> 
    3. Ensure that the required post-processors are included in your Search profile.
  8. Reapply any custom database query template (TPL) files.

    The Search server supports DSL. However, it does not support EMFs, SDOs, and logical schemas. Therefore, any retrieved data from the database must be parsed by custom code and added into the main response where applicable. Any search-related custom queries can be reused in the Search server. For more information, see Creating a custom query postprocessor.

    • The Search server does not support any Query template tags. Each of the query parameters must be passed into the query services as query parameters.
    • To allow the customized query to run under workspace schema, append bull$SCHEMA$ to the table.
    • If the customized query has different query syntax for Oracle, define a new query name. And in the code, use a different query name after determining the dbType, for example:
      <!-- ======================================================================================= --> 
      <!--  Retrieve search configuration for given master catalog by all store's default language --> 
      <!-- ======================================================================================= --> 
      
      ...
      
      BEGIN_SQL_STATEMENT 
              name=SELECT_SRCHCONF_DEFAULT_ORACLE 
              base_table=SRCHCONF 
              sql= 
              SELECT STORECAT.CATALOG_ID, 
                     STORE.LANGUAGE_ID, 
                     SRCHCONF.INDEXSCOPE, 
                         SRCHCONF.CONFIG 
                FROM $SCHEMA$.STORECAT STORECAT, $SCHEMA$.STORE STORE, $SCHEMA$.SRCHCONF SRCHCONF 
               WHERE STORECAT.MASTERCATALOG = '1' 
                 AND STORECAT.STOREENT_ID = STORE.STORE_ID 
                 AND STORE.STATUS IN (?status?, 1) 
                 AND TO_CHAR(STORECAT.CATALOG_ID) = SRCHCONF.INDEXSCOPE 
                 AND SRCHCONF.INDEXTYPE = ?indexType? 
      END_SQL_STATEMENT 
      
      All customized assets that need to retrieve data from the database, must use SearchQueryService to read data, as seen in the following code example:
      SearchQueryService service = new SearchQueryService(); 
        HashMap parameters = new HashMap(); 
        ArrayList list = new ArrayList(); 
        list.add(getMasterCatalog(coreName)); 
        parameters.put(STR_MASTER_CATALOG_ID, list); 
        list = new ArrayList(); 
        list.add(tokens[1]); 
        parameters.put(STR_INDEX_TYPE, list); 
        list = new ArrayList(); 
        list.add(getLanguageId(coreName)); 
        parameters.put(LANGUAGE_ID, list); 
        List<Object[]> results = service.executeQueryName("SELECT_SRCHCONFEXT", parameters); 
        if (results.size() > 0) { 
                indexSubtype = new ArrayList<String>(); 
                final String STR_INDEXSUBTYPE = "INDEXSUBTYPE"; 
                for (Object[] row : results) { 
                        indexSubtype.add((String) row[1]); 
                } 
                imapSearchIndexSubtype.put(coreName, indexSubtype); 
        } 
      
  9. Migrate your custom Java assets.

    Due to the use of containerization, all customizations must be built by WCB script and deployed by your CI/CD pipeline. By default, custom configuration assets can be added under the search-config-ext directory, and custom java assets can be added under search-logic-ext directory. Then, the default WCB script and CI/CD pipeline can build and deploy those assets to Search container.

    1. You can reuse your custom expression providers.

      For IBM Websphere Commerce Version 7 BOD-based services, all of the default expression providers are packaged under com.ibm.commerce.catalog.facade.server.services.search.expression.solr. For HCL Commerce Version 9, all expression providers are moved to package com.ibm.commerce.search.internal.expression.provider. The naming pattern was changed too. So your custom expression provider needs to be changed to override the new expression provider. The main logic should remain compatible with BOD search.

    2. You can reuse your custom expression preprocessors.

      In IBM Websphere Commerce Version 7, BOD-based Search query preprocessors operated on the native physical SolrQuery object inherited from the following parent AbstractSolrSearchQueryPreprocessor, packaged in com.ibm.commerce.foundation.internal.server.services.search.query.solr.

      In HCL Commerce Version 9 Search, the parent is packaged in com.ibm.commerce.search.internal.expression.preprocessor.

    3. Reapply your custom query result postprocessors.

      If your customized postprocessors operate on the native Query, they can be reused by overriding related postprocessor. However, if the custom postprocessor operates on the SolrCatalogNavigationViewImpl, it cannot be reused.

      Alternatively, change your custom code to operate on the SearchResponse. For example, the following snippet shows how to use the SearchResponse:
      
      List<Map<String, Object>> catalogEntryViews = (LinkedList<Map<String, Object>>)iSearchResponseObject.getResponse().get("external object name");
      
      Where the object name is the external object name. For more information about resolving external names, see the sample custom postprocessors in the wc-search.xml file.
    4. Reapply your custom search query result filters.

      Custom result filters that operate on the logical CatalogNavigationViewType noun are not supported in the HCL Commerce Version 9 Search server. All custom result filters must be reimplemented by using Search query postprocessors. For more information, see Creating a custom query postprocessor.

    5. Reapply your Business Object Mediators.

      In IBM Websphere Commerce Version 7, Business Object Mediators operated on the logical CatalogNavigationViewType noun. This is no longer supported on the Search server. Instead, you can use search query postprocessors. All custom mediators that extend the AbstractReadBusinessObjectPartMediatorImpl parent class must be reimplemented by using Search query postprocessors.

  10. Migrate your storefront search services.

    In IBM Websphere Commerce Version 7, catalog navigation used the getDataTag. In HCL Commerce Version 9, you can use the REST-based equivalent RESTTag.

    Any custom storefront pages that use the CatalogNavigationView BOD services must be updated to use the corresponding REST services. For example, the following snippet is a getData BOD search service that is used to get products by category:
    <wcf:getData type="com.ibm.commerce.catalog.facade.datatypes.CatalogNavigationViewType" var="catalogNavigationView"
       expressionBuilder="getCatalogEntrySearchResultsByIDView" scope="request" varShowVerb="showCatalogNavigationView"
       maxItems="100" recordSetStartNumber="0" scope="request">
       <c:forEach var="marketingSpotData" items="${marketingSpotDatas.baseMarketingSpotActivityData}">
           <c:if test='${marketingSpotData.dataType eq "CatalogEntryId"}'>
               <wcf:param name="UniqueID" value="${marketingSpotData.uniqueID}"/>
           </c:if>
       </c:forEach>
       <wcf:contextData name="storeId" data="${WCParam.storeId}" />
       <wcf:contextData name="catalogId" data="${WCParam.catalogId}" />
    </wcf:getData>
    <c:set var="eSpotCatalogIdResults" value="${catalogNavigationView.catalogEntryView}"/>
    The following snippet is the equivalent REST-based service:
    <wcf:getData type="com.ibm.commerce.catalog.facade.datatypes.CatalogNavigationViewType" var="catalogNavigationView"
       expressionBuilder="getCatalogEntrySearchResultsByIDView" scope="request" varShowVerb="showCatalogNavigationView"
       maxItems="100" recordSetStartNumber="0" scope="request">
       <c:forEach var="marketingSpotData" items="${marketingSpotDatas.baseMarketingSpotActivityData}">
           <c:if test='${marketingSpotData.dataType eq "CatalogEntryId"}'>
               <wcf:param name="UniqueID" value="${marketingSpotData.uniqueID}"/>
           </c:if>
       </c:forEach>
       <wcf:contextData name="storeId" data="${WCParam.storeId}" />
       <wcf:contextData name="catalogId" data="${WCParam.catalogId}" />
    </wcf:getData>
    <c:set var="eSpotCatalogIdResults" value="${catalogNavigationView.catalogEntryView}"/> 

    The response data is being formatted in a dot notation response similar to BOD to minimize the changes that are required in the storefront. In some cases, the response is simplified and flattened to simpler name-value pairs, rather than using internal maps to group certain fields.

    You can examine the JSON response by printing the response object by using the following code:

    <wcf:json object="${catalogNavigationView}"/>

  11. Map your BOD services to REST services.

    All BOD search services can be covered by REST search services. There are three main search handlers: CategoryViewHandler, ProductViewHandler, and SiteContentHandler. Each perform different search functionality. For example, ProductViewHandler can search by Category, productId, productIds, partNumber, partNumbers, and searchTerm.

    Use the following table to help map your BOD services to REST services.

    Table to illustrate mapping between BOD and REST services.

    BOD service Expression builder REST resource REST service
    CatalogNavigationView getCatalogNavigationView ProductViewHandler (Search) store/{storeId}/productview/bySearchTerm/{searchTerm}
    getCatalogNavigationAttachmentView store/{storeId}/productview/bySearchTerm/{searchTerm}
    getCatalogNavigationViewByCategory store/{storeId}/productview/byCategory/{categoryId}
    getCatalogNavigationBreadCrumbView store/{storeId}/productview/byCategory/{categoryId}
    getCatalogNavigationCatalogEntryView store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogEntryViewAllWithoutAttachmentsByID store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogEntrySearchResultsByIDView store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogEntryViewParentInfoByIDNoEntitlementCheck store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogEntryViewForShoppingCart store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogEntryViewPriceWithAttributesByID store/{storeId}/productview/byId/{productId}, store/{storeId}/productview/byIds
    getCatalogNavigationCatalogGroupView CategoryViewHandler (Search) store/{storeId}/categoryview/byId/{categoryId}, store/{storeId}/categoryview/byIds
    getCatalogNavigationCatalogGroupViewByIdentifier store/{storeId}/categoryview/{categoryIdentifier}
    getCatalogNavigationCatalogGroupViewByCatalogId store/{storeId}/categoryview/@top
    getCatalogNavigationCatalogGroupViewByParentCatalogGroup store/{storeId}/categoryview/byParentCategory/{parentCategoryId}
    getWebContentView SiteContentHandler (Search) store/{storeId}/sitecontent/webContentsBySearchTerm/{searchTerm}
    store/{storeId}/sitecontent/brandSuggestions
    store/{storeId}/sitecontent/categorySuggestions
    1. Import EnvironmentSetup.jspf in the page where you used to call BOD service to follow JSP programming guide. In this JSPF file, searchHostName, searchContextPath are defined, so in the JSP where you want to call search rest, this variable could be used directly.
    2. Using the mapping between BOD with search rest, find the equivalent rest service, then change getDataTag to RESTtag.
    3. With BOD request, store side could use CatalogNavigationViewType noun to retrieve needed object from the response of BOD request, this noun basically represents a business response of a catalog browsing request.
  12. Map your BOD search profiles to REST search profiles.
    The following table illustrates the mapping between the search profiles that are used by the CatalogNavigationViewBOD services and the corresponding REST search profiles. There are several factors that differentiate search profiles from each other. When you compare search profiles, consider the following factors.
    Query fields
    Controls the search scope.
    Results fields
    Controls the returned fields.
    Expression providers
    Contributes to the selection criteria object.
    Preprocessors
    Prepares the final search expression object.
    Postprocessors
    Mediates the search response and contributes to the final response object.

    Mapping between BOD Search profiles and REST Search profiles.

    BOD search profile REST search profile
    IBM_ComposeProductListByCategoryId IBM_findProductsByCategory
    IBM_ComposeCategoryFacetListByCategoryId Deprecated by IBM_findProductsByCategory
    IBM_BreadCrumb IBM_BreadCrumbByCategoryUniqueId
    IBM_findFacetsByCategory Deprecated by IBM_findProductsByCategory
    IBM_ComposeFacetListByCategoryId IBM_ComposeFacetListByCategoryId
    IBM_findCatalogEntryWithoutDescriptionByNameAndShortDescription IBM_findProductsBySearchTerm
    IBM_findCatalogEntryWithoutDescriptionByNameAndShortDescriptionInDetail Deprecated by IBM_findProductsBySearchTerm
    IBM_findCatalogGroupByFacet Deprecated by IBM_findProductsBySearchTerm
    IBM_findCatalogEntryByName IBM_findProductsByNameOnly
    IBM_findCatalogEntryByUnstructureField IBM_findProductsByUnstructureOnly
    IBM_findCatalogEntryByNameAndShortDescriptionOnly IBM_findProductsByNameAndShortDescriptionOnly
    IBM_findCatalogEntryByNameAndShortDescription Deprecated by IBM_findProductsByNameAndShortDescriptionOnly
    IBM_findCatalogEntryByNameAndShortDescriptionInDetail Deprecated by IBM_findProductsByNameAndShortDescriptionOnly
    IBM_findCatalogEntryIdByNameAndShortDescription Deprecated by IBM_findProductsByNameAndShortDescriptionOnly
    IBM_findCatalogEntryDetails IBM_findProductByIds_Details
    IBM_findCatalogEntryAll Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryAll_PriceMode Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntrySKUs Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryDetailsWithComponents Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryDetailsWithComponentsAndAttachments Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryDetailsWithMerchandisingAssocDetails Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryDetails_PriceMode Deprecated by IBM_findProductByIds_Details
    IBM_findComponentsSummary Deprecated by IBM_findProductByIds_Details
    IBM_findComponentsSummaryDetails Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntryDetailsWithMerchandisingAssocSummary Deprecated by IBM_findProductByIds_Details
    IBM_fetchRelatedCatalogEntryDetailedInfo Deprecated by IBM_findProductByIds_Details
    IBM_findCatalogEntrySummary IBM_findProductByIds_Summary
    IBM_findCatalogEntryByID Deprecated by IBM_findProductByIds_Summary
    IBM_findCatalogEntryPrice Deprecated by IBM_findProductByIds_Summary
    IBM_findCatalogEntryDynamicKitSummary Deprecated by IBM_findProductByIds_Summary
    IBM_fetchRelatedCatalogEntrySummaryInfo Deprecated by IBM_findProductByIds_Summary
    IBM_CatalogEntryCategoryEntitlement Deprecated by IBM_findProductByIds_Summary
    IBM_CatalogEntryEntitlement Deprecated by IBM_findProductByIds_Summary
    IBM_findCatalogEntryPriceWithAttributes_PriceMode Deprecated by IBM_findProductByIds_Summary
    IBM_findCatalogEntryAttachments IBM_findProductByIdsWithAttributesAndAttachments
    IBM_findCatalogEntryDetailsWithAttachments Deprecated by IBM_findProductByIdsWithAttributesAndAttachments
    IBM_findCatalogEntryPriceWithAttributes Deprecated by IBM_findProductByIdsWithAttributesAndAttachments
    IBM_findAttachmentByCatentryId Deprecated by IBM_findProductByIdsWithAttributesAndAttachments
    IBM_findCatalogEntryParentInfoNoEntitlementCheck IBM_findProductByIds_Summary_WithNoEntitlementCheck
    IBM_findCatalogEntryForShoppingCart IBM_findProductByIds_Summary_WithNoEntitlementCheck
    IBM_findCatalogGroupSummary IBM_findCategoryByUniqueIds, IBM_findCategoryByIdentifier
    IBM_findCatalogGroupDetails IBM_findSubCategories
    IBM_Global_WebContent IBM_findWebContentsBySearchTerm
    IBM_findAttachmentByContent Deprecated by IBM_findWebContentsBySearchTerm
    IBM_findNavigationSuggestion_Brands IBM_findNavigationSuggestion_Brands
    IBM_findNavigationSuggestion_Categories IBM_findNavigationSuggestion_Categories
    IBM_Global There is no exact match for this Search profile on the HCL Commerce Version 9 Search server. Consider the following search profiles as replacements:
    • IBM_findProductsByCategory (for navigation)
    • IBM_findProductsBySearchTerm (for keyword search)
    • IBM_findProductByIds_Details (for the product display page)
    IBM_Global_Unstructured There is no exact match for this search profile on the Search server. Consider the following search profiles as replacements:
    • IBM_findWebContentsBySearchTerm (for web content)
    • IBM_findProductsByUnstructureOnly (for searching product attachments)
    IBM_findNavigationSuggestions There is no exact match for this search profile on the Search server. Consider the following search profiles as replacements:
    • IBM_findNavigationSuggestion_Categories (for category suggestions)
    • IBM_findNavigationSuggestion_Brands (for brand suggestions)
  13. Map your BOD expression providers to REST expression providers by referencing the following table.
    BOD-based search expression provider REST-based search expression provider Description
    SolrSearchBasedMerchandisingExpressionProvider SearchBasedMerchandisingExpressionProvider Calls the marketing component to run search rules.
    SolrSearchByCatalogExpressionProvider SearchByCatalogExpressionProvider Handles searching by catalog, considering the sales catalog in the current business context.
    SolrSearchByCategoryExpressionProvider SearchByCategoryExpressionProvider Handles searching by category, considering the sales catalog in the current business context.
    SolrSearchByCustomExpressionProvider SearchByCustomExpressionProvider Includes custom expressions that are stored in _wcf.search.expr.
    SolrSearchByFacetExpressionProvider SearchByFacetExpressionProvider Handles searching by facet requests.
    SolrSearchByKeywordExpressionProvider SearchByKeywordExpressionProvider Handles searching by keyword requests.
    SolrSearchByKeywordRelevancyExpressionProvider SearchByKeywordRelevancyExpressionProvider Handles searching by keyword requests that use the dismax query parser.
    SolrSearchByManufacturerExpressionProvider SearchByManufacturerExpressionProvider Handles searching by brand name requests.
    SolrSearchByPriceExpressionProvider SearchByPriceExpressionProvider Handles searching by price range requests generated from the Advanced Search page.
    SolrSearchByPublishedEntryOnlyExpressionProvider SearchByPublishedEntryOnlyExpressionProvider Generates conditions for restricting search results to only published entries.
    SolrSearchByStorePathExpressionProvider SearchByStorePathExpressionProvider Generates conditions to handle the store path.
    SolrSearchCategoryEntitlementExpressionProvider SearchCategoryEntitlementExpressionProvider Performs category entitlement.
    SolrSearchFacetConditionExpressionProvider SearchFacetConditionExpressionProvider Generates a list of attribute-related facets and currency-specific price range facets for the current search request.
    SolrSearchInventoryExpressionProvider SearchInventoryExpressionProvider Handles searching related to the Inventory index.
    SolrSearchProductEntitlementExpressionProvider SearchProductEntitlementExpressionProvider Performs product entitlement.
    SolrSearchSequencingExpressionProvider SearchProductSequencingExpressionProvider Arranges product entries in the search result by ranking.
    SolrSearchTermAssociationExpressionProvider SearchTermAssociationExpressionProvider Gets synonyms and replaces the search term to fetch the final result.
    SolrSearchTypeExpressionProvider SearchTypeExpressionProvider Handles the match type for keyword search requests, such as Any and Exclude SKU.
    SolrSearchWebContentStoreInfoExpressionProvider SearchWebContentStoreInfoExpressionProvider Handles adding conditions to search store-specific site contents
  14. Map your BOD post-processors to REST post-processors by referencing the following table.
    BOD-based search preprocessor REST-based search preprocessor
    SolrSearchResultGroupingQueryPreprocessor SearchResultGroupingQueryPreprocessor
    SolrSearchDebugQueryPreprocessor SearchDebugQueryPreprocessor
    SolrSearchEDismaxQueryPreProcessor SearchEDismaxQueryPreProcessor
    SolrSearchFacetQueryPreprocessor SearchFacetQueryPreprocessor
    SolrSearchHighlighterQueryPreprocessor SearchHighlighterQueryPreprocessor
    SolrSearchMainQueryPreprocessor SearchMainQueryPreprocessor
    SolrSearchPaginationQueryPreprocessor SearchPaginationQueryPreprocessor
    SolrSearchPreviewQueryPreprocessor SearchPreviewQueryPreprocessor
    SolrSearchResultFieldQueryPreprocessor SearchResultFieldQueryPreprocessor
    SolrSearchSortingQueryPreprocessor SearchSortingQueryPreprocessor
    SolrSearchSpellCorrectionQueryPreprocessor SearchSpellCorrectionQueryPreprocessor
    Note: The following are new post-processors:
    • SearchCustomQueryPreprocessor
    • SearchJoinQueryPreprocessor
    • SearchManualSequenceOverrideQueryPreprocessor
    • SearchProductSequenceDebugInfoQueryPreprocessor
    • SearchRelevancyByProductGroupingQueryPreprocessor
    • SearchResponseFormatQueryPreprocessor
  15. Map your BOD search result filters and postprocessors to REST search result filters and postprocessors by referencing the following table.
    BOD-based search result filter REST-based search postprocessor
    SearchCatalogEntryMerchandisingAssocResultFilter SearchCatalogEntryViewMerchandisingAssocQueryPostprocessor
    SearchCatalogEntryViewAttachmentsResultFilter SearchCatalogEntryViewAttachmentsQueryPostprocessor
    SearchCatalogEntryViewAttributesAllowedValueResultFilter SearchCatalogEntryViewAttributesQueryPostprocessor
    SearchCatalogEntryViewAttributesResultFilter SearchCatalogEntryViewAttributesQueryPostprocessor
    SearchCatalogEntryViewDescriptionResultFilter SearchCatalogEntryViewDescriptionQueryPostprocessor
    SearchCatalogEntryViewPackageBundleResultFilter SearchCatalogEntryViewComponentsQueryPostprocessor
    SearchCatalogEntryViewPriceResultFilter SearchCatalogEntryViewPriceQueryPostprocessor
    SearchCatalogEntryViewSingleSKUResultFilter The logic that is deprecated by indexing the childCatentry_id field.
    SearchCatalogEntryViewSKUResultFilter SearchCatalogEntryViewSKUQueryPostprocessor
    SearchCatalogEntryViewStoreDisplayAttributesResultFilter SearchCatalogEntryViewAttributesQueryPostprocessor
    SearchCatalogGroupEntitlementResultFilter SearchCategoryEntitlementQueryPostprocessor, SearchChildCategoryEntitlementQueryPostprocessor
    SearchCatalogNavigationViewDynamicKitResultFilter SearchCatalogEntryViewDynamicKitQueryPostprocessor
    SearchCatalogNavigationViewPreviewResultFilter SearchPreviewQueryPostprocessor
    SearchCatalogNavigationViewSEOTitleMetaDataFilter The Search Server does not return SEO metadata. Metadata is returned from the WebSphere Commerce server.
    SearchNavigationSuggestionsResultFilter SearchCategorySuggestionQueryPostprocessor, SearchBrandSuggestionQueryPostprocessor
  16. Reapply your Solr customizations.

    HCL Commerce Version 9 uses Solr 5.5.4. All customization based on the previous Solr versions must be implemented on Solr 5.5.4.

  17. Migrate any customized search services.
    1. Create a HCL Commerce Version 9 endpoint in your development environment.
      1. Go to the <WCDE_installdir>\workspace\search-ear directory.
      2. Copy the search-rest.war to the search-rest-ext.war.
      3. Import the WAR file by right-clicking search-rest-ext.war project, then click Properties > Web Project setting.
      4. Change the context root to search/ext/resource.
      5. Write your Java code and save it to the src directory under the search-logic-ext project.
      6. Register that class to resources.properties in the search-rest-ext/WebContent/WEB-INF/config directory.
        Note: Remove any existing classes from the properties file.
      7. Create your preprocessors and post processors to the src directory in the search-logic-ext project.
      8. Create a Search profile by using preprocessors and post processors in the wc-search.xml file, which is found in the /src/runtime/config/com.ibm.commerce.search directory.
      9. Associate the Search profile with the corresponding method in the wc-rest-resourceconfig.xml file, which is found in the folder /src/runtime/config/com.ibm.commerce.rest directory.
    2. Configure WCB for Search.
      1. Download the WCBSamples.zip file.
      2. Extract the ZIP file. Copy the WCBSamples/search/wcbd to your <WCDE_installdir>/wcbd directory. Then, copy WCBSamples/search/Build_Local_Repository to <WCDE_installdir>.
      3. Open the build-local-search.properties file, and update the following properties with the values specific to your environment.
        wc.home=W:/WCDE_V9
        was.home=W:/IBM/WebSphere/AppServer
        web.module.list=search-rest-ext
      4. Open the extract-local-search.properties file, and update the following property with the value specific to your environment.
        local.extract.dir=W:/WCDE_V9/Build_Local_Repository/search
      5. Clear the contents of the <WCDE_installdir>\Build_Local_Repository\search\workspace directory.
      6. Copy search-config-ext, search-logic-ext, andsearch-rest-ext folders to the <WCDE_installdir>\Build_Local_Repository\search\workspace directory.
      7. Open a command prompt, go to the <WCDE_installdir>\wcbd directory, then execute the following command.
        wcbd-ant.bat -buildfile wcbd-build.xml -Dbuild.type=local -Dapp.type=search -Dbuild.label=demo
      8. Go to the <WCDE_installdir>\wcbd\dist\server directory and verify that your wcbd-deploy-server-local-search-demo.zip package is created. You extract this package in the following step.
    3. Prepare the customized Search Docker image. This step assumes that you are using Docker Compose in an authoring environment. For more information about creating this environment, see Deploying an HCL Commerce Version 9.0.0.0 to 9.0.1.17 authoring environment with Docker Compose.
      1. Create a directory that is named cust to host your docker-compose.yml file. Then, create a cust/search directory to host your customized package, application.xml file, and Docker file.
      2. Extract your wcbd-deploy-server-local-search-demo.zip to the cust/search/CusDeploy directory, then copy the <WCDE_installdir>\workspace\search-ear\META-INF\application.xml file to the cust/search directory.
      3. In the cust/search directory, create a Docker file with the following contents.
        FROM <Docker_registry>/commerce/search-app
        COPY CusDeploy /SETUP/Cus
        RUN /SETUP/bin/applyCustomization.sh
        COPY CusDeploy/Code/search-app/search-rest-ext.war/profile/apps/search-ear.ear/search-rest-ext.war/ 
        COPY application.xml /profile/apps/search-ear.ear/META-INF
        
    4. Start the docker-compose environment.
      1. Go to the \cust directory run the following command:
        docker-compose up -d --build
      2. After all services are started, build your index by running the following curl command:
        curl -X POST -k -u spiuser:<password> https://localhost:5443/wcs/resources/admin/index/dataImport/build?masterCatalogId=10001
      3. After the index is built, browse the following URL to verify that the site is working as expected:
        https://localhost:3738/search/ext/resources/store/1/extproductview/byCategory/10001?currency='USD'&searchSource='O'&pageSize=2&pageNumber=1&langId=-1
  18. Migrate your data cache.

    In HCL Commerce Version 9, data cache is enabled on the Search server. For more information about data cache, see Enabling cache monitoring.

  19. Migrate your custom scheduled jobs.
    In HCL Commerce Version 9, a scheduler exists on the Search server. You can use the following table to recreate your scheduled Search jobs.
    • SRCH_SCHACTIVE
    • SRCH_SCHBRDCST
    • SRCH_SCHCONFIG
    • SRCH_SCHERRLOG
    • SRCH_SCHREPORT
    • SRCH_SCHSTATUS

What to do next

The next step in the migration process is to build and deploy your custom containers. After those containers are deployed, you can build your index. For more information about building your index, see Building the HCL Commerce Search index.