Migrating from Solr to Elasticsearch

The following table describes the Solr-based and Elasticsearch-based search solutions and their concepts. A guideline is also presented for migrating from the Solr-based search solution to the newer Elasticsearch-based solution.

Solution overviews

Solr-based search solution

The Solr-based search solution is used in previous releases of WebSphere Commerce, and HCL continues to support its use with Commerce+.

Elastic-based search solution

With the release of Commerce+, a new search and indexing platform based on the popular open-source Elastic search engine was introduced. In this new search platform paradigm:

The Ingest Service provides a new data loading architecture, allowing for easier customization and incremental index updates.
The Query Service provides enhanced search relevancy for site users navigating through the catalog on the storefront.

This micro service-based search solution retains backwards compatibility with the Commerce+ Transaction server, storefront, and externalized customization (xC) customizations. Advancements in natural language parsing, part-of-speech tagging, and text mining enable more sophisticated merchandising and analysis capabilities.

Migration guidelines

The following table contains general guidelines to help you migrate your Solr-based search solution to the Elastic-based search solution.


Migration Item	With Solr-based search	With Elastic-based search	Migration guideline
Ingest / Indexing
Index Schema	Index schema definitions (such as field, filter and analyzers) are defined in schema.xml and x-schema.xml files. See ../../search/refs/rsdschemaxml.html for details.	Defined in data specification for each object type (product, category, attribute, etc.) that is part of the NiFi Ingest Service connector.	Index field definitions that were previously located in the Solr schema.xml are now defined in data specification for each object type (product, category, attribute, etc.) that is part of the NiFi Ingest Service connector.
Pre-process Configuration	Pre-process configuration is defined in XML files and data is processed in temporary database tables and views. For more information on search preprocessing, see Configuring the search preprocessor. The di-preprocess command is available to launch pre-processing for the search index in WebSphere Commerce. This command has been removed from Commerce+, and pre-processing is now automatically processed as part of the index.	Defined in NiFi Ingest Service connector (consisting of input, mapping, processing and output pipelines).	The approach to preprocessing data for the index has changed significantly with the new Commerce+ architecture. Preprocess configuration is no longer defined in XML files and there are no temporary database tables needed for flattening of the index data. The new Ingest Service handles the extraction or ingestion and has the flexibility to process data from many different sources and to transform the data in memory in a high-performance Apache NiFi cluster. The extract, transform, and load (ETL) processes are defined in NiFi Ingest Service connector (consisting of input, mapping, processing and output pipelines).
Extension indexes	Some data, such as inventory and price, can be built into a separate index so that it can be updated separately with its own frequency. For an example of setting up a separate inventory index, see Configuring the search preprocessor.	Not required with the new incremental index update function.	Extension indexes are no longer necessary. This solution was introduced for use in the Solr-based search solution to avoid rebuilding the entire base Catalog Entry (product) index document when only certain frequently changed fields, such as inventory and price, are updated. Now with the ability of incremental index update, it is possible to update only the desired fields in any existing indexed document. Limitations of extension indexes, including the inability to perform sorting, faceting, grouping are now lifted with Commerce+ with the Elastic-based based search solution. This is due to the fact all of the fields are now in the same product index.
Unstructured content indexes	The unstructured content such as PDF files or attachment files for products can be indexed and categorized under the catalog entry search index. For information on indexing unstructured content with the Solr-based search solution, see Unstructured and site content.	A single ingest connector needs to be created for each source of documents.	In Commerce+, this process has been simplified. A single ingest connector needs to be created for each source of documents. This process allows you to more easily customize the extensions that will consume the data and load it into the indexes of the search component.
Site content crawler	Unmanaged site content of store can be crawled and indexed into the Solr index using the crawler REST API.	A crawler processor can be used to gather and index web page URLs and/or scrape and index page content. Over 250 freely available NiFi processors can be used to handle different ingestion scenarios. There are also many freely available use case examples to aid in their implementation.	With Commerce+, the site crawler can continue to be used to gather the URLs of the pages, but it will not automatically load the data into the search index. However, a custom connector and pipeline can be created to load the data. The following articles highlight how to build a web crawler pipeline with NiFi: Apache NiFi In Depth: WebCrawler Template Web scraping with NiFi and Scrapy via the ExecuteProcess processor
Launch building index	The di-buildindex command is available to launch index builds for WebSphere Commerce version 7 and 8. The build index REST API is available in Commerce+ to launch index builds.	The NiFi pipeline output will trigger an Elastic index build or update.	In Commerce+ the search index can be built by triggering a NiFi connector job. See Building the Elasticsearch index for more details.
Query / Runtime
Search runtime configuration	Search application features related configuration, such as search profile, query expressions, facets, etc. can be defined in wc-server.xml. See ../../search/concepts/csdwcsearchxml_new.html for details. Search server related configuration, such as spell check, relevancy, search runtime properties, etc. can be defined in wc-component.xml file. See ../../search/concepts/csdwccomponentxml_new.html for details.	In general search configurations are stored in zookeeper with Query REST services.	In Commerce+, ZooKeeper is used to store your custom configurations. At runtime, each microservice polls ZooKeeper for any custom configurations to automatically override default behaviors, such as query responses. For the Ingest Service, ZooKeeper stores connector descriptors that are used for loading new custom NiFi connectors. For more information on setting up custom profiles in ZooKeeper, as well as implementing custom search options, see Configuring Query services in ZooKeeper.
Search profiles	Search profiles control the behavior of search on the storefront by manipulating the final search query and formatting the search responses. The search profile is defined in the wc-search.xml and contains the fields that are being searched, expression providers, the query pre-processors and post-processors to be used, and other relevant information.	Search profiles still define the search runtime behavior. However, profile information is now stored in the dynamic configuration service using ZooKeeper, and can be managed using REST.	Existing Solr-based search profiles should be assessed, along with the defined expression providers, and any custom query pre and post processors in the profiles. The introduction of Natural Language Processing (NLP) in the new Commerce+ Elastic-based search solution may provide an opportunity to remove runtime customizations that are defined in your Solr-based search profiles. To set up search profiles in ZooKeeper, see Configuring Query services in ZooKeeper.
Query expression providers	Query expression providers allow for modifications to be made to queries before they are read by query pre-processors and added to the search query. This allows for more control over search parameter values.	The existing query expression provider extension programming framework is also provided with the Elastic-based search solution. Note: The Query Service does not provide direct database access. Data should be indexed, or additional microservices should be built to expose any custom data that is required.	Custom expression providers related to search relevancy and ranking should be assessed and evaluated considering the addition of Natural Language Processing (NLP).
Query pre-processors	Query pre-processors are used to modify the query before it is processed by Solr. You can use control parameters that are provided for the search request to add data to the query. For example, you can add a sort parameter based on the value in the `_wcf.search.sort` control parameter.	The existing query pre-processor provider extension programming framework is also provided with the Elastic-based search solution. Note: The Query Service does not provide direct database access. Data should be indexed, or additional microservices should be built to expose any custom data that is required.	The Elastic-based search solution provides significant enhancements to search matching and ranking with the introduction of Natural Language Processing (NLP) that may provide an opportunity to remove runtime customizations related search relevancy. Custom query pre-processors related to search relevancy and ranking should be assessed and evaluated considering the addition of NLP.
Query post-processors	Query post-processors are used to modify the query results before they are returned as the search response.	The Elastic index, and the structure of the search response are both structured differently from Solr. Hence, the two solutions also use different post-processors. Any custom post-processors will need to be recreated to conform with the Elastic solution. Note: The Query Service does not provide direct database access. Data should be indexed, or additional microservices should be built to expose any custom data that is required.	Custom query post-processors should be assessed to determine whether the customization could be recreated an a NiFi Ingest Service connector, and used to structure the static Elastic index. Custom post-processors that cannot be moved to index-time will need to be recreated, as the Elastic search response is in a different format from the Solr search response.
Storefront REST API	Commerce+ V1 REST API	Commerce+ V1 and V2 REST API	Storefronts should be evaluated for compatibility, and updated to utilize the Commerce+ V1 REST API.
Configuration
Solr-specific configurations	Some Solr configurations are in Solr specific configuration files. For example, core configuration properties and request URL mappings are in solr.xml; and Solr request handler mapping and Solr parameters are in solrconfig.xml file.	These configurations are Solr specific, and are not needed for the Elastic-based search solution.	These configurations are not needed in Elastic-based search solution.
Search configurations in STORECONF	Some search properties, such as price mode and entitlement, are configured as an entry in STORECONF database table.	Existing properties that are configured in STORECONF are still valid. Some new properties are also stored in STORECONF. For example, a property to tell if a store is headless or not.	No migration is needed.
Search Merchandising & Relevancy
Search rules	Search Rules can be defined in the Marketing tool in Commerce Lab.	Existing search rules are still valid.	No migration is needed.
Search term associations	Search Term Associations are defined in the Catalogs tool in Commerce Lab.	Existing search term associations are still valid.	No migration is needed.
Tuning Relevancy	Search relevancy can be tuned in Commerce Lab with the combination of search rules, search term association, and other search tuning functions.	Search relevancy can be tuned with the Natural Language Processing (NLP) feature.	Search relevancy in the Elastic-based search solution provides significant search matching and ranking improvements with the inclusion of Natural Language Processing (NLP) and part-of-speech tagging. New NLP filters have been added that provide expanded opportunities to improve the search experience.
Hero product image	The hero product image functionality returns the most representative item for a product search.	The hero product image functionality returns the most representative item for a product search.	No migration is needed.
Deployment
Search caching	Search caching uses a data object cache using Dynacache with Kafka and ZooKeeper for cache invalidation.	HCL Cache, with Redis as cache provider, provides REST and data object caching that offers event-based, course-grained, cache invalidation. Time-based (TTL) cache invalidation is provided as a fallback.	Client microservices to expose custom data should be evaluated for caching and invalidation requirements. Custom Kafka events (event bus) should be migrated to Redis.
Local store	A local store, that is, a store running on the Transaction server, works with Solr-based search.	A REST-based local store can work with the Elastic-based search platform. A BOD-based local store will not work with the Elastic-based search solution.	It is recommended to use a remote store, or headless store to work with the Elastic-based search solution.
Sharding and scalability	Limited index scaling. No query or runtime sharding or scaling.	Scalable Ingest Service microservices. The Query Service provides runtime dynamic scalability and distributed search services using Elastic clustering.	No migration is needed.