Index synchronization and delta updates in Commerce Search

The Commerce Search index can typically fall out of synchronization with the latest production data over time. To maintain accuracy of the search result data, reindexing must be done at a certain time during normal business operation. There are three forms of search reindexing, and depending on your business requirements, costs and benefits are associated with each of the following options: Full reindexing, delta reindexing, and emergency updates.

Consider the following scenarios when you select a reindexing option:
Full reindexing
Performing a full reindexing rebuilds the entire search index. After the indexing completes, the search engine automatically uses the updated index. Full reindexing is effective when many updates are applied over time and the index starts to degrade in performance due to fragmentation. However, a full reindexing can take considerable time and therefore is best done on an extended schedule, for example on a weekly basis.
Delta reindexing
A delta reindexing makes only incremental updates to the existing operational search index. Delta reindexing is effective when the amount of changed data is manageable and small enough to be applied as a batch. Typically, delta reindexing is done at regular intervals in a staging environment. After the changes are ready, it is published from the staging environment over to production through replication, thus minimizing the business impact. However, you may encounter business scenarios where a delta reindex might be too complicated, and therefore a full reindexing is preferred.
To run a delta reindexing, set the fullbuild flag to false when the system builds the search index. Otherwise, the default value is true, which does a full preprocess and index build.

Building the index

Use the build index REST call to do both data preprocessing and index building.

For full information about the index build REST call, see Building the Commerce Search index.

The build process extracts and flattens Commerce+ data and then outputs the data into a set of temporary tables inside the Commerce+ database. The data in the temporary tables is then used by the index building utility to populate the data into search indexes by using the Data Import Handler (DIH). When are using multiple indexes, for example, each language with its own separate index, the index is built multiple times.

For more information, see Building the search index.

Events that trigger search reindexes

The search index is triggered for reindexing when certain actions take place within Commerce Search. Reindexing is necessary to avoid working with stale data, ensuring that the content you are working with or previewing is up-to-date.

In the Management Center, with workspaces enabled:
  • Task group commit
  • Store preview
  • Show facets (catalog entry index)
  • UpdateSearchIndex scheduler job
    Note: This scheduler task triggers reindexing only on the base index. It skips workspace indexes to prevent unnecessary reindexing that is caused by uncommitted changes.
In the Management Center, with workspaces disabled:
  • Store preview
  • Show facets (catalog entry index)
  • UpdateSearchIndex scheduler job
In the Aurora starter store:
  • UpdateSearchIndex scheduler job.
    Note: This scheduler job detects changes that are made from the authoring environment by using the TI_DELTA_CATENTRY and TI_DELTA_CATGROUP temporary database tables. That is, after the quick publish task is approved, the runtime automatically publishes the search change history to the temporary tables in both the staging and production databases. It then starts a reindexing for emergency fixes from the production environment.
Using utilities:

When to perform full search index builds

The Commerce Search index is automatically built when certain business tasks are performed, as outlined in Common business tasks and their impact to the Commerce Search index. In several cases, common business tasks result in delta index builds that do not pose a significant risk to production system performance. However, doing several delta index builds without occasional full index builds might result in the search index gradually degrading over time due to fragmentation. To avoid this issue, doing full search index builds when possible ensures that the search index performs well over time.

When Lucene receives a delete request, it does not delete entries from the index, but instead marks them for deletion and adds updated records to the end of the index. This marking results in the catalog unevenly spreading out across different segment data files in the search index, and might result in increased search response times. If you have a dedicated indexing server, consider scheduling a periodic full search index build. Make this build a background task that runs once per month, so that the deleted entries are flushed out, and to optimize the data.

Scheduler jobs

Commerce Search contains scheduler jobs to trigger for index synchronization.

For more information, see Scheduling jobs.

Search index approaches

Although catalog entries are mentioned in index synchronization and delta updates in Commerce Search, the same approach applies for the other search indexes or BODs such as ChangeCatalogGroup.

Search index optimization

There are several search optimization hints and tips to consider when administering Commerce Search. For more information, see Search index optimization.