Collecting Elasticsearch based search-related MustGather data for Near-Real-Time (NRT) updates

Prepare the following MustGather information before you contact HCL Support to help with the troubleshooting process.

About this task

This MustGather can be used to investigate the cause of Elasticsearch-based NRT issues. If you are trying to answer one of the following questions, this is the MustGather you will want to use:

Why am I not seeing a change made in Management Center to catalog objects (SKUs, variants, products, etc) visible on the storefront?
Why am I not seeing my catalog upload changes visible on the storefront?
Why am I not seeing my dataload changes to my catalog visible on the storefront?

Procedure

Subscribe to the WCNifiDistributedMapCache topic in Redis.

[root@mymachine ~]# docker exec -it commerce_redis_1 bash
I have no name!@c155d5fa5da1:/$ redis-cli 
127.0.0.1:6379> subscribe {cache-auth-services/cache/WCNifiDistributedMapCache}-invalidation 
Reading messages... (press Ctrl-C to quit)

Enable the following tracing on the Transaction server.

*=info:com.ibm.commerce.catalog.commands.*=all:com.ibm.commerce.catalog.facade.server.helpers.RuleBasedCategoryHelper=all:com.ibm.commerce.catalog.changehistory.*=all:com.ibm.commerce.foundation.internal.server.services.changehistory.*=all:com.ibm.commerce.foundation.server.services.changehistory.*=all:com.ibm.commerce.foundation.server.command.bod.BusinessObjectDocumentProcessor=all

Reproduce your NRT issue. Record the specific steps that you used to reproduce the problem and any details that might be relevant (for example, catentryId values for modified SKUs, variants, or products, or catgroupId values for modified categories).
Collect the trace files generated in the following directory on your Transaction server.
/opt/WebSphere/AppServer/profiles/default/logs/container/{{container_name}}/
Note all the messages reported to the WCNifiDistributedMapCache topic in Redis.

What to do next

Reviewing the data

Change History Event

When a catalog change in made in Management Center or a dataload/catalog upload is ran, a change history event is generated to track this change. For example, I changed the "Hawthorne Towel" product (catentryId 14503 in our default catalog) to have name "Hawthorne Towel TEST123". Searching for the ChangeCatalogEntry BOD request will show the new definition for this product to be used:

[7/6/23 16:41:52:073 GMT] 000000f6 BusinessObjec > com.ibm.commerce.foundation.server.command.bod.BusinessObjectDocumentProcessor processBusinessObjectDocument(BusinessObjectDocumentType, String) ENTRY <?xml version="1.0" encoding="UTF-8"?>
<_cat:ChangeCatalogEntry xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:_cat="http://www.ibm.com/xmlns/prod/commerce/9/catalog" xmlns:_wcf="http://www.ibm.com/xmlns/prod/commerce/9/foundation" xmlns:oa="http://www.openapplications.org/oagis/9" releaseID="9.0" versionID="7.0.0.0">
  <oa:ApplicationArea xsi:type="_wcf:ApplicationAreaType">
    <oa:CreationDateTime>2023-07-06T16:41:52.071Z</oa:CreationDateTime>
    <oa:BODID>01fa58b0-1c1c-11ee-945e-8330646aed01</oa:BODID>
    <_wcf:BusinessContext intent="Authoring">
      <_wcf:ContextData name="timeZoneId">GMT</_wcf:ContextData>
      <_wcf:ContextData name="locale">en_US</_wcf:ContextData>
      <_wcf:ContextData name="storeId">11</_wcf:ContextData>
    </_wcf:BusinessContext>
  </oa:ApplicationArea>
  <_cat:DataArea>
    <oa:Change>
      <oa:ActionCriteria>
        <oa:ActionExpression actionCode="Change" expressionLanguage="_wcf:XPath">/CatalogEntry[1]/Description[1]</oa:ActionExpression>
      </oa:ActionCriteria>
    </oa:Change>
    <_cat:CatalogEntry>
      <_cat:CatalogEntryIdentifier>
        <_wcf:UniqueID>14503</_wcf:UniqueID>
      </_cat:CatalogEntryIdentifier>
      <_cat:Description language="-1">
        <_cat:Name>Hawthorne Towel TEST123</_cat:Name>
      </_cat:Description>
      <_wcf:SEOURL>
        <_wcf:ParentStoreIdentifier>
          <_wcf:UniqueID>11</_wcf:UniqueID>
        </_wcf:ParentStoreIdentifier>
      </_wcf:SEOURL>
    </_cat:CatalogEntry>
  </_cat:DataArea>
</_cat:ChangeCatalogEntry> com.ibm.commerce.catalog

To confirm that a change history event was created for this change, you can look for the tracing output from EchoBusinessObjectChangeHistoryDelegateMediatorImpl.createChangeHistory(). For example, here is the change history event creation tracing for the "Hawthorne Towel" update from above.

[7/6/23 16:41:52:819 GMT] 000000f6 EchoBusinessO 1 com.ibm.commerce.foundation.server.services.changehistory.mediator.EchoBusinessObjectChangeHistoryDelegateMediatorImpl 
createChangeHistory The change history record to be created is: com.ibm.commerce.foundation.internal.server.services.changehistory.beans.TaskGroupChangeHistoryDataSet@a87cbc8e 
objectId = 14503 ui_object_name = Product objectType = CatalogEntry objectCode = MPHBA031_3103 MEMBER_ID = -1000 LASTUPDATE = 2023-07-06 16:41:52.793 ACTION = U STOREENT_ID = 12001 
WORKSPACE = IBM_WC_BASE TASKGRP = IBM_WC_BASE TASK = null

To confirm that a Redis message was generated for the change history event, you can look for the tracing output from EchoBusinessObjectChangeHistoryDelegateMediatorImpl.sendRedisChangeHistoryEvent(). For example, here is the tracing for the message being sent to Redis for the above change history event.

[7/6/23 16:41:52:863 GMT] 000000f6 EchoBusinessO 1 com.ibm.commerce.foundation.server.services.changehistory.mediator.EchoBusinessObjectChangeHistoryDelegateMediatorImpl sendRedisChangeHistoryEvent 
[{"workspace":"IBM_WC_BASE","objectCode":"MPHBA031_3103","catalogId":"11001","changeHistoryId":"10544","languageId":"-1","action":"U","nounName":"CatalogEntry","storeId":"12001","changeHistoryEventId":"1688661712825","objectId":"14503","taskGroup":"IBM_WC_BASE","actionDetail":{"updateSEOURL":"true"}}]

Redis messages

When subscribed to the WCNifiDistributedMapCache topic in Redis, you will see messages sent to this topic to report that an NRT event occurred and an ingest is to be kicked off to pull in these changes. For example, here is the message tied to the update made in Management Center.

[root@mymachine ~]# docker exec -it commerce_redis_1 bash
I have no name!@c155d5fa5da1:/$ redis-cli 
127.0.0.1:6379> subscribe {cache-auth-services/cache/WCNifiDistributedMapCache}-invalidation 
Reading messages... (press Ctrl-C to quit)


1) "message"
2) "{cache-auth-services/cache/WCNifiDistributedMapCache}-invalidation"
3) "[p:3e8153eee071;t:1688661712872]> {\"workspace\":\"IBM_WC_BASE\",\"objectCode\":
\"MPHBA031_3103\",\"catalogId\":\"11001\",\"changeHistoryId\":\"10544\",\"languageId\":
\"-1\",\"action\":\"U\",\"nounName\":\"CatalogEntry\",\"storeId\":\"12001\",\"changeHistoryEventId\":
\"1688661712825\",\"objectId\":\"14503\",\"taskGroup\":\"IBM_WC_BASE\",\"actionDetail\":
{\"updateSEOURL\":\"true\"}}"

If you are making a change that will affect your workspaces, you may also see a WorkspaceEvent message similar to the following.

1) "message"
2) "{cache-auth-services/cache/WCNifiDistributedMapCache}-invalidation"
3) "[p:616516a9bc98;t:1688661761125]> {\n\t\"nounName\": \"WorkspaceEvent\",\n\t\"storeId\": 
\"12001\",\n\t\"languageId\": \"-1\",\n\t\"catalogId\": \"11001\",\n\t\"workspace\": 
\"IBM_WC_BASE\",\n\t\"taskGroup\": \"IBM_WC_BASE\",\n\t\"objectId\" : \"14503\",\n\t\"runId\" : 
\"wn-b345b49e-40e2-4806-a391-d84a022d4c12\",\n\t\"workspaceEventDetails\": {\n\t\t\"workspaceEventMasterCatalogId\": 
\"11001\",\n\t\t\"workspaceEventSupportedCatalog\": \"11001,11502,11501\",\n\t\t\"workspaceEventDefaultCatalog\": 
\"\",\n\t\t\"workspaceEventConnectorName\": \"auth.product.cas\"\n\t}\n}"